By using the nonequilibrium Green's function method, the ballistic thermal rectification in the three-terminal graphene nanojunction is studied. The dynamics of atoms is described by the interatomic fourth-nearest neighbor force-constant model. The nanojunction has a Y-shaped structure, created by a combination of a straight graphene nanoribbon and a leaning branch as the control terminal holding a fixed temperature. No heat flux flows through the control terminal. There exists a temperature bias between the two ends of the graphene nanoribbon serving as the left and right terminals, respectively. The primary goal of this paper is to demonstrate that the ballistic thermal rectification can be introduced by the asymmetric structure with different connection angles between terminals. The control terminal has a smaller connection angle with respect to the left terminal than to the right terminal. The forward direction is defined as being from the left terminal to the right terminal. The results demonstrate that, given the same control temperature and absolute temperature bias, the heat flux in the graphene nanoribbon tends to run preferentially along the forward direction. When the difference between the connection angles increases, the rectification ratio rises. Compared with that of the zigzag graphene nanoribbon, the rectification ratio of the armchair nanoribbon is much sensitive to the direction the control terminal. However, the greatest rectification ratio is found in the zigzag graphene nanoribbon which has a connection angle of 30 degrees with respect to the armchair branch. In addition, the direction of the control terminal can be adjusted to raise more than 50% of the rectification ratio of the graphene thermal rectifier based on the width discrepancy between the left and right terminals. The mechanism of the ballistic thermal rectification is also discussed. In the three-terminal graphene nanojunction, a smaller connection angle with respect to the control terminal leads to more phonon scatterings. The confirmation of this conclusion comes from a comparison of phonon transmission between different couples of terminals, which shows that in most of the frequency spectrum, the phonon transmission between the control terminal and the left terminal is smaller than between the control terminal and the right terminal. Given the same control terminal temperature and temperature bias, the asymmetric connection angles therefore will introduce a higher average temperature of the left and right terminals, and a larger heat flux in the forward process. Moreover, the average temperature difference between in the forward process and in the reverse process is found to be proportional to the temperature bias, and the proportionality coefficient will become bigger if the asymmetry is strengthened.