The off-axis illumination pupil function of the wavefront coding system with cubic phase mask is analyzed by coordinate-rotation for the first time. The approximate expression of the pupil function shows that off-axis illumination brings about the extension effect of cubic phase and defocusing. The extension effect is independent of the sign of the incident angle; however, the extension effect will get larger as the absolute value of the angle increases. This causes a decrease in modulation transfer function (MTF) and the extension effects on both sides of the point spread function (PSF) envelope. When the off-axis illumination is in the tangential plane, the extension effects of cubic phase and defocusing in y direction is larger than that in x direction, which results in larger extension of PSF and lower MTF in the y direction. When off-axis illumination is in the sagittal plane, the extension effects of cubic phase and defocusing in x direction is larger than that in the y direction, which results in larger extension of PSF and lower MTF in x direction.