A typical relative rotation system is considered whose chaotic motion and basin erosion caused by external excitation is investigated in this paper. And a delayed position feedback control is applied in the system for suppressing the two types of complex dynamical behaviors. Firstly, the excitation amplitude threshold of chaotic motion and the basin erosion of an uncontrolled relative rotation system is obtained by the Melnikov method. Secondly, the condition of Hopf bifurcation of a delay controlled system is discussed so as to obtain the available ranges of control parameters in the Melnikov method. Then the necessary condition for the global bifurcation of a delay controlled system is obtained. Finally, the evolutions of the dynamical behavior of the delay controlled system together with its control parameters are presented numerically using the 4th Runge-Kutta method and the point-to-point mapping method, which confirm the validity of the theoretical prediction. It is found that the chaotic motion and basin erosion can be suppressed effectively by delayed position feedback control when the gain is positive and the time delay is short.