The bifurcation and chaotic motion of some nonlinear relative rotation system with heteroclinic orbit is studied. By using dissipative system Lagrange equation, the dynamics equation of nonlinear relative rotation system under combination harmonic excitations is established. Firstly, the bifurcation response equation of relative rotation system under combination resonance is deduced with the method of multiple scales. Singularity analysis is employed to obtain the transition set of steady motion. Secondly, the Melnikov function of heteroclinic orbit is solved according to heteroclinic orbit parameter equation of relative rotation system. The critical condition of chaos about Smale commutation is given. Finally, numerical method is employed to analyze the influences of different system parameters on chaotic motion by bifurcation diagram, the maximum Lyapunov, phase trajectory and Poincare map.