In this paper, we present an optimal design of TE0n nonuniform ripple-wall mode converter in circular waveguide. The research work is based on the general coupled wave theory and numerical optimization method. The results of numerical calculation are in good agreement with those of simulation by HFSS. Compared with traditional TE0n mode converter of periodic waveguide perturbations, the TE0n nonuniform ripple-wall mode converter can achieve a high mode conversion efficiency with less corrugated periods. The length of the converter is shortened nearly by a half and the operating bandwidth with over 95% mode conversion is increased by 150%. The research work provides an important theoretical reference and a physics model for designing high power gyrotron mode converter with small axial size, wide operating bandwidth and high conversion efficiency.