The 252 rank completely diagonalized Hamiltonian matrix of the 3d5 configuration with trigonal symmetry have been established by group theory, crystal field theory and irreducible tensor operator method. The spectral fine structure, zero-field splitting parameters, and Jahn-Teller effect, as well s the influence of spin doublet state and spin quartet state on the ground-state energy levels in LiNbO3∶Fe3+ crystals were studied with this matrix. The results show that the contribution of spin quartet state on the ground- state energy levels is the most important, the contribution of spin doublet state is weak but can not be neglected. The calculated results are in good agreement with the experiments. On their basis, the influence of spin-orbit interaction and spin-spin interaction on the spectra fine structure and zero-field splitting parameters were further studied, and we found that the influence of spin-orbit interaction is dominant, but the influence of spin-spin interaction can not be neglected. The research shows this substance has J-T effect in the spectral structure of spin quartet state. The reason is the combined action of spin-orbit interaction and trigonal distortion.