Using the density functional theory(B3LYP) method, the 6-311++G(3df,2pd), AUG-cc-PVTZ, AUG-cc-PVQZ basis sets for H and effective core potentials for Y, the energies, equilibrium structure and harmonic frequency of the ground states of YH(D,T) molecules are calculated. Based on the theory of atomic and molecular reaction statics, the reasonable dissociation limits of the ground states of YH(D,T) molecules are derived. By comparing the calculation results with the existing experimental and theoretical values, we find that the mixed basis sets LANL2TZ/AUG-cc-PVQZ are most suited for the calculation of the molecules. Consequently, the potential energy surfaces of the ground states of YH(D,T) molecules are scanned at the B3LYP/LANL2TZ/AUG-cc-PVQZ level of theory. The potential energy curves of the ground states of YH(D,T) molecules are obtained by the least square fitting to the Murrell-Sorbie potential energy function. The spectroscopic constants (Be, e, e, ee, De) and force constants ((f2, f3, f4)are calculated and compared with experimental results, indicating that the calculation results are in good agreement with the experimental data.