In order to improve the time delay estimation accuracy of the observed profile in the X-ray pulsar based navigation, the spectral characteristics of the observed profile of X-ray pulsar and the drawback of the classical Taylor fast Fourier transform (FFT) time delay estimation method are analyzed. It is found that when estimating the time delay, we can abandon the higher frequency components that are always affected by noise seriously, but only utilize the information about the low frequency part. Based on this idea, by modifying the weigh function of the classical Taylor FFT time delay estimation method, a new time delay estimation algorithm based on the optimal frequency band is proposed, in which the optimal frequency band is determined by establishing the relationship between the selected frequency band and the time delay estimation accuracy under different signal-to-noise ratios (SNRs). Then by using the real data obtained with the proportional counter array, the low-energy (2-60 keV) detection instrument boarded on the Rossi X-ray Timing Explorer satellite, the optimal frequency as a function the SNR of observed profile is given for the PSR B0531+21 (namely the Crab pulsar) through the Monte-Carlo technique. Since the parameters of different pulsars are known, in practical navigation, the optimal frequency in an observation time for a certain pulsar can be estimated in advance by using the simulation data or the obtained real data of the pulsar, which can remarkably alleviate the onboard computational burden. Finally, a series of numerical simulations and experiments using real data of Crab pulsar are designed to evaluate the performance of the proposed time delay estimation algorithm. The main results can be summarized as follows: the proposed estimator outperforms the normally used fast approximate maximum-likelihood (FAML), cross correlation (CC), nonlinear least square (NLS) and weighted nonlinear least-square (WNLS) estimators when the observation time is short or the source flux is small; when the observation time is long or the source flux is large, its estimation accuracy is almost the same as those of CC and NLS estimators and lower than those of the FAML and WNLS estimators, but its computational complexity is smaller than those of NLS, FAML and WNLS estimators. The above results indicate the high estimation accuracy and high computational efficiency of the proposed time delay estimation method, which can be used in the case that the observation time is restricted to be short or the source flux of the usable pulsar is small in X-ray pulsar based navigation.