Fabry-Peort (FP) cavity-based quantum key distribution has some advantages in the phase-coded quantum key distribution (QKD). Locking the frequency of the FP cavity transmission to the frequency of single photon carrying the key information determines the security of QKD. In this paper, we propose an FP cavity locked to a single photon (the mean photon number is about 0.1) frequency using single-photon modulation/demodulation method. The single photon detector working in the Geiger mode is used to detect modulated single photons then direactly demodulated by the lock-in amplifier. The discrete single photon response pulses accumulated fulfills the phase sensitive detection, and the signal-to-noise ratio of signal frequency discrimination is up to 112. The transmission frequency fluctuation of the locked FP is limited to 2 MHz.