Optical fiber communication systems are going to adopt the use of advanced modulation formats. It is thus important to develop measurement techniques and solutions capable of quantifying such signals. Linear optical sampling is an effective technique to characterize the quality of an advanced modulation format for high-speed optical signal with high fidelity, while the passively mode-locked fiber laser is an enabling module to implement the linear optical sampling. In this paper, we obtain a trade-off relationship between the repetition rate of passively mode-locked fiber laser and the linewidth of high-speed signal under test, after the introduction of operation principle for linear optical sampling. It is found that, for the quadrature phase shift keying (QPSK) signal, when the ratio of the linewidth of the signal under test to the repetition rate of passively mode-locked fiber laser is less than 1.5×10-3, the linear optical sampling-induced impairments can be ignored when there occurs phase noise. Therefore, the phase estimation can be successfully made by using the Viterbi-Viterbi algorithm applied to the block of samples corresponding to the modulation format phase states. Next, we use an optical sampling pulse with a repetition rate of 95.984 MHz, and carry out the optical linear sampling to a 28 Gbaud QPSK signal with a linewidth of 100 kHz. The error vector magnitude (EVM) has long been a commonly used parameter for quantifying the quality of advanced modulation signals. Using the standard coherent detection algorithm, we can successfully recover the constellation with the error vector magnitude (EVM) error less than 1%. Theoretical investigations agree well with the experimental characterization. Such a conclusion is helpful to optimize the design of passively mode-locked fiber laser for optical sampling application.