Aims : High-resolution spectrographs are central to modern exoplanet research and are particularly effective for detecting Earth-like planets whose radial velocity (RV) signals can be only a few tens of centimeters per second. Achieving this level of precision requires highly accurate wavelength calibration. A key factor in this process is the modeling of the instrumental profile (IP), which describes the response of the spectrograph to incoming light. The true IP of a high-resolution instrument is often complex. It may show asymmetry or extended wings and change across the detector because of optical aberrations, variations in fiber illumination, and environmental effects. These features lead to systematic errors in the measured line centers when traditional parametric models such as Gaussian functions are used, and they limit the achievable RV precision.
Methods: This work introduces a non-parametric IP modeling method based on Gaussian Process Regression (GPR). The IP is treated as a smooth function with a flexible covariance structure instead of being constrained by a predefined analytic form. GPR learns both the global structure and small-scale features of the line shape directly from the data. Since the IP varies slowly across the detector, the method divides each spectral order into several consecutive spatial segments. Each segment is fitted independently, capturing local variations. The model includes measurement uncertainties and provides a probabilistic description of the IP. Adjacent segments are linked with smooth interpolation to ensure a continuous IP across the entire order. Model performance is evaluated using reduced chi-squared and root mean square error (RMSE), allowing quantitative assessment and comparison with traditional approaches.
Results: The method is tested with laser frequency comb (LFC) exposures from the fiber-fed High Resolution Spectrograph (HRS) on the 2.16 m telescope at Xinglong Observatory. The LFC produces a dense and highly stable set of emission lines and is well suited for validating IP reconstruction. Three experiments show clear and consistent improvements. Using odd-numbered lines to predict evennumbered ones within a single exposure reduces the RMSE by 35.6% compared with a Gaussian model, showing better determination of line centers. Applying an IP model trained on one exposure to a later exposure reduces the RMSE by 42.5%, demonstrating improved stability when the model is transferred between exposures. A comparison between two channels in the same exposure shows a 37.1% improvement in calibration consistency, indicating reduced channel-tochannel systematics.
Conclusions: The results show that GPR provides a more accurate description of the instrumental profile and its spatial variation than traditional parametric models. The improved reconstruction of the IP leads to more accurate line center measurements and a more stable and precise wavelength solution. This capability is important for pushing the RV precision of high-resolution spectrographs toward the centimeter-per-second level. GPR offers a promising approach for modeling instrumental profiles and supports the precision required for detecting Earth-like exoplanets.