Abstract: Three-axis magnetometer is a measuring tool commonly used for magnetic field measurement, which is widely used in geomagnetic navigation, ocean magnetic survey, geomagnetic exploration and other fields. As a magnetic field vector measuring tool, the magnetic measurement error of the three-axis magnetometer mainly comes from three-axis non-orthogonal, sensitivity inconsistency of sensitive axis and zero offset error. The calibration of the magnetometer affects the measurement performance of the magnetometer. In order to solve the calibration problem of three-axis magnetometer, the error compensation model for the magnetometer is established. The calibration method of the three-axis magnetometer based on sine fitting is proposed. The output data of the three-axis magnetometer is fitted into the sine curve when the magnetometer is rotated on a plane. The calibration parameters of the three-axis magnetometer are calculated by using the information of amplitude, initial angle and translational value of sine curve. The simulation results show that the proposed method can solve the calibration problem without the process of nonlinear optimization, and the calculated calibration parameters are in good agreement with the set values. The measurement noise has little effect on the three-axis sensitivity coefficient and the non-orthogonal angle.