Abstract: Geomagnetic matching technology can provide passive external resource of correction information for underwater vehicle, which improves the long voyage navigation accuracy of inertial navigation system. The filtering matching algorithm is the core technology in geomagnetic aided inertial navigation, which can effectively mitigate the influence from the uncertainty of geomagnetic observation noise. Based on track simulation data, the earth magnetic anomaly grid 2 (EMAG2) is selected as reference map in this paper. To properly model the observation noise of geomagnetic anomaly in unpredictable geomagnetic environment and the error from measuring instruments, this paper proposes the matching algorithm of geomagnetic anomaly filter based on residual errors. The observation noise variance is estimated adaptively through the residual-based adaptive estimation (RAE) filter. Meanwhile, the availability and robustness of the algorithm are improved by combining the optimal filter selection criteria. The validation experiments of RAE filter are conducted in different maritime space of the South China Sea. It is shown that the drifting errors of inertial navigation system in longitude and latitude can be reduced based on the RAE filter. Moreover, the positioning accuracy and reliability of the aided navigation system can be improved significantly. In obvious geomagnetic fluctuating region under the simulation condition, position accuracy is raised to 0.751 53 n mile and 0.778 45 n mile respectively in latitude and longitude directions through RAE filter.