Abstract: This paper discusses the ill-posed nonlinear least squares problem, and proposes an adaptive relaxation algorithm based on the regularization method for stabilizing the nonlinear parameter estimation. The improved algorithm achieves the adaptive selection on the regularization parameter and iterative step by using an incremental geometric regularization parameter and the minimal residual criterion. The numerical convergence experiments of the method are performed. The results show that the numerical precision of our proposed method is better than that of the linearized adjustment estimation, and the convergence property is more efficient than the iterative Tikhonov regularization method.