Abstract: Objectives: Multiple object tracking (MOT) plays a key role in intelligent unmanned system, providing accurate and reliable object perception information for autonomous unmanned systems. Among them, three dimensional (3D) visual MOT stands out due to its strong environmental perception ability and low sensor cost. However, the existing 3D visual MOT algorithm is usually implemented based on the filtering framework, and only uses the detection boundary box to update the object motion state, so its MOT performance in complex scenes is vulnerable to the uncertainty of object detection. Therefore, a 3D visual MOT algorithm considering motion constraints is proposed. Methods: In the aspect of object data association, a six degree of freedom object motion model is established to accurately predict the object trajectory to guide the object detection boundary box matching and reduce object confusion and false association. In the aspect of object trajectory optimization, the vehicle motion constraints in vehicle navigation are extended to the object, and the zero velocity update (ZUPT) and non-holonomic constraint (NHC) of the object trajectory state are established. Combined with the object motion prediction trajectory and the detection boundary box information, the factor graph optimization model based on sliding window is constructed, which realizes the joint accurate estimation of the multi-object motion state and improves the prediction and tracking accuracy of the multi-object trajectory. Results: The experimental results show that the algorithm is 15.69% and 12.96% higher than AB3D MOT and PC3T MOT on multiple object tracking accuracy (MOTA), and 0.96% higher than PC3T MOT on multiple object tracking precision (MOTP). Conclusions: The algorithm enhances the accuracy and continuity of pose estimation in complex 3D environment, and achieves robust MOT performance.