Abstract: Objectives: RGB and Motion features are very important for ship video abnormal event detection. We need to extract these features in video more accurately and apply them to the detection of abnormal events in ship video. Meanwhile, due to the huge cost of frame-level annotations, we also need to solve the problem of not providing frame-level annotations in the model training stage, but using video-level annotations for model training. In addition, we also need to solve the problem of the scarcity of ship video abnormal event database. Method: We acquired a large number of surveillance videos of ships on the sea surface and constructed a data set of abnormal events of ship video after processing. Also, we proposed a ship abnormal event detection model based on deep spatiotemporal autoencoder network and multi-instance learning, using a deep multi-instance ranking framework, without obtaining frame-level annotations, only video-level information is needed. In addition, In the spatial autoencoder, in order to obtain more accurate RGB features, the deconvolution layer in the decoder is replaced by a pixel shuffle layer. In the temporal autoencoder, in order to highlight the regions with large motion variation, a variance-based attention mechanism is introduced, so that fast-moving objects have a larger motion loss. Results: We compared the proposed method with the two benchmark methods and a previous state-of-the-art method. The experimental results show that the proposed method has higher detection accuracy. In addition, we observed that variance-based attention can significantly improve the detection effect of fast motion, such as unexpected stopping and overspeed. Conclusion: This shows that RGB and motion features play an important role in ship abnormal event detection and also proves the necessity of multi-instance ranking model.