Thoracic echocardiography (TTE) can provide sufficient cardiac structure information, evaluate hemodynamics and cardiac function, and is an effective method for atrial septal defect (ASD) examination. This paper aims to study a deep learning method based on cardiac ultrasound video to assist in ASD diagnosis. We chose four standard views in pediatric cardiac ultrasound to identify atrial septal defects; the four standard views were as follows: subcostal sagittal view of the atrium septum (subSAS), apical four-chamber view (A4C), the low parasternal four-chamber view (LPS4C), and parasternal short-axis view of large artery (PSAX). We enlist data from 300 children patients as part of a double-blind experiment for five-fold cross-validation to verify the performance of our model. In addition, data from 30 children patients (15 positives and 15 negatives) are collected for clinician testing and compared to our model test results (these 30 samples do not participate in model training). In our model, we present a block random selection, maximal agreement decision, and frame sampling strategy for training and testing respectively, resNet18 and r3D networks are used to extract the frame features and aggregate them to build a rich video-level representation. We validate our model using our private dataset by five cross-validation. For ASD detection, we achieve 89.33±3.13 AUC, 84.95±3.88 accuracy, 85.70±4.91 sensitivity, 81.51±8.15 specificity, and 81.99±5.30 F1 score. The proposed model is a multiple instances learning-based deep learning model for video atrial septal defect detection which effectively improves ASD detection accuracy when compared to the performances of previous networks and clinical doctors.