In recent years, with the significant improvement of computer performance, deep learning technology has shown an explosive development trend and has been widely used in various fields. In this context, the computer-generated hologram (CGH) generation algorithm based on deep learning provides a new method for displaying the real-time high-quality holograms. The convolutional neural network is a most typical network structure in deep learning algorithms, which can automatically extract key local features from an image and construct more complex global features through operations such as convolution, pooling and full connectivity. Convolutional neural networks have been widely used in the field of holographic displays due to their powerful feature extraction and generalization abilities. Compared with the traditional iterative algorithm, the CGH algorithm based on deep learning has a significantly improved computing speed, but its image quality still needs further improving. In this paper, an attention convolutional neural network based on the angular spectrum diffraction model is proposed to improve the quality as well as the speed of generating holograms. The whole network consists of real-valued and complex-valued convolutional neural networks: the real-valued network is used for phase prediction, while the complex-valued network is used to predict the complex amplitude of the SLM surface, and the phase of the complex amplitude obtained after prediction is used for holographic coding and numerical reconstruction. An attention mechanism is embedded in the down sampling stage of the phase prediction network to improve the feature extraction capability of the whole algorithm, thus improving the quality of the generated phase-only holograms. An accurate diffraction model of the angular spectrum method is embedded in the whole network to avoid labeling the large-scale datasets, and unsupervised learning is used to train the network. The proposed algorithm can generate high-quality 2K (resolution ratio of 11920×1072) holograms within 0.015 s. The average peak signal-to-noise ratio of the reconstruction images reaches up to 32.12 dB and the average structural similarity index measure of the generated holograms can achieve a value as high as 0.934. Numerical simulations and optical experiments verify the feasibility and effectiveness of the proposed attentional convolutional neural network algorithm based on the diffraction model of angular spectrum method, which provides a powerful help for applying the deep learning theory and algorithm to the field of real-time holographic display.