Coffee is one of the crucial agricultural product in the global economy, particularly for Ethiopia. However, diseases like brown eye spot, wilt, and rust are the most determinant constraints the productivity and quality of coffee export. Existing of variational autoencoder (VAE). MobileNetV3, acting on extracted features contain complementary information that enriches a unified feature map. Second, the extracted images from models are fused in the early fusion network. That is deployed to learn the rich information from the extracted feature. The late fusion network is implemented to learn the fused feature before a classification network classifies coffee leaf diseases. The proposed hybrid feature fusion approach is evaluated on a challenging, real world Robusta Coffee Leaf (RoCoLe) dataset with various diseases, including red spider mite and leaf rust disease. As a result, an autonomous method for detecting and classifying coffee plant disease become very crucial for better productivity. To determine whether a particular image of a leaf disease or if it is healthy, we created a deep learning model and RNN trained with image dataset collected from the Wolaita Sodo agricultural research center consisting of 1,120 and augmentation technique also applied to handle data over-fitting problem and totally 3,360 images were used.