Speaker diarization finds continuous speaker segments in an audio stream and clusters them by speaker identity. In this paper we propose a method for Speaker diarization by using a new area of machine learning, i.e Deep learning. For Speaker segmentation we trained one of the Deep Learning Network by short-term spectral features to predict given speech segments belongs to same or different speaker and for Speaker clustering HMM speaker models has been used for speaker recognition from the given set of speakers.