Photovoltaic (PV) grid integration has been the epicenter of research across the globe since their intermittent nature of solar generation can be more predictable. Irradiance forecast using different methods for various time horizons has been the center of attention in the recent literature. In this study, a framework for a very short term irradiance forecast is proposed via combining image processing and machine learning. A series of whole sky images is used for this purpose. Cloud detection and movement tracking are accomplished based on image processing algorithms, future position of the clouds and occlusion to the sun. Then, the irradiance drop is predicted using machine learning algorithms. The effectiveness of the proposed technique is evaluated by the Root Mean Square Error (RMSE) between the actual and forecast values of solar irradiance.