Use of high amount of reclaimed asphalt pavement (RAP) in roadway construction brings considerable environmental and economic benefits. However, these gains are accompanied by challenges in design and construction to ensure the pavement quality is not compromised. Among the factors to consider is to determine how the RAP aged binder affects the adhesion between the binder and aggregate. This study was undertaken to address the preceding question using different aggregates when brought in contact with the RAP binder. The Atomic Force Microscopy (AFM) for recovered binders and one of the original ones were made, in order to understand their chemical composition and its relationship with the adhesion. Adhesion was evaluated using the bond strength test (BBS), wettability test, and surface free energy. The results were used to establish the Humidity Damage Index (HDI). This index is defined as the relationship between the work of adhesion asphalt-aggregate and the work of debonding or reduction in the system´s surface free energy, Results indicated that (Z)Recovered RAP(CO) is highly resistant to moisture damage in field combined with Sandstone SDM(CO) and medium resistance with Limestone(CO) and Sandstone SDA(CO). The different mineralogy’s aggregates showed a significant influence of the HDI result. The combination of aggregates and the Colombian 60–70 dmm asphalt P indicated high susceptibility to moisture damage. The combination of Silica (US) with PG 64-22 asphalt showed the highest resistance to moisture damage. The AFM and SARA results show the huge difference between recovered binders from Colombia and the USA, in fact the (Z) Recovered RAP(US) had better colloidal distribution than the original binder pen 60–70 dmm (CO).