The parent rock properties, weathering degree and soil formation conditions of soil in different regions and different levels are different, which results in the obvious difference of grain size composition of soil. At present, the research on granite residual soil often ignores the mechanism of its grain size composition on engineering properties. In this study, a series of soil samples with different grain size composition were configured based on the content of gravel grain in the granite residual soil. This paper mainly studies the influence of the content of gravel grain group and fine grain group of the compaction and strength of granite residual soil. The results show that the optimal moisture content and cohesion of granite residual soil are mainly controlled by the content of fine grain, and the maximum dry density and internal friction angle are mainly controlled by gravel grain. For each 1% increase in the fine grain content of the soil, the optimal moisture content of the soil will approximately increase by 0.11%, and correspondingly, the cohesion of the soil will approximately increase by 0.27 kPa. In addition, the engineering properties of granite residual soil with different grain size composition are also affected by soil structure. Finally, based on the microstructure characteristics of granite residual soil, three different aggregates are proposed, and seven micro models with different structural characteristics are established. Micro models established by different aggregates can control different engineering properties of soil. The research results are helpful to understand the relationship between the engineering properties and grain size composition of granite residual soil and provide reference for the refinement of its engineering classification.