Non-point source (NPS) pollution contributes greatly to the contamination of surface water quality and has aroused widespread concerns. NPS pollution is influenced by a multitude of site-related factors whose effects are complicated. We estimated NPS pollution with a soil and water assessment tool (SWAT) model in China’s Fan River watershed. A new method, boosted regression tree (BRT), was proposed to study the relationship of impact factors on NPS pollution. We analyzed the effects of elevation, land use, soil, and slope on the patterns of sediment transport, total nitrogen (TN), and total phosphorus (TP). The results showed that R² values were higher than 0.76, and NSE was higher than 0.67. The SWAT model can estimate NPS pollution effectively in a study area. Although the spatial pattern of sediment and TP was quite consistent, the relationship between sediment and TN was weak. The contribution of impact factors for sediment TN and TP were different. Slope is the most important impact factor for sediment and TP load. Land use is the most important impact factor for TN load. The BRT model can reduce barriers to factor complexity and promote understanding of the NPS pollution formation mechanism. We proposed control strategies of pollution sources, and our research has proven to be useful for the explanation of impact factors in NPS pollution study, which is meaningful for NPS pollution control.