The heavy metal pollution of soil caused by mining activities has attracted widespread attention. This study investigates the spatial distributions, pollution characteristics, and sources of nine heavy metals (Ni, Cu, Pb, Zn, Cr, Cd, As, Sb, and Hg) in the farmland soil around an antimony mine, Hunan Province, China. The results show that the average contents of Ni, Cu, Pb, Zn, Cr, Cd, As, Sb, and Hg are all above the background values of the surface soil in Hunan Province, and the mean concentrations of Cd and As have exceeded China’s soil pollution risk screening values for agricultural land. The highest Sb content obtained in this study is 1234.3 times of the maximum allowable antimony content in the soil. The average content of heavy metal elements in the farmland soil samples shows the following trend: Sb>Zn>Cr>As>Pb>Ni>Cu>Cd>Hg. The content of each heavy metal in garden land is higher than that in arable land. Geoaccumulation index analysis results suggest the overall pollution level of the farmland soil is light. In some areas, the pollution level of Cd has reached moderate to strong pollution, the pollution levels of Sb and Zn appear to be moderate pollution to moderate to severe pollution, and the pollution levels of Ni, Cu, Pb, Cr, As, and Hg are at moderate pollution to light pollution. The correlation analysis and positive matrix factorization (PMF) analysis reveal that As, Sb, and Hg are mainly caused by the mining activity, the agricultural activity is the main reason for the accumulation of Cd, and natural factors are the main sources of Cu, Pb, Cr, Ni, and Zn in the study area.