This study targeted a historical zinc smelting site from southwestern China and investigated the distribution, mobility, and dispersion patterns of trace metals in the surrounding environment. A series of soil, smelting slag, stream sediment, stream water, and groundwater samples were collected from the smelting site and analyzed for Pb, Zn, and Cd. The results showed that the smelting site was seriously polluted with Pb, Zn, and Cd from past smelting fume deposition and smelting slags. Metal fractions and ratios were applied to identify metal dispersion and mobility. During smelting, Cd was preferably volatilized and transported into the surrounding environment, Zn was predominantly recuperated, and Pb was predominantly retained in smelting slags. Metals in contaminated soils were more mobile than those in slags, but slags still posed long-term risk to the surrounding environment with elevated metals. The relative amounts of vertical transport of metals appeared to follow the order of Cd > Zn > Pb, similar to the mobility order by BCR sequential extractions. Sediments and suspended particulate matter (SPM) in the local stream were derived from the contaminated soils and slags. Compared to the local groundwater with little metal pollution, the stream water at the smelting site was heavily polluted with Pb, Zn, and Cd. Elevated metals in the stream water occurred predominantly in SPM due to soil and slag erosion.