Recently, the water quality of the second longest River in China (the Yellow river) is deteriorating severely. To investigate the heavy metals contaminated degrees, potential risks and pollution sources of heavy metals in the river, we measured the concentrations of 8 heavy metals (Fe, Mn, Cu, Ni, Zn, Cr, Pb and Cd) from 122 surface sediment samples collected along the upstream Yellow River. According to the results of enrichment factor (EF) analysis and the geoaccumulation index (I_geo) evaluation, we found Cd accumulated significantly for moderately contaminated level and the other heavy metals existed at uncontaminated levels. We conducted risk assessment by using calculation of the sediment quality guidelines (TEC-PEC SQGs) and potential ecological risk index (RI), and the results implied that the concentrations of Ni, Cr and Cd had potential harm for aquatic organisms and the upstream Yellow River had integral no toxic. Moreover, the results of multivariate analysis (principle component analysis, hierarchical cluster analysis and correlation analysis) indicated that Fe, Mn, Cu, Zn, Cr and Cd in the river primarily originated from natural sources, while Ni and Pb mainly derived from different human activities. In addition, partial Cu and Cd come from anthropogenic sources of Ni discharge.