Due to the gradual agglomeration of economic activities and the continuous reinforcement of spatial linkages in specific geographic locations, the geospatial factor should become an important starting point to understand the relationship between industrial restructuring and energy conservation and emission reduction. This paper first introduces a non-separable hybrid DEA model that considers undesirable output to measure the energy efficiencies of 285 prefecture or higher-level cities in China during 2003-2016; then, a dynamic spatial panel model is used to investigate the influence of different types of industrial agglomerations and agglomeration modes on energy efficiency. According to the obtained study results, for the investigation period, the overall energy efficiency of China with regard to pollutants remained at a low level and presented a “U-shaped” decreasing-increasing trend. To be specific, China’s energy efficiency distribution presented a trend of “high in the east and low in the west.” The energy efficiency of East China changed relatively gently, while the energy efficiencies of central China and western China changed dramatically. China’s energy efficiency also presented a significant spatial agglomeration effect, i.e., cities with close energy efficiencies are usually adjacent to each other. At the national level, agglomeration of the manufacturing sector significantly inhibited the increase of energy efficiency; the agglomeration of the producer service sector and the co-agglomeration of the manufacturing sector and the producer service sector both facilitated an increase of energy efficiency. The influence of industrial agglomeration on energy efficiency differed across different city scale grades. Based on these conclusions, the paper proposes the following policy implications: 1) make full use of the energy savings and emission reduction effect of agglomeration; 2) accelerate the optimization of industrial layout; 3) develop high-end service industry and productive service industry; and 4) create an agglomeration environment that encourages benign industrial competition.