Food security is crucial to a nation’s survival and development, yet the shift of cultivated land toward non-grain crops has led to a reduction in both the area and quality of farmland, while also potentially exerting a negative impact on the agricultural environment. The increase in agricultural production and farmers’ income has laid a solid foundation for ensuring food security. However, agricultural production methods that come at a high cost to resources and the environment have profound negative impacts on the ecological environment. Agricultural ecological efficiency (AEE) serves as a crucial indicator for assessing the efficiency of agricultural resource utilization and environmental pressure. Research on AEE can provide valuable insights for the government in formulating agricultural policies. As China’s agricultural structure undergoes transformation, the phenomenon of non-grain production (NGP) is becoming increasingly prominent, posing challenges to the ecological environment. This paper aims to evaluate the ecological sustainability of China’s agricultural sector and analyze the challenges posed by NGP to AEE. Based on field survey data from Jiangxi Province, this paper employs the super-efficiency SBM model, treating agricultural pollution as an undesirable output, to measure AEE. We construct an endogenous switching model and a mediation model to explore the impact of NGP on AEE. The average AEE of the sampled farmers was 0.363, indicating significant room for improvement. NGP significantly reduced AEE; farmers engaged in NGP exhibited a 40.90% decrease in efficiency compared to a hypothetical scenario of grain-oriented production. Mechanism analysis revealed that NGP led to reduced adoption of socialized services, increased agricultural labor input, and higher usage of agricultural pollutants, thereby lowering AEE. Therefore, mitigating the negative impacts of NGP requires systematic interventions, such as increasing the supply of socialized services, optimizing labor allocation, and reducing the application of chemicals.