With the continuous emission of greenhouse gases, the carbon trading market has become a powerful weapon to contain it. It is indispensable to analyze the carbon price of China that acts as the largest emitter of carbon dioxide worldwide. Therefore, this paper proposes an innovative hybrid carbon price forecasting model that incorporates fast ensemble empirical mode decomposition (FEEMD) and extreme learning machine optimized by particle swarm optimization (PSO-ELM) with external and internal influencing factors considered. The original carbon price series are disassembled into several intrinsic mode functions (IMFs) and one residual via FEEMD. The PSO-ELM is then employed to forecast the sub-series. It’s remarkable that the inputs of the PSO-ELM model are divided into external and internal influencing factors. Factor analysis is used to extract potential factors from energy prices, macroeconomics and other influencing factors associated with the original carbon price as external influencing factors, and the partial autocorrelation function (PACF) is exploited to select internal influencing factors. A case study in Hubei Province, China shows that the proposed carbon price forecasting model is superior to the contrast models in terms of the smallest prediction error (MAE = 0.1274 yuan, MAPE = 0.8368%) and the strongest stability (RMSE = 0.0116 yuan). And the forecasting results demonstrate that the developed model with external and internal influencing factors considered can highly improve carbon price prediction performance and have potential in a wider range of carbon price forecasting. In addition, accurate carbon price forecasting can help the government realize macro control and the investors fulfill risk minimization.