Despite the widespread adoption of Gradient Boosting Decision Trees (GBDTs), practitioners lack systematic criteria for determining when linear models are more effective. This knowledge gap impacts model selection in applications where computational efficiency, interpretability, and extrapolation capabilities are required. This study addresses this issue through five systematic experiments that isolate data characteristics: linearity dominance, feature interactions, extrapolation requirements, small-sample scenarios, and interpretability needs. Our multi-dimensional evaluation framework integrates predictive performance with computational and interpretability costs, providing a comprehensive empirical comparison of linear regression and GBDTs. Linear models significantly outperformed GBDTs under four critical conditions.