A Comparative Analysis of Code Smell Detection: Rule-Based, Machine Learning, and Deep Learning Approaches

Abstract

Code smells indicate potential design flaws that can reduce software maintainability and quality. This paper presents a comparative analysis of three primary detection methodologies: Rule-Based, Machine Learning (ML)-Based, and Deep Learning (DL)-Based approaches. Rule-based methods, though widely used for their efficiency and interpretability, struggle with rigidity and high false positive rates. ML-based approaches improve adaptability by learning from labeled datasets, but their effectiveness depends on feature engineering and dataset quality. DL-based techniques further enhance detection accuracy by automatically extracting semantic and structural code patterns, yet they introduce high computational costs and lack interpretability. Our findings suggest that hybrid detection frameworks, combining rule-based heuristics with ML/DL models, achieve a better balance of accuracy, efficiency, and generalization. Future research should focus on explainable AI (XAI) for DL-based detection, cross-language generalization, and real-world integration into IDEs and CI/CD pipelines. In addition, we propose that integrating XAI techniques, such as LIME, SHAP, and attention visualization, can enhance the interpretability of deep learning models in code smell detection, offering more transparent insights into model decisions.