Comment classification: feature engineering and predictive modelling
Feature engineering over 198,000 user comments across four imbalanced classes, lifting F1-Macro from a 0.6221 baseline to 0.8293 with a blend of 3 models, led by an Optuna-tuned LightGBM.
- Primary Metric
- 0.8293 F1-Macro (39.6K Held-Out Rows)
- Dataset Scale
- 198,000 Comments · 4 Label Classes
- Feature Corpus
- 10,000+ Word & Character n-grams + Behavioral Signals
- Hyperparameter Tuning
- 100-Trial Optuna Study (LightGBM)
Context and Objective
Detecting toxic and obfuscated user comments requires handling subtle spelling variations, character substitutions, and severe class skew across multi-class labels. I used F1-Macro, which weights every class equally, so the most common class could not dominate the score.
Feature Engineering and Statistical Screening
I engineered over 10,000 word and character-level n-grams combined with behavioral text signals. I also ranked 11 non-text columns across 198,000 rows with chi-square and Kruskal-Wallis tests to see which ones related to the label.
Optimization and Results
- Optuna Hyperparameter Search: Executed a 100-trial study over 10 LightGBM parameters, using 3-fold cross-validation with median pruning to stop unpromising branches early.
- Performance Progression: Baseline Logistic Regression achieved 0.6221 F1-Macro. Tuned LightGBM reached 0.8202 F1-Macro. A blend of 3 models reached the final 0.8293 F1-Macro on about 39,600 held-out rows.
Stack
Python · scikit-learn · LightGBM · Optuna · pandas · numpy