Optimized Ensemble Learning for Classification

The dataset was a million rows and twenty-two anonymised columns, with a binary target. No feature meant anything you could reason about, which removes domain intuition as a tool and leaves you with what the data actually looks like.

What it looked like was awkward. Nine rows in ten belonged to the majority class. One column was almost entirely missing and another was speckled through with gaps. Several columns were nearly constant. There were outliers. And of twenty-two features, exactly one — Column18 — correlated strongly with the target.

Learning from what is missing

Missing values were filled by Bayesian ridge regression, but not before recording that they were missing: for the heavily-empty columns, a flag saying whether this row had a value at all; for the near-constant ones, a flag saying whether this row held the usual value or something else. Both flags survive into the correlation matrix above, and both carry signal that imputation would otherwise have erased.

Outliers were capped by Z-score and IQR rather than dropped. Categorical columns were min-max scaled, continuous ones Z-normalised — that order, because capping the outliers first is what makes Z-normalisation safe.

What we tried

Random forests, with and without class weights. Deep networks, with and without class weights. An autoencoder pretrained on the inputs with a classifier head on top. A much larger network. An autoencoder trained only on the minority class, classifying by reconstruction error. A Siamese network under contrastive loss, classifying in the learned embedding space.

None of them was the answer, and the reason is the same in every case: a single model tuned on this data trades one class against the other.

Two bad models make a good one

The final model stops trying to find one good classifier and deliberately builds two bad ones.

• An XGBoost classifier tuned to be right about the minority class. On the test set it calls 2,521 rows positive and 2,507 of them really are — but it finds only a tenth of the positives that exist. As a classifier it is close to useless.
• A deep network tuned to be right about the majority class. It misses two positives out of 24,678 — but it drags 9,437 negatives along with them.

Neither is a model you would deploy. What each one is good at is being trusted in one direction: when the first says yes, it is yes; when the second says no, it is no. A small meta network takes both outputs as its input features and learns where to believe which.

That combination reaches 97.8% accuracy on the test set, with 0.85 precision and 0.93 recall on the minority class — an F1 of 0.89, from two components that score 0.18 and 0.84. Train and test scores sit close together, which argues against overfitting, though it does not by itself establish how the model would behave on new data.

That is the finding worth keeping. On tabular data of this shape, effort spent on understanding the missingness and on combining complementary models paid better than effort spent on model capacity.

Team GSTN_715 — this was a team entry, not a solo project. The repository carries the full write-up, the model comparison, the trained ensemble and the scripts to retrain it or run predictions.