ResNet-50 reached 0.696 weighted F1 and 0.695 accuracy, beating the strongest from-scratch CNN by +0.098 F1 and +8.4 percentage points in accuracy. In this three-person graduate ML study, I contributed to testing whether ImageNet pretraining still helps when inputs are only 48×48 grayscale and the majority class is 16 times larger than the smallest. The harder problem was separating architecture gains from interpolation, augmentation, class imbalance, and noisy labels.
FER-2013 makes naive shortcuts unreliable. Across 35,887 images and seven emotion classes, happy represents 25% of the dataset while disgust represents only 1.5%, creating a 16:1 imbalance. The images are also only 48×48 grayscale. EDA ruled out simpler feature-based approaches: per-class distributions of mean pixel intensity and pixel standard deviation were nearly identical, so global brightness and contrast carried little discriminative signal. Mean FFT magnitude spectra were also nearly indistinguishable and low-frequency dominated, which pushed the modeling strategy toward spatial geometry such as brow position, mouth curvature, and eye shape. Manual inspection exposed a second problem: duplicate groups of up to 14 near-identical images with conflicting labels, placeholder graphics, and non-face objects. That noise motivated augmentation and loss-function ablations rather than assuming clean supervision.
What Mattered Most
Methodology
&
Insights
I treated the study as a controlled comparison rather than a single-model tuning exercise. The baseline capacity study covered four CNNs, from a 2-stage 32-64 underfitting check to a 4-stage 64-128-256-512 stress test. Transfer-learning inputs were upscaled from 48×48 to 224×224 with nearest-neighbor, bilinear, bicubic, or Lanczos interpolation. The multi-model benchmark crossed those four methods with VGG-16, ResNet-50, EfficientNet-B0, and a baseline CNN for 16 runs. Transfer models used staged training: five epochs with the ImageNet backbone frozen and a new seven-class head trained with Adam at 1e-3, followed by fifteen epochs with differential learning rates of 1e-4 for the backbone and 5e-4 for the head, weight decay 1e-4, and cosine annealing. I then compared augmentation, loss, sampling, optimizer, learning rate, weight decay, and fine-tuning depth to isolate which decisions actually moved performance.
What Actually Moved Performance
The strongest full-data ResNet-50 reached 0.696 weighted F1 and 0.695 accuracy, versus 0.598 weighted F1 and 0.611 accuracy for the strongest from-scratch CNN. The gain was +0.098 F1 and +8.4 percentage points in accuracy. Within the subset benchmark, interpolation barely moved ResNet-50: weighted F1 ranged from 0.5678 to 0.5804, only a 0.0126 spread, so architecture mattered more than the upscaling choice. Fine-tuning depth mattered more. Training the last two blocks plus head reached 0.579 for ResNet-50, compared with 0.535 for the last block and 0.361 for the head alone. Augmentation also showed diminishing returns: 0.544 with none, 0.577 with flip/rotation/jitter, and 0.584 after adding scaling; translation, crop, and erasing returned 0.574–0.579. Loss choice was similarly secondary, with five sampling and loss variants landing within 0.016 weighted F1. More preprocessing was not automatically better.
Rigor Before Results
A separate intended full-fine-tune condition in the depth ablation failed a post-hoc code review: the backbone had not actually been unfrozen. I marked that condition invalid and excluded it from interpretation. I also do not present these metrics as an untouched final test estimate. The canonical FER-2013 test split was consulted repeatedly, so the results are exploratory comparisons. Phases 1–2 used all 28,709 training images, while Phase 3 and the ablations used a 3,436-image class-capped subset; cross-phase scores are therefore not directly comparable. Class imbalance also made accuracy insufficient. Macro F1 and per-class recall were tracked, and disgust recall remained near zero in the strongest scratch CNN. Because FER-2013 lacks demographic controls, minority-class errors may intersect with demographic underrepresentation. Grad-CAM is qualitative evidence of attention, not evidence of equal reliability across groups.