Fusing pretrained person and appearance features for dense classroom behavior recognition.
This repository provides the official implementation of RAFNet, associated with our manuscript submitted to The Visual Computer.
RAFNet combines frozen GroupRec person embeddings and ConvNeXt appearance features
through adaptive gated fusion. The complete model adds spatial aggregation over
students in the same image. The base variant without Spatial is called Gated.
The implementation lives in prism/, the project's earlier development name.
NCST primary F4 is Macro F1 over the fixed four target behaviors after excluding true Null instances. Predictions of Null on target instances still count as errors. Values below are mean ± population SD over seeds 41–45, except the single-seed EduYOLO adaptation.
| Model | Encoder training | NCST F4 (%) | NCST F5 (%) |
|---|---|---|---|
| ConvNeXt-only | Frozen | 60.16 ± 0.81 | 47.13 ± 0.66 |
| Gated | Frozen | 62.73 ± 0.96 | 49.19 ± 0.82 |
| RAFNet (Gated + Spatial) | Frozen | 63.35 ± 1.42 | 49.56 ± 1.11 |
| ViT-B/16 | Fine-tuned | 63.42 ± 0.87 | 50.76 ± 0.97 |
| EduYOLO adapted | Trained from scratch | 50.51 | 41.75 |
RAFNet minus ConvNeXt has a mean F4 difference of 3.19 percentage points and a paired classroom–date bootstrap 95% interval of [−0.05, 7.27]. Spatial minus Gated has a mean difference of 0.62 points and interval [−1.05, 2.47].
On SCB3-U, RAFNet is trained on the dataset's training split and achieves 98.04 ± 0.08 five-class Macro F1 on its test split.
See paper reproduction for the three main training configurations, feature layout, and metric definitions. They use the existing training driver:
python prism/scripts/train.py --model paper/rafnet --seeds 41,42,43,44,45 --gpu 0
python prism/scripts/evaluate_paper.py --model rafnet
python prism/scripts/summarize_paper.py --predictions results/paper_predictionsThe evaluation command exports predictions from saved checkpoints; the summary
command writes the main and per-class tables from saved labels and logits.
Evaluation uses cached features, and table aggregation uses saved predictions.
Ablation configurations are also under prism/configs/paper/.
The experiments use Python 3.12 and PyTorch 2.8 with CUDA 12.8. Download features, checkpoints, and predictions from the v2 release. Setup instructions, download commands, and asset coverage are listed in paper reproduction.
prism/models/: fusion classifiers and within-image spatial aggregation.prism/configs/paper/: paper training and ablation configurations.prism/scripts/train.py: training and validation-based checkpoint selection.prism/scripts/evaluate_paper.py: checkpoint inference on cached features.prism/scripts/summarize_paper.py: main and class-wise F1 summaries.demo/: face-blurred examples with cached features and checkpoints.model/,yolox/, and related directories: GroupRec feature-extraction stack.
Raw NCST imagery is not publicly distributed. Demo features were extracted from original images, while demo images were blurred for display. Third-party pretrained weights and SMPL assets are subject to their respective licenses and access terms. Instructions for the original Pixi workflow are in v1 reproduction.
Code uses the MIT license. Third-party data and weights retain their own terms. This work builds on GroupRec, HRNet, ConvNeXt, and the upstream pose and detection tooling.