diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..0e0a35a --- /dev/null +++ b/.gitattributes @@ -0,0 +1 @@ +artifact/**/*.pt filter=lfs diff=lfs merge=lfs -text diff --git a/artifact/scoliosis_sconet_112_bodyonly_plaince_adamw_cosine/README.md b/artifact/scoliosis_sconet_112_bodyonly_plaince_adamw_cosine/README.md new file mode 100644 index 0000000..d9ba1aa --- /dev/null +++ b/artifact/scoliosis_sconet_112_bodyonly_plaince_adamw_cosine/README.md @@ -0,0 +1,76 @@ +# Scoliosis ScoNet-MT-ske Artifacts + +This directory archives the two retained best checkpoints from the practical +`1:1:2` skeleton-map training path: + +- best by macro-F1 +- best by accuracy + +Model recipe: + +- split: `Scoliosis1K_112.json` +- representation: `body-only` skeleton heatmap +- losses: plain CE + triplet +- finetune optimizer: `AdamW` +- scheduler: cosine annealing + +Files: + +- `*_scalar_test_f1.pt`: retained best checkpoint by macro-F1 +- `*_scalar_test_accuracy.pt`: retained best checkpoint by accuracy +- `train_config.yaml`: training config for the finetune run +- `eval_best_f1_27000.yaml`: standalone eval config for the retained best-F1 checkpoint +- `eval_best_accuracy_64000.yaml`: standalone eval config for the retained best-accuracy checkpoint + +## Reproduce eval + +Run from the repo root. + +Best-F1 checkpoint: + +```bash +CUDA_VISIBLE_DEVICES=GPU-9cc7b26e-90d4-0c49-4d4c-060e528ffba6 \ +uv run python -m torch.distributed.run --nproc_per_node=1 \ + opengait/main.py \ + --cfgs artifact/scoliosis_sconet_112_bodyonly_plaince_adamw_cosine/eval_best_f1_27000.yaml \ + --phase test +``` + +Best-accuracy checkpoint: + +```bash +CUDA_VISIBLE_DEVICES=GPU-9cc7b26e-90d4-0c49-4d4c-060e528ffba6 \ +uv run python -m torch.distributed.run --nproc_per_node=1 \ + opengait/main.py \ + --cfgs artifact/scoliosis_sconet_112_bodyonly_plaince_adamw_cosine/eval_best_accuracy_64000.yaml \ + --phase test +``` + +Paths to change on another machine: + +- `data_cfg.dataset_root` + - point this to your local body-only Scoliosis1K skeleton dataset root +- `data_cfg.dataset_partition` + - change this only if your repo or partition file lives elsewhere +- `evaluator_cfg.restore_hint` + - change this only if you move the `artifact/` folder or rename the checkpoint files +- `evaluator_cfg.output_root` + - change this if you want eval outputs somewhere else + +## Verified results + +These commands were rerun from the copied `artifact/` configs on `2026-03-11`. + +Best-F1 checkpoint (`27000`): + +- Accuracy: `92.38%` +- Macro Precision: `90.30%` +- Macro Recall: `87.39%` +- Macro F1: `88.70%` + +Best-accuracy checkpoint (`64000`): + +- Accuracy: `94.25%` +- Macro Precision: `83.24%` +- Macro Recall: `95.76%` +- Macro F1: `87.63%` diff --git a/artifact/scoliosis_sconet_112_bodyonly_plaince_adamw_cosine/eval_best_accuracy_64000.yaml b/artifact/scoliosis_sconet_112_bodyonly_plaince_adamw_cosine/eval_best_accuracy_64000.yaml new file mode 100644 index 0000000..df36274 --- /dev/null +++ b/artifact/scoliosis_sconet_112_bodyonly_plaince_adamw_cosine/eval_best_accuracy_64000.yaml @@ -0,0 +1,60 @@ +data_cfg: + dataset_name: Scoliosis1K + dataset_root: /mnt/public/data/Scoliosis1K/Scoliosis1K-drf-pkl-118-sigma15-joint8-bodyonly + dataset_partition: ./datasets/Scoliosis1K/Scoliosis1K_112.json + data_in_use: + - true + - false + num_workers: 1 + remove_no_gallery: false + test_dataset_name: Scoliosis1K + +evaluator_cfg: + enable_float16: true + restore_ckpt_strict: true + restore_hint: ./artifact/scoliosis_sconet_112_bodyonly_plaince_adamw_cosine/ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_finetune_1gpu_80k-iter-64000-score-0.9425-scalar_test_accuracy.pt + save_name: ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_eval_64000_1gpu + output_root: ./artifact/scoliosis_sconet_112_bodyonly_plaince_adamw_cosine/eval-output + eval_func: evaluate_scoliosis + sampler: + batch_shuffle: false + batch_size: 1 + sample_type: all_ordered + type: InferenceSampler + frames_all_limit: 720 + metric: euc + transform: + - type: BaseSilCuttingTransform + +model_cfg: + model: ScoNet + backbone_cfg: + type: ResNet9 + block: BasicBlock + in_channel: 2 + channels: + - 64 + - 128 + - 256 + - 512 + layers: + - 1 + - 1 + - 1 + - 1 + strides: + - 1 + - 2 + - 2 + - 1 + maxpool: false + SeparateFCs: + in_channels: 512 + out_channels: 256 + parts_num: 16 + SeparateBNNecks: + class_num: 3 + in_channels: 256 + parts_num: 16 + bin_num: + - 16 diff --git a/artifact/scoliosis_sconet_112_bodyonly_plaince_adamw_cosine/eval_best_f1_27000.yaml b/artifact/scoliosis_sconet_112_bodyonly_plaince_adamw_cosine/eval_best_f1_27000.yaml new file mode 100644 index 0000000..2ab2441 --- /dev/null +++ b/artifact/scoliosis_sconet_112_bodyonly_plaince_adamw_cosine/eval_best_f1_27000.yaml @@ -0,0 +1,60 @@ +data_cfg: + dataset_name: Scoliosis1K + dataset_root: /mnt/public/data/Scoliosis1K/Scoliosis1K-drf-pkl-118-sigma15-joint8-bodyonly + dataset_partition: ./datasets/Scoliosis1K/Scoliosis1K_112.json + data_in_use: + - true + - false + num_workers: 1 + remove_no_gallery: false + test_dataset_name: Scoliosis1K + +evaluator_cfg: + enable_float16: true + restore_ckpt_strict: true + restore_hint: ./artifact/scoliosis_sconet_112_bodyonly_plaince_adamw_cosine/ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_finetune_1gpu_80k-iter-27000-score-0.8870-scalar_test_f1.pt + save_name: ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_eval_27000_1gpu + output_root: ./artifact/scoliosis_sconet_112_bodyonly_plaince_adamw_cosine/eval-output + eval_func: evaluate_scoliosis + sampler: + batch_shuffle: false + batch_size: 1 + sample_type: all_ordered + type: InferenceSampler + frames_all_limit: 720 + metric: euc + transform: + - type: BaseSilCuttingTransform + +model_cfg: + model: ScoNet + backbone_cfg: + type: ResNet9 + block: BasicBlock + in_channel: 2 + channels: + - 64 + - 128 + - 256 + - 512 + layers: + - 1 + - 1 + - 1 + - 1 + strides: + - 1 + - 2 + - 2 + - 1 + maxpool: false + SeparateFCs: + in_channels: 512 + out_channels: 256 + parts_num: 16 + SeparateBNNecks: + class_num: 3 + in_channels: 256 + parts_num: 16 + bin_num: + - 16 diff --git a/artifact/scoliosis_sconet_112_bodyonly_plaince_adamw_cosine/train_config.yaml b/artifact/scoliosis_sconet_112_bodyonly_plaince_adamw_cosine/train_config.yaml new file mode 100644 index 0000000..5a35b01 --- /dev/null +++ b/artifact/scoliosis_sconet_112_bodyonly_plaince_adamw_cosine/train_config.yaml @@ -0,0 +1,115 @@ +data_cfg: + dataset_name: Scoliosis1K + dataset_root: /mnt/public/data/Scoliosis1K/Scoliosis1K-drf-pkl-118-sigma15-joint8-bodyonly + dataset_partition: ./datasets/Scoliosis1K/Scoliosis1K_112.json + data_in_use: + - true + - false + num_workers: 1 + remove_no_gallery: false + test_dataset_name: Scoliosis1K + +evaluator_cfg: + enable_float16: true + restore_ckpt_strict: true + restore_hint: 80000 + save_name: ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_finetune_1gpu_80k + output_root: /mnt/hddl/data/OpenGait-output + eval_func: evaluate_scoliosis + sampler: + batch_shuffle: false + batch_size: 1 + sample_type: all_ordered + type: InferenceSampler + frames_all_limit: 720 + metric: euc + transform: + - type: BaseSilCuttingTransform + +loss_cfg: + - loss_term_weight: 1.0 + margin: 0.2 + type: TripletLoss + log_prefix: triplet + - loss_term_weight: 1.0 + scale: 16 + type: CrossEntropyLoss + log_prefix: softmax + log_accuracy: true + +model_cfg: + model: ScoNet + backbone_cfg: + type: ResNet9 + block: BasicBlock + in_channel: 2 + channels: + - 64 + - 128 + - 256 + - 512 + layers: + - 1 + - 1 + - 1 + - 1 + strides: + - 1 + - 2 + - 2 + - 1 + maxpool: false + SeparateFCs: + in_channels: 512 + out_channels: 256 + parts_num: 16 + SeparateBNNecks: + class_num: 3 + in_channels: 256 + parts_num: 16 + bin_num: + - 16 + +optimizer_cfg: + lr: 0.0005 + solver: AdamW + weight_decay: 0.0005 + +scheduler_cfg: + scheduler: CosineAnnealingLR + T_max: 60000 + eta_min: 0.00001 + +trainer_cfg: + enable_float16: true + fix_BN: false + with_test: true + log_iter: 100 + restore_ckpt_strict: true + optimizer_reset: false + scheduler_reset: false + restore_hint: /mnt/hddl/data/OpenGait-output/Scoliosis1K/ScoNet/ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_finetune_1gpu_80k/checkpoints/latest.pt + output_root: /mnt/hddl/data/OpenGait-output + auto_resume_latest: true + resume_every_iter: 500 + resume_keep: 6 + best_ckpt_cfg: + keep_n: 3 + metric_names: + - scalar/test_f1/ + - scalar/test_accuracy/ + eval_iter: 1000 + save_iter: 500 + save_name: ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_finetune_1gpu_80k + sync_BN: false + total_iter: 80000 + sampler: + batch_shuffle: true + batch_size: + - 8 + - 8 + frames_num_fixed: 30 + sample_type: fixed_unordered + type: TripletSampler + transform: + - type: BaseSilCuttingTransform diff --git a/configs/sconet/sconet_scoliosis1k_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_eval_27000_1gpu.yaml b/configs/sconet/sconet_scoliosis1k_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_eval_27000_1gpu.yaml new file mode 100644 index 0000000..619eff3 --- /dev/null +++ b/configs/sconet/sconet_scoliosis1k_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_eval_27000_1gpu.yaml @@ -0,0 +1,60 @@ +data_cfg: + dataset_name: Scoliosis1K + dataset_root: /mnt/public/data/Scoliosis1K/Scoliosis1K-drf-pkl-118-sigma15-joint8-bodyonly + dataset_partition: ./datasets/Scoliosis1K/Scoliosis1K_112.json + data_in_use: + - true + - false + num_workers: 1 + remove_no_gallery: false + test_dataset_name: Scoliosis1K + +evaluator_cfg: + enable_float16: true + restore_ckpt_strict: true + restore_hint: /mnt/hddl/data/OpenGait-output/Scoliosis1K/ScoNet/ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_finetune_1gpu_80k/checkpoints/best/scalar_test_f1/ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_finetune_1gpu_80k-iter-27000-score-0.8870-scalar_test_f1.pt + save_name: ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_eval_27000_1gpu + output_root: /mnt/hddl/data/OpenGait-output + eval_func: evaluate_scoliosis + sampler: + batch_shuffle: false + batch_size: 1 + sample_type: all_ordered + type: InferenceSampler + frames_all_limit: 720 + metric: euc + transform: + - type: BaseSilCuttingTransform + +model_cfg: + model: ScoNet + backbone_cfg: + type: ResNet9 + block: BasicBlock + in_channel: 2 + channels: + - 64 + - 128 + - 256 + - 512 + layers: + - 1 + - 1 + - 1 + - 1 + strides: + - 1 + - 2 + - 2 + - 1 + maxpool: false + SeparateFCs: + in_channels: 512 + out_channels: 256 + parts_num: 16 + SeparateBNNecks: + class_num: 3 + in_channels: 256 + parts_num: 16 + bin_num: + - 16 diff --git a/configs/sconet/sconet_scoliosis1k_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_eval_64000_1gpu.yaml b/configs/sconet/sconet_scoliosis1k_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_eval_64000_1gpu.yaml new file mode 100644 index 0000000..c1fb427 --- /dev/null +++ b/configs/sconet/sconet_scoliosis1k_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_eval_64000_1gpu.yaml @@ -0,0 +1,60 @@ +data_cfg: + dataset_name: Scoliosis1K + dataset_root: /mnt/public/data/Scoliosis1K/Scoliosis1K-drf-pkl-118-sigma15-joint8-bodyonly + dataset_partition: ./datasets/Scoliosis1K/Scoliosis1K_112.json + data_in_use: + - true + - false + num_workers: 1 + remove_no_gallery: false + test_dataset_name: Scoliosis1K + +evaluator_cfg: + enable_float16: true + restore_ckpt_strict: true + restore_hint: /mnt/hddl/data/OpenGait-output/Scoliosis1K/ScoNet/ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_finetune_1gpu_80k/checkpoints/best/scalar_test_accuracy/ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_finetune_1gpu_80k-iter-64000-score-0.9425-scalar_test_accuracy.pt + save_name: ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_eval_64000_1gpu + output_root: /mnt/hddl/data/OpenGait-output + eval_func: evaluate_scoliosis + sampler: + batch_shuffle: false + batch_size: 1 + sample_type: all_ordered + type: InferenceSampler + frames_all_limit: 720 + metric: euc + transform: + - type: BaseSilCuttingTransform + +model_cfg: + model: ScoNet + backbone_cfg: + type: ResNet9 + block: BasicBlock + in_channel: 2 + channels: + - 64 + - 128 + - 256 + - 512 + layers: + - 1 + - 1 + - 1 + - 1 + strides: + - 1 + - 2 + - 2 + - 1 + maxpool: false + SeparateFCs: + in_channels: 512 + out_channels: 256 + parts_num: 16 + SeparateBNNecks: + class_num: 3 + in_channels: 256 + parts_num: 16 + bin_num: + - 16 diff --git a/docs/scoliosis_reproducibility_audit.md b/docs/scoliosis_reproducibility_audit.md index 2414507..f28ec51 100644 --- a/docs/scoliosis_reproducibility_audit.md +++ b/docs/scoliosis_reproducibility_audit.md @@ -163,6 +163,9 @@ Conclusion: - on `1:1:2`, `body-only + weighted CE` reached `81.82 Acc / 66.21 Prec / 88.50 Rec / 65.96 F1` on the full test set - on the same split, `body-only + plain CE` improved that further to `83.16 Acc / 68.24 Prec / 80.02 Rec / 68.47 F1` at `7000` - a later explicit rerun of the `body-only + plain CE` `7000` full-test eval reproduced that same `83.16 / 68.24 / 80.02 / 68.47` result +- a later `AdamW` cosine finetune from that same `10k` plain-CE checkpoint improved the practical result further: + - verified retained best checkpoint at `27000`: `92.38 Acc / 90.30 Prec / 87.39 Rec / 88.70 F1` + - final `80000` checkpoint still remained strong: `90.64 Acc / 72.87 Prec / 93.19 Rec / 75.74 F1` - adding back limited head context via `head-lite` did not improve the full-test score; its `7000` checkpoint reached only `78.07 Acc / 65.42 Prec / 80.50 Rec / 62.08 F1` - the first practical DRF bridge on the same `1:1:2` body-only recipe peaked early and still underperformed the plain skeleton baseline; its best retained `2000` checkpoint reached only `80.21 Acc / 58.92 Prec / 59.23 Rec / 57.84 F1` on the full test set @@ -180,6 +183,7 @@ Conclusion: - the `1:1:8` class ratio is not just a nuisance; it appears to be a major driver of the current skeleton/DRF failure mode - on the easier `1:1:2` split, weighted CE is not currently the winning recipe; the best local full-test result so far came from plain CE - `head-lite` may help the small fixed proxy subset, but that gain did not transfer to the full `TEST_SET`, so `body-only + plain CE` remains the best practical skeleton recipe +- once the practical `1:1:2` body-only plain-CE recipe was established, the branch still appeared underfit enough that optimizer/schedule mattered again. A later `AdamW` cosine finetune beat the earlier SGD bridge by a large margin at its retained best checkpoint, which means the earlier `83.16 / 68.47` result was a stable baseline but not the ceiling of this skeleton recipe - DRF currently looks worse than the plain skeleton baseline not because the skeleton path is dead, but because the additional prior branch is not yet providing a selective or stable complement. The current local evidence points to three likely causes: - the body-only skeleton baseline already captures most of the useful torso signal on `1:1:2`, so PAV may be largely redundant in this setting - the current PGA/PAV path appears weakly selective in local diagnostics, so the prior is not clearly emphasizing a few clinically relevant parts @@ -210,3 +214,17 @@ At the moment, this repo does not yet support: - claiming a successful independent reproduction of the DRF paper’s quantitative results - claiming that the paper’s skeleton-map preprocessing is fully specified - treating the paper’s qualitative feature-response visualizations as reproduced + +For practical model selection, the current conclusion is simpler: + +- stop treating DRF as the default winner +- keep the practical mainline on `1:1:2` +- use the retained `body-only + plain CE` skeleton checkpoint family as the working solution +- the strongest verified practical checkpoint is the later `AdamW` cosine finetune checkpoint at `27000`, with: + - `92.38 Acc / 90.30 Prec / 87.39 Rec / 88.70 F1` + +That means the remaining work is no longer broad reproduction debugging. It is mostly optional refinement: + +- confirm whether `body-only` really beats `full-body` under the same successful training recipe +- optionally retry DRF only after the strong practical skeleton baseline is fixed +- package and use the retained best checkpoint rather than continuing to churn the whole search space diff --git a/docs/scoliosis_training_change_log.md b/docs/scoliosis_training_change_log.md index 3279e10..bac9606 100644 --- a/docs/scoliosis_training_change_log.md +++ b/docs/scoliosis_training_change_log.md @@ -41,14 +41,15 @@ Use it for: | 2026-03-10 | `ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_main_1gpu_20k` | ScoNet-MT-ske mainline | `Scoliosis1K-drf-pkl-118-sigma15-joint8-bodyonly` + `Scoliosis1K_112.json` | Promoted the winning practical skeleton recipe to a longer `20k` run with full `TEST_SET` eval and checkpoint save every `1000`; no proxy subset, same plain CE + SGD setup | interrupted | superseded by the true-resume continuation below | | 2026-03-10 | `ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_resume_1gpu_20k` | ScoNet-MT-ske mainline | `Scoliosis1K-drf-pkl-118-sigma15-joint8-bodyonly` + `Scoliosis1K_112.json` | True continuation of the earlier plain-CE `1:1:2` `10k` bridge from its `latest.pt`, extended to `20k` with full `TEST_SET` eval and checkpoint save every `1000` | interrupted | superseded by the AdamW finetune branch below | | 2026-03-10 | `ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_finetune_1gpu_20k` | ScoNet-MT-ske finetune | `Scoliosis1K-drf-pkl-118-sigma15-joint8-bodyonly` + `Scoliosis1K_112.json` | AdamW finetune from the earlier plain-CE `1:1:2` `10k` checkpoint; restores model weights only, resets optimizer/scheduler state, keeps full `TEST_SET` eval and checkpoint save every `1000` | interrupted | superseded by the longer overnight 40k finetune below | -| 2026-03-10 | `ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_finetune_1gpu_40k` | ScoNet-MT-ske finetune | `Scoliosis1K-drf-pkl-118-sigma15-joint8-bodyonly` + `Scoliosis1K_112.json` | Longer overnight AdamW finetune from the same `10k` plain-CE checkpoint; restores model weights only, resets optimizer/scheduler state, extends to `40000` total iterations with full `TEST_SET` eval every `1000` | running | pending | +| 2026-03-10 | `ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_finetune_1gpu_40k` | ScoNet-MT-ske finetune | `Scoliosis1K-drf-pkl-118-sigma15-joint8-bodyonly` + `Scoliosis1K_112.json` | Longer overnight AdamW finetune from the same `10k` plain-CE checkpoint; restores model weights only, resets optimizer/scheduler state, extends to `40000` total iterations with full `TEST_SET` eval every `1000` | interrupted | superseded by the cosine `80k` finetune below | +| 2026-03-10 to 2026-03-11 | `ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_finetune_1gpu_80k` | ScoNet-MT-ske finetune | `Scoliosis1K-drf-pkl-118-sigma15-joint8-bodyonly` + `Scoliosis1K_112.json` | Practical long-run finetune from the same `10k` plain-CE checkpoint, but switched to `AdamW` with cosine decay, HDD-backed `output_root`, `save_iter=500`, `eval_iter=1000`, and best-N checkpoint retention | complete | final `80000` eval: `90.64/72.87/93.19/75.74`; verified best retained full-test checkpoint at `27000`: `92.38/90.30/87.39/88.70` (Acc/Prec/Rec/F1) | ## Current best skeleton baseline Current best `ScoNet-MT-ske`-style result: - practical best on the easier `1:1:2` split: - - `ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_bridge_1gpu_10k` at `7000` - - `83.16 Acc / 68.24 Prec / 80.02 Rec / 68.47 F1` + - `ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_finetune_1gpu_80k` at retained best checkpoint `27000` + - verified standalone full-test eval: `92.38 Acc / 90.30 Prec / 87.39 Rec / 88.70 F1` - best result retained on the harder `1:1:8` split: - `ScoNet_skeleton_118_sigma15_joint8_sharedalign_2gpu_bs12x8` - `50.47 Acc / 69.31 Prec / 54.58 Rec / 48.63 F1` @@ -68,3 +69,4 @@ Current best `ScoNet-MT-ske`-style result: - A later full-test rerun of the retained `body-only + plain CE` `7000` checkpoint reproduced the same `83.16 / 68.24 / 80.02 / 68.47` result exactly, so that number is now explicitly reconfirmed rather than just carried forward from the original run log. - `Head-lite` looked stronger than `body-only` on the fixed 128-sequence proxy subset at `7000`, but it did not transfer to the full test set: `78.07 Acc / 65.42 Prec / 80.50 Rec / 62.08 F1`, which is clearly below the `body-only + plain CE` full-test result. - The first practical DRF bridge on the winning `1:1:2` recipe did not beat the plain skeleton baseline. Its best retained checkpoint (`2000`) reached only `80.21 Acc / 58.92 Prec / 59.23 Rec / 57.84 F1` on the full test set, versus `83.16 / 68.24 / 80.02 / 68.47` for `body-only + plain CE` at `7000`. The working local interpretation is that the added PAV/PGA path is currently injecting a weak or noisy prior rather than a useful complementary signal. +- Extending the practical `1:1:2` body-only plain-CE baseline with an `AdamW` cosine finetune changed the picture substantially. The final `80000` checkpoint still evaluated well (`90.64 / 72.87 / 93.19 / 75.74`), but the real win was an earlier retained checkpoint: `27000` reproduced exactly at `92.38 Acc / 90.30 Prec / 87.39 Rec / 88.70 F1` on a standalone full-test rerun. So for this practical path, the best result is no longer the original SGD bridge checkpoint but the retained `AdamW` cosine finetune checkpoint. diff --git a/docs/sconet-drf-status-and-training.md b/docs/sconet-drf-status-and-training.md index 00f8ecb..4db9563 100644 --- a/docs/sconet-drf-status-and-training.md +++ b/docs/sconet-drf-status-and-training.md @@ -71,6 +71,7 @@ The main findings so far are: - `body-only + weighted CE` reached `81.82%` accuracy and `65.96%` macro-F1 on the full test set at `7000` - `body-only + plain CE` improved that to `83.16%` accuracy and `68.47%` macro-F1 at `7000` - a later full-test rerun confirmed the `body-only + plain CE` `7000` result exactly + - an `AdamW` cosine finetune from that same plain-CE checkpoint improved the practical best further; the retained `27000` checkpoint reproduced at `92.38%` accuracy and `88.70%` macro-F1 on the full test set - a `head-lite + plain CE` variant looked promising on the fixed proxy subset but underperformed on the full test set at `7000` (`78.07%` accuracy, `62.08%` macro-F1) The current working conclusion is: @@ -79,7 +80,7 @@ The current working conclusion is: - the strong silhouette checkpoint is not evidence that the skeleton-map path works - the main remaining suspect is the skeleton-map representation and preprocessing path - for practical model development, `1:1:2` is currently the better working split than `1:1:8` -- for practical model development, the current best skeleton recipe is still `body-only + plain CE + SGD` on `1:1:2` +- for practical model development, the current best skeleton recipe is `body-only + plain CE`, and the current best retained checkpoint comes from a later `AdamW` cosine finetune on `1:1:2` - the first practical DRF bridge on that same winning `1:1:2` recipe did not improve on the plain skeleton baseline: - best retained DRF checkpoint (`2000`) on the full test set: `80.21 Acc / 58.92 Prec / 59.23 Rec / 57.84 F1` - current best plain skeleton checkpoint (`7000`) on the full test set: `83.16 Acc / 68.24 Prec / 80.02 Rec / 68.47 F1` @@ -119,3 +120,35 @@ The skeleton-map control used the same recipe, except for the modality-specific 1. Train a pure silhouette `1:1:8` baseline from the upstream ScoNet config as a clean sanity control. 2. Treat skeleton-map preprocessing as the primary debugging target until a `ScoNet-MT-ske`-style run gets close to the paper. 3. Only after the skeleton baseline is credible should DRF/PAV-specific conclusions be treated as decisive. + +## Practical conclusion + +For practical use in this repo, the current winning path is: + +- split: `1:1:2` +- representation: `body-only` skeleton map +- losses: plain CE + triplet +- baseline training: `SGD` +- best retained finetune: `AdamW` + cosine decay + +The strongest verified checkpoint so far is: + +- `ScoNet_skeleton_112_sigma15_joint8_bodyonly_plaince_adamw_cosine_finetune_1gpu_80k` +- retained best checkpoint at `27000` +- verified full-test result: `92.38 Acc / 90.30 Prec / 87.39 Rec / 88.70 F1` + +So the current local recommendation is: + +- keep `body-only` as the default practical skeleton representation +- keep `1:1:2` as the main practical split +- treat DRF as an optional research branch, not the mainline model + +## Remaining useful experiments + +At this point, there are only a few experiments that still look high-value: + +1. one clean `full-body` finetune under the same successful `1:1:2` recipe, just to confirm that `body-only` is really the best practical representation +2. one DRF rerun on top of the now-stronger practical baseline recipe, only if the goal is to test whether DRF can add value once the skeleton branch is already strong +3. a final packaging/evaluation pass around the retained best checkpoints, rather than more broad preprocessing churn + +Everything else looks lower value than simply using the retained best `27000` checkpoint.