The p4 models' directions: directions-1

Pinned commits: {"phi": "fd61ea629bfc5a12c3b217e40829869c7898b0de", "p4": "04eee31281f5636b2abdd6d84809d80ecafb22fb", "store": "e5df5faaba85f0fafff1dcab6120a52d8bef429c"}

Verdicts

• D1 the JSON ranker ranks better than plain word overlap (store's labels): holds
• D2 a shorter pretraining budget ranks better than the longest: holds
• D2 some pretraining budget's .pt raises the JSON ranker on every seed: does not hold
• D4 the fable labels are more learnable than the store's (trained and scored on themselves, best budget): holds
• D4 the opus labels are more learnable than the store's (trained and scored on themselves, best budget): holds
• D4 training on the fable labels predicts the store's labels at least as well as training on them: holds
• D4 training on the opus labels predicts the store's labels at least as well as training on them: does not hold
• D4 the two teachers agree with each other more than either agrees with the store's labels: holds
• D5 the φ structure matters: not measured

Implied settings

• pool-effort: low or unnamed (the JSON ranker alone)
• pretraining budget: 1 min (best held-out ranking)
• labels: train on the teachers' labels: they are more learnable than the store's

D2 by pretraining budget (mean over seeds)

JSON ranker alone: 0.3788 (word overlap 0.3129); label-set agreement (F1): {'store~fable': 0.5322, 'store~opus': 0.5024, 'fable~opus': 0.7538}