Which Tasks Are Worth Testing
Which Samples Are Worth a Human Look
This week scanned 86 HF orgs · 50 GitHub orgs · 71 blogs · 125 X accounts
Papers published November 11–12 examine adaptive human auditing and fixed-budget preference data, respectively [P0]; Surge AI updates three specialized agent evaluation benchmarks on November 10 [P0]; NVIDIA adds 12,512 stair motion sequences to GRAIL on November 10 [P1]. This week’s strongest data demand signal: specialized multimodal tasks and scoring criteria.
Key Findings
This week's 5 high commercial value findings
Published on 2026-11-12, “Human or Synthetic? What a Fixed Budget Buys for Post-Training” shifts the comparison of human and synthetic preference data from fixed sample counts to SFT and DPO configurations under a fixed budget. Published on 2026-11-11, “Anytime-Valid Empirical Likelihood Inference from LLM Annotations with Adaptive Human Auditing” introduces the AAELI framework, incorporating adaptive human auditing to address bias and calibration issues in model outputs relative to human judgment. The available abstracts do not provide sufficient experimental results to establish that either data approach is superior across the board.
On 2026-11-10, the scan recorded 60, 28, and 489 downloads for Surge AI’s `surgeai/Handbook-v1.1`, `surgeai/ComplexConstraints-v1.1`, and `surgeai/Chartography-harbor`, respectively. Handbook targets long-context enterprise tasks involving company tools and MCP servers across five enterprise domains. ComplexConstraints contains 75 expert-designed prompts, each paired with 10–40 scoring criteria. Chartography contains 100 real-world prompts and specialized charting tasks involving Sankey diagrams, candlestick charts, contour plots, Bode plots, and more.
The data card for `nvidia/PhysicalAI-Robotics-Locomanipulation-GRAIL` explicitly records the addition of `stair_p3` and `stair_p4` on 2026-11-10, containing 12,512 stair ascent and descent motion sequences across 2,000 new procedurally generated stair assets, with sequence lengths of 5–12 seconds. The snapshot shows 48,453 downloads and 51 likes. Predating the reporting window, `nvidia/G1-Factory-Benchmark` (2026-10-20, 269 downloads) contains 900 human teleoperation demonstrations across nine industrial tasks; `nvidia/PhysicalAI-Human-1M-100K` (2026-11-04, 8,244 downloads) provides first-person video, language descriptions, and head and hand poses.
`google/multimodal-knowledge-agent` is dated 2026-11-09, with 2,239 downloads and 25 likes in the snapshot. It provides 10 showcase examples evaluating complex, multi-step, real-world visual knowledge work in specialized domains, spanning document, geospatial, and image modalities. As context predating the reporting window, `google/physical-engineerings-last-exam` is dated 2026-10-18, with 42,331 downloads and 50 likes; its long-horizon engineering tasks were created by licensed professional engineers across four disciplines. On GitHub, `allenai/olmocr` has 19,822 stars and `google-deepmind/science-skills` has 3,472 stars, both reflecting existing snapshot counts.
Dated 2026-11-09, `facebook/personalized-hearing` recorded 425 downloads and 1 like. It targets long-term personalized memory evaluation for wearable hearing assistants and is explicitly restricted to benchmarking use. Released the same day, `facebook/xrstereo` recorded 201 downloads and 3 likes, providing rectified stereo video pairs and dense disparity generated by moving cameras in static scenes. Both are labeled CC BY-NC 4.0. Subsequently, `facebook/meta-encoder-v1` was released or updated on 2026-11-12, recording 163 downloads and 31 likes, and includes multimodal embeddings and video retrieval labels.
Demand Signals
Infer training data demands from model releases
Want to discuss this issue?
Auto-generated by AI Dataset Radar · Updated weekly
AI Dataset Radar →