Radar Brief Week 37, 2026 · 2026-11-07 — 2026-11-14

Which Tasks Are Worth Testing
Which Samples Are Worth a Human Look

This week scanned 86 HF orgs · 50 GitHub orgs · 71 blogs · 125 X accounts

0
Valuable Datasets
0
Related Papers
0
Blog Posts
0
Active Repos
One-line Summary

Papers published November 11–12 examine adaptive human auditing and fixed-budget preference data, respectively [P0]; Surge AI updates three specialized agent evaluation benchmarks on November 10 [P0]; NVIDIA adds 12,512 stair motion sequences to GRAIL on November 10 [P1]. This week’s strongest data demand signal: specialized multimodal tasks and scoring criteria.

Key Findings

This week's 5 high commercial value findings

P0 Papers published November 11–12 examine adaptive human auditing and fixed-budget preference data, respectively [P0]

Published on 2026-11-12, “Human or Synthetic? What a Fixed Budget Buys for Post-Training” shifts the comparison of human and synthetic preference data from fixed sample counts to SFT and DPO configurations under a fixed budget. Published on 2026-11-11, “Anytime-Valid Empirical Likelihood Inference from LLM Annotations with Adaptive Human Auditing” introduces the AAELI framework, incorporating adaptive human auditing to address bias and calibration issues in model outputs relative to human judgment. The available abstracts do not provide sufficient experimental results to establish that either data approach is superior across the board.

Business implications → Together, these studies point to a testable opportunity: customers need not just more feedback, but clarity on which samples warrant human judgment under a limited budget. Knowlyr can offer a service combining model preprocessing, uncertain-sample selection, paid human judgment, and error auditing, demonstrating value through cost, error detection rates, and downstream performance rather than broadly claiming that human data always outperforms synthetic data.
P0 Surge AI updates three specialized agent evaluation benchmarks on November 10 [P0]

On 2026-11-10, the scan recorded 60, 28, and 489 downloads for Surge AI’s `surgeai/Handbook-v1.1`, `surgeai/ComplexConstraints-v1.1`, and `surgeai/Chartography-harbor`, respectively. Handbook targets long-context enterprise tasks involving company tools and MCP servers across five enterprise domains. ComplexConstraints contains 75 expert-designed prompts, each paired with 10–40 scoring criteria. Chartography contains 100 real-world prompts and specialized charting tasks involving Sankey diagrams, candlestick charts, contour plots, Bode plots, and more.

Business implications → Competitors are codifying expert judgment into tasks, scoring criteria, and evaluation environments rather than simply delivering data. Promising entry points for Knowlyr include multi-constraint instruction review, specialized chart reasoning, and enterprise process compliance judgment. The value of human judgment lies in defining what is truly correct for the business, explaining constraint conflicts, and adjudicating edge cases. Download counts do not yet demonstrate procurement at scale, but competition in productization is already clear.
P1 NVIDIA adds 12,512 stair motion sequences to GRAIL on November 10 [P1]

The data card for `nvidia/PhysicalAI-Robotics-Locomanipulation-GRAIL` explicitly records the addition of `stair_p3` and `stair_p4` on 2026-11-10, containing 12,512 stair ascent and descent motion sequences across 2,000 new procedurally generated stair assets, with sequence lengths of 5–12 seconds. The snapshot shows 48,453 downloads and 51 likes. Predating the reporting window, `nvidia/G1-Factory-Benchmark` (2026-10-20, 269 downloads) contains 900 human teleoperation demonstrations across nine industrial tasks; `nvidia/PhysicalAI-Human-1M-100K` (2026-11-04, 8,244 downloads) provides first-person video, language descriptions, and head and hand poses.

Business implications → These materials show that procedurally generated environments and human behavioral demonstrations are parallel approaches, not simple substitutes. Rather than competing directly on simulation generation scale, Knowlyr should pilot motion failure attribution, task completion judgment, contact risk review, and first-person action intent judgment. Growth in synthetic motion volume does not automatically establish real-world deployment safety; human judgment is needed to supplement task and risk criteria. GRAIL’s CC BY-NC 4.0 noncommercial license restrictions also require attention.
P1 Google releases a multimodal knowledge agent benchmark with 10 showcase examples on November 9 [P1]

`google/multimodal-knowledge-agent` is dated 2026-11-09, with 2,239 downloads and 25 likes in the snapshot. It provides 10 showcase examples evaluating complex, multi-step, real-world visual knowledge work in specialized domains, spanning document, geospatial, and image modalities. As context predating the reporting window, `google/physical-engineerings-last-exam` is dated 2026-10-18, with 42,331 downloads and 50 likes; its long-horizon engineering tasks were created by licensed professional engineers across four disciplines. On GitHub, `allenai/olmocr` has 19,822 stars and `google-deepmind/science-skills` has 3,472 stars, both reflecting existing snapshot counts.

Business implications → Demand is expanding from extracting document text to completing specialized tasks using visual evidence, but 10 examples do not yet constitute a large-scale training corpus. Potential service categories include cross-document evidence chains, joint judgment of drawings and tables, and review of specialized task processes and outcomes. Human judgment is primarily needed to identify missing evidence, detect domain-specific errors, and provide verifiable acceptance criteria; OCR accuracy alone is not a substitute.
P2 Meta releases datasets for long-term auditory personalization and synthetic stereo video on November 9 [P2]

Dated 2026-11-09, `facebook/personalized-hearing` recorded 425 downloads and 1 like. It targets long-term personalized memory evaluation for wearable hearing assistants and is explicitly restricted to benchmarking use. Released the same day, `facebook/xrstereo` recorded 201 downloads and 3 likes, providing rectified stereo video pairs and dense disparity generated by moving cameras in static scenes. Both are labeled CC BY-NC 4.0. Subsequently, `facebook/meta-encoder-v1` was released or updated on 2026-11-12, recording 163 downloads and 31 likes, and includes multimodal embeddings and video retrieval labels.

Business implications → These are early signals for long-term memory and spatial perception; current download counts are insufficient to confirm demand at scale. Knowlyr can prepare judgment task designs covering memory relevance, individual preference conflicts, incorrect memory correction, and privacy boundaries. Such services must be based on informed consent, data minimization, and the ability to withdraw consent. The noncommercial or use-restricted datasets above cannot be treated directly as commercial training assets.

Demand Signals

Infer training data demands from model releases

Data Type Intensity Trend Related Signals
Specialized multimodal tasks and scoring criteria
Very strong ↑ New
Google MKA released November 9; Surge updates three benchmarks on November 10
Multi-constraint judgment for enterprise agents
Strong ↑ New
Handbook covers five enterprise domains; ComplexConstraints contains 75 expert prompts
Robot motion and failure review
Strong ↑ New
NVIDIA GRAIL adds 12,512 motion sequences on November 10 · 2,000 stair assets
Human auditing of synthetic feedback
Moderate ↑ New
November 11 AAELI paper studies adaptive human auditing; November 12 paper compares fixed-budget preference data
Long-term memory and personalization judgment
Strong ↑ New
Meta personalized-hearing released November 9, 425 downloads
Stereo vision and spatial consistency
Strong ↑ New
Meta xrstereo released November 9, 201 downloads
Prompt expansion and preservation of image intent
Strong ↑ New
MiniMax H3-Image-PE released or updated November 8, 1,668 downloads
Multilingual capability regression evaluation
Moderate ↑ New
November 11 paper reports math gains accompanied by declining Indic question-answering performance
Coding agent trajectories and verifiers
Strong ↑ New
Terminal-Corpus has 13,866 downloads; Snorkel blog emphasizes verifier hardening
Coding agent data ↓ Dropped Present in previous issue, absent this issue
Robotics/embodied AI data ↓ Dropped Present in previous issue, absent this issue
Document OCR data ↓ Dropped Present in previous issue, absent this issue
RLHF preference data ↓ Dropped Present in previous issue, absent this issue
Multilingual speech data ↓ Dropped Present in previous issue, absent this issue
Safety/content moderation data ↓ Dropped Present in previous issue, absent this issue
Synthetic visual data ↓ Dropped Present in previous issue, absent this issue

Want to discuss this issue?

Kai
Kai Founder & CEO
苏文
苏文 AI Documentation & Release Engineer
陆明哲
陆明哲 AI Product Manager

Auto-generated by AI Dataset Radar · Updated weekly

AI Dataset Radar →