23 Languages Enter Agent Evaluation
Human judgment Moves from Translation to Task Acceptance
This week scanned 86 HF orgs · 50 GitHub orgs · 71 blogs · 125 X accounts
Qwen released BabelArena [P0], covering 23 languages and 16,146 task instances; NVIDIA released a dataset of evaluation rules for system and user instruction hierarchies [P0]; Meta released 800 frozen memory histories, while a paper collected approximately 1,000 memory preference judgments [P1]. Strongest data demand signal this week: multilingual Agent tasks and judgment.
Key Findings
This week's 5 high commercial value findings
On 2026-10-22, Qwen released `Qwen/BabelArena`, covering 23 languages, 13 domains, and 4 types of Agent benchmarks, with 702 standard tasks and 16,146 task instances; it recorded 9,637 scanned downloads and 9 likes. Its localization scope includes not only prompts, but also components encountered by Agents, such as tasks and tool descriptions. Published on 2026-10-21, the paper "Task-Preserving Multilingual Instruction Tuning for Open-Ended QA in WMT 2026 MIST" likewise studies multilingual localization that preserves task semantics.
On 2026-10-22, NVIDIA released `nvidia/Nemotron-System-User-Instruction-Hierarchy-v1`, recording 787 scanned downloads and 7 likes. It contains model-generated evaluation rules for determining whether responses follow the instruction hierarchy between system and user messages. On 2026-10-20, Allen AI released `allenai/novelty-judge-bench`, with 512 downloads and 4 likes, addressing the instability of LLM novelty evaluation. Evaluation-related entries in the scan increased from 1 to 6 compared with the previous issue, but this reflects a change in the composition of scan results and cannot be directly interpreted as fivefold market demand growth.
On 2026-10-20, Meta released `facebook/MemProBench`, providing 800 frozen memory histories, with 867 scanned downloads and 6 likes. Published on 2026-10-22, the paper "Memory-Aware Reward Modeling: Learning Human Preferences for LLM Memory Use" collected approximately 1,000 pairwise judgments from 87 participants to study whether memory use genuinely improves user experience. The paper "Abstract Priorities, Concrete Rewards: How Stated Preferences Relate to What People Reward in AI Responses," also published that day, uses 1,396 participants from the PRISM dataset to compare stated preferences with actual reward preferences.
On 2026-10-19, NVIDIA released `nvidia/Nemotron-Personas-Germany` and `nvidia/Nemotron-Personas-UK`, receiving 3,077 and 2,144 downloads and 53 and 34 likes, respectively. Both generate synthetic personas based on real-world population distributions. On the same day, Meta updated object_pose/v2 for `facebook/show3d-dataset`, covering 1,176 training recordings and 390 test recordings, and released the test portion of hand_pose/v3 and interaction_field/v1. The dataset recorded 142,553 scanned downloads and 14 likes and is licensed under CC BY-NC 4.0.
On 2026-10-18, Anthropic released `Anthropic/scorpion-toxin-binder-design`, recording 6,217 scanned downloads and 43 likes under the CC BY 4.0 license. The materials describe Claude designing binding proteins for 5 scorpion toxins. Adaptyv Bio and Twist Bioscience, two contract research organizations, measured binding through surface plasmon resonance experiments and reported 17 designs as binding proteins.
Demand Signals
Infer training data demands from model releases
Want to discuss this issue?
Auto-generated by AI Dataset Radar · Updated weekly
AI Dataset Radar →