Data Validation Scientist

The Role

You will answer one question: Does our data actually work?

We build evaluation datasets, but data quality can't be self-proclaimed.
You need to prove through experiments that models trained with our data are genuinely better.

What You'll Do

Design Validation Experiments

• Design controlled experiments to verify the impact of different datasets on model performance
• Define evaluation metrics to measure "how much the data improved the model"
• Control variables to ensure credible experimental conclusions

Run Model Training

• Use 3B/7B small models for rapid validation experiments
• Proficient in fine-tuning, SFT, DPO, and other methods
• Experienced with training frameworks like LLaMA-Factory and Axolotl

Analyze Experimental Results

• Interpret training results and assess data effectiveness
• Identify data issues (which data is useful, which is noise)
• Produce visual reports that non-technical stakeholders can understand

Feed Back into Data Iteration

• Guide data collection and labeling improvements based on experimental results
• Collaborate with Evaluation Scientists to form a "data -> validation -> improvement" loop

What We're Looking For

Required Skills

• Model training: Familiar with LLM fine-tuning workflows (SFT, DPO, RLHF concepts)
• Experiment design: Understands variable control, control groups, and statistical significance
• Result analysis: Can draw reliable conclusions from experimental data
• Tool proficiency: PyTorch, Transformers, and common training frameworks

Required Mindset

• Genuinely curious about how data affects models
• Pursues experimental rigor — not satisfied with "it ran, so it's done"
• Willing to communicate with non-technical teams and explain experimental findings

Nice to Have

• Experience with RLHF or human feedback
• Published relevant papers
• Experience in data quality assessment
• Familiarity with reward model training

Not a Fit If You

• Only tune hyperparameters without understanding the "why"
• Only care about models and not about data
• Cannot explain results to non-technical people