Annotation Specialist $30,000 USD/year Pay is set based on global value, not the local market.
Most roles = hourly rate x 40 hrs x 50 weeks
Description
If accuracy matters more to you than speed, this position is a strong fit. The labels you create serve as training data for AI systems that support thousands of students daily. Precise behavioral tagging improves model intelligence. Inconsistent labeling teaches the model incorrect patterns.
LearnWith.AI develops AI-driven learning experiences through learning science, data analytics, and expert collaboration. This position converts raw student session recordings into highly accurate, rubric-based labels the team relies on. You will review recorded student sessions, pinpoint critical behavioral events, and apply rigorous classification rules to determine what occurred and when. You will also assess LLM-generated pre-annotations, correct errors, and document unusual cases to help engineers refine the system.
This is not freelance, ad-hoc annotation work. It involves a consistent workflow within one product area, featuring direct feedback channels, calibration with gold-standard examples, and advancement tied to accuracy and reliability. If you value transparent expectations, quantifiable quality standards, and contributions that directly shape model effectiveness, we should speak.
What you will be doing
- Label student session recordings by detecting, categorizing, and timestamping behavioral events according to a comprehensive rubric
- Evaluate and refine LLM pre-annotations by eliminating false positives, inserting omitted events, and sharpening timestamp accuracy
- Document clear rationale for ambiguous decisions, including rubric citations and the reasoning framework applied
- Record edge cases and clarification requests for unclear situations, and maintain an annotation log with session metadata
- Participate in calibration sessions, integrate QA feedback, and implement rubric revisions to enhance accuracy consistently
What you will NOT be doing
- Develop AI models, conduct experiments, or perform research into student behavior patterns
- Create the annotation rubric or alter category definitions based on subjective interpretation
- Prioritize speed over accuracy, consistency, or timestamp exactness
- Handle sporadic, disconnected tasks across unrelated fields with no context or feedback mechanism
Key responsibilities
This position ensures that student session recordings are transformed into labeled datasets with ≥95% accuracy and precise timestamps, reliably indicating when model performance advances or declines.
Candidate requirements
- Minimum 1 year of experience in data annotation, content moderation, QA evaluation, or comparable rubric-based review roles
- Excellent English reading comprehension and capacity to adhere to detailed written instructions without rule deviation
- Capacity to maintain concentration and precision during 4–6 hours of video-focused work daily
- Skill in detecting subtle visual and on-screen behavioral signals and classifying them uniformly across multiple sessions
- Proficient written documentation abilities for articulating edge cases, reasoning, and clarification inquiries
- Dependable internet connection suitable for video streaming
- Comfort reviewing, correcting, and enhancing AI/LLM-produced annotations
Applying for a role? Here’s what to expect.
Crossover's skill assessment process combines innovative AI power with decades of human research, to take the guesswork, human bias, and pointless filters out of recruiting high-performing teams.
Chat-style
screening interview.
Cognitive 
aptitude test.
Prove real-world 
job skills.
Interview with the hiring manager.
Pass
proctored test.
Accept job offer.
Frequently asked questions
Reviews
Meet some people who've landed similar jobs
Recruitment sucks. So we’re fixing it.
The Olympics of work
It’s super hard to qualify—extreme quality standards ensure every single team member is at the top of their game.
Premium pay for premium talent
Over 50% of new hires double or triple their previous pay. Why? Because that’s what the best person in the world is worth.
Shortlist by skills, not bias
We don’t care where you went to school, what color your hair is, or whether we can pronounce your name. Just prove you’ve got the skills.
Join the world's largest community of 
AI-first remote workers.