AI Evaluator | $22.64/hr Remote

Position: Generalist – English & Arabic

Type: Hourly contract

Compensation: $22.64/hour

Location: Remote

Commitment: 10–40 hours/week

Role Responsibilities

• Evaluate LLM-generated responses for accuracy, relevance, and effectiveness across diverse topics.
• Conduct fact-checking using trusted public sources and external tools.
• Generate high-quality human evaluation data by annotating response strengths, gaps, and factual inaccuracies.
• Assess reasoning quality, clarity, tone, and completeness of AI-generated responses.
• Ensure outputs align with expected conversational behavior and system guidelines.
• Apply consistent annotations using defined taxonomies, benchmarks, and evaluation frameworks.

Requirements

• Should hold Bachelor’s degree completed.
• Native-level or near-native fluency in Arabic (ILR 5 / CEFR C2) with strong English proficiency.
• Proven experience using large language models and understanding real-world LLM usage.
• Excellent writing skills with the ability to provide clear, nuanced, and structured feedback.
• Strong attention to detail and analytical thinking abilities.
• Comfortable working across multiple topics, domains, and evolving requirements.
• Background in structured analytical fields such as research, policy, analytics, linguistics, or engineering.
• Strong college-level mathematics skills.

Application Process (Takes 20 Min)

• Upload resume
• Interview (15 min)
• Submit form