Types of evaluation tasks

This depends on the project. Some projects involve multiple task types and will rotate you between them. Others are focused on a single type. Your project lead will clarify what's expected during onboarding.

Yes. Each task type has specific quality rubrics. For RLHF, justifications must be specific and evidence-based. For annotation, precision and consistency are measured against calibration sets. Your project lead will share the relevant quality guidelines during onboarding.

Task availability fluctuates based on project needs. If your queue is empty, check back later or reach out to your project lead on Discord. Gaps between tasks are normal, especially between project phases.

Yes. Response Writing (SFT) and Red-Teaming typically command higher rates due to the skill level required. RLHF and annotation rates vary by domain complexity. Your rate is specified in your contract and doesn't change based on the task type unless your contract explicitly states otherwise.

Rating & Scoring and RLHF tasks are generally the most accessible for new evaluators. They have clear rubrics and structured formats. Response Writing (SFT) is the most demanding and is typically offered to experienced evaluators.

You can express a preference to your project lead, but task assignments are based on project needs and your demonstrated abilities. Performing well on your current tasks is the best way to influence future assignments.

Review the project guidelines and rubric carefully. Use Discord to ask your project lead if you're unsure. Some projects include calibration tasks at the start to help you understand expectations. Pay attention to any feedback on your submissions.

Your project lead will typically provide feedback and guidance. If quality doesn't improve, you may be reassigned to a different task type or project. This is not necessarily a negative outcome — different people excel at different types of work.