Robot Task Evaluation & QA

Every robot you have seen fold a shirt, stack a box, or roll through a warehouse got there because a person watched it try, decided whether it worked, and wrote down exactly what went wrong. That person is a robot task evaluator. It is one of the most overlooked jobs in embodied AI, and one of the more accessible ways in: the operator tier usually needs no degree, just patience, a careful eye, and the discipline to grade the two-hundredth attempt as honestly as the first.

This guide teaches the work itself: what you actually do, how a robot run is scored, why so many demonstrations get thrown away, the gear you will touch, and how to break in with zero experience. For the pay numbers, the platforms hiring right now, and live openings, the companion hub is RobotTraining.jobs's evaluation and QA pay guide. This page is the how, that one is the where and how much.

1. What robot task evaluation and QA really is

A trained robot runs on a policy: a neural network that takes in what the robot sees and feels, and decides how it moves. Training a policy is only half the job. The other half is finding out whether the policy is any good, and that cannot be measured from a spreadsheet. It has to be tested on real hardware, in the messy physical world, one attempt at a time. That is your job.

The core loop is the same at almost every lab. You set up a physical scene (a table with a mug, a towel, and a bin, say), you trigger the robot to run its current policy, you watch the attempt, and you grade it against a rubric. Then you reset the scene and do it again, and again, often dozens or hundreds of times per task. The scores you produce tell engineers whether the last change to the model helped or hurt, and your written failure notes tell them what to fix next.

It matters because you cannot trust a robot you have not measured, and you cannot measure it casually. NVIDIA built RoboLab specifically because ad-hoc evaluation "lags behind gains in robotics foundation models": the models were improving faster than teams could reliably tell whether they were improving. Structured human evaluation is the bottleneck, which is exactly why it is a paid, growing role and not an afterthought.

Evaluation sits right next to, but is distinct from, teleoperation and data collection. Collectors generate the demonstrations a robot learns from. Evaluators grade what the trained robot does with them. Many operators do both across a single shift, and the same labs hire for both.

2. The four kinds of evaluation work

"Evaluation and QA" is an umbrella over four related jobs. Most roles are a blend, but it helps to see them separately.

The first three are the operator tier: the accessible on-ramp. Red-teaming and safety review is the expert tier, and it pays more because scarce judgment, not clicks, is the product. We will come back to why that pay gap exists in section 6.

3. How a robot run is actually scored

This is the heart of the craft, so it is worth understanding from first principles. The naive way to grade a robot is pass or fail: did it complete the task, yes or no? That number, the success rate, is still the headline metric at most labs, and RoboChallenge (a large-scale real-robot benchmark) reports it as the primary score. But a single bit per attempt is a terribly thin signal. A robot that grabbed the right mug and missed the bin ties with a robot that never moved, and an engineer learns nothing from either.

So serious evaluation programs layer on more:

• Progress score. The task is split into stages (grasp the mug, lift it, carry it, place it, release), and you award partial credit for each stage the robot reaches before it fails. NVIDIA's RoboLab calls these graded task scores and awards partial credit for completed subtasks inside multi-step instructions, so a near-miss no longer looks identical to a total failure.
• Trajectory quality. Was the motion smooth and efficient, or jerky and hesitant? Some rigs score path length and motion smoothness automatically, but a human eye still catches the wobble that a metric misses.
• Failure tagging. When the robot fails, you classify why: wrong object, slipped grasp, collision, froze, ignored the instruction. These tags are the single most useful thing you produce, because they tell engineers which failure mode to attack next.
• Written notes. A short narrative of what happened. The best-practice literature on policy evaluation is explicit that teams should report the number of runs, the initial conditions, the success criteria, and a qualitative description of the failures observed, not just an aggregate percentage.

Two things separate a good evaluator from a warm body. First, consistency: you must apply the exact same rubric to trial 1 and trial 200, even when you are bored and the robot keeps failing the same way. Second, volume literacy: a difference between 90% and 92% success can be pure noise, and Toyota Research Institute has shown that tightening a confidence interval around a 90% success rate can require roughly fifteen times more rollouts. You do not need the statistics to hire, but understanding why you are asked to run so many trials, and why you must not "help" the robot to make the numbers look better, is what makes you trusted with the harder work.

4. Why so many demonstrations get rejected

A large slice of QA work is not grading the robot at all. It is grading the humans who trained it. Before a batch of teleoperated demonstrations is fed into the next model, a reviewer watches them and throws out the bad ones, because bad demonstrations actively poison a training set.

What gets a demo rejected? Not just failure. A demonstration can complete the task and still be useless. Research on closing the teleoperation loop describes episodes that are " task-successful but suboptimal for downstream use due to inefficient motion, repeated corrections, or operation near robot joint limits." In plain terms, you reject a demo when the operator:

• corrected themselves repeatedly (the robot learns the hesitation, not the skill),
• moved unevenly or far slower or faster than normal,
• drove the arm to the edge of its physical range,
• went off-task, touched the wrong object, or did something unsafe,
• or produced a trajectory a trained policy simply could not replay.

Acceptance is usually a threshold, not a vibe. Production teams set per-task acceptance criteria and often use a gold set (a batch of pre-graded examples) to calibrate reviewers against each other, so two people grading the same demo reach the same verdict. Learning to see the difference between "it worked" and "it worked cleanly" is the skill this part of the job builds.

5. The tools and gear you will use

Evaluation and QA is lighter on hardware than teleoperation, but you are still on a robot floor, and the day-one kit is real.

• The robot and its scene. Humanoids, bimanual arm rigs, or mobile platforms, plus the props and fixtures for each task. Resetting the scene identically between trials (same object, same starting position) is part of the job, because inconsistent setup silently corrupts the numbers.
• An evaluation console or grading interface. A tablet or laptop app where you trigger a run, then enter success, progress score, and failure tags. Many labs build this in-house; the workflow mirrors public benchmarks like NVIDIA's RoboLab, which pairs automated success detection with human scoring.
• Camera and log review. For demo acceptance and regression work, you review recorded episodes from multiple camera angles alongside robot telemetry, scrubbing back and forth to judge a grasp frame by frame.
• A rubric and a gold set. The written scoring standard for each task, plus calibration examples that keep your grading aligned with the rest of the team.
• Safety gear and awareness. You are near moving hardware. Following the stop protocol and the controlled-reset routine matters more than any tool.

You do not buy any of this. If a listing asks you to pay for equipment, training, or a "certification" before you start, it is a scam. Legitimate labs provide the hardware and train you on the rubric.

6. What it honestly pays

Read from today's open roles. 9 open roles on RobotTraining.jobs are in robot evaluation and QA, and 2 of them state an hourly rate. The board publishes a typical band for a kind of work only when enough employers state a rate; the pay explorer shows whether this kind has one yet. They come from individual postings with different duties and locations, not from a category-wide wage survey; the pay explorer and pay report show the full measured sample and its coverage.

An advertised hourly amount is gross compensation, not verified take-home earnings. Salary, equity, benefits, overtime and contractor expenses must be compared separately. We do not infer what a beginner or an expert earns by splitting a pay distribution into tiers.

For accepted-footage or task payments, include setup, recording, upload, review and any rework in your time calculation. An hour of accepted video is a unit of output, not necessarily an hour of labor. There is no guaranteed volume of accepted tasks or paid hours.

7. Who is hiring, and what they screen for

The pipeline runs through two channels. The first is the humanoid and manipulation labs directly. Figure, 1X, Tesla's Optimus program, Boston Dynamics, Agility Robotics, RoboForce, and Generalist AI all post operator, technician, model-evaluation, and QA roles on their own careers pages. The second is the autonomous-vehicle and service-robot companies ( Nuro, Waymo, Zoox) that need people to review and score autonomous runs.

For the operator and QA tier, hiring managers want reliability over resume. The traits that get you hired and kept: consistency (you grade the last trial as carefully as the first), attention to detail, comfort following a strict rubric without improvising, physical presence and dexterity for on-site scene resets, and clear, specific written notes. A degree is usually not required, and prior gaming, lab, warehouse, or quality-inspection experience reads as a plus.

Specialized safety-review jobs can require robotics, test or domain experience. Evaluate the requirements on the actual robotics posting. Generic AI red-teaming contracts do not establish a pay premium for physical robot evaluation.

8. How to start with no experience

You do not need a robotics background. You need to prove you are reliable and careful. Here is the honest first-principles path.

1. Apply directly on labs' own careers pages. Filter for the titles that carry this work: "operator," "data collection," "model evaluation," "robotics technician," "QA." Skip third-party recruiters where you can. Start from the employers on the evaluation and QA hub, which links straight to live openings.
2. Solve for location first. Most on-site roles cluster in a few hubs (the Bay Area, Austin, the Boston area). Location is the main gate for the operator tier. If you cannot relocate, aim at remote demo-review and red-team work instead, and be clear-eyed that remote rates are lower. Red-teaming chatbots and other AI models is a separate, remote job, and the red-teaming guide on AITraining.jobs, which we also built, walks through how that work is done and how to start.
3. Speak the language of consistency. In your application and interview, give a concrete example of repetitive, precise, standards-driven work you did without cutting corners (inspection, lab work, medical, manufacturing, competitive gaming). That is the exact trait they are screening for.
4. Learn the vocabulary before you interview. Understand success rate, progress score, failure tagging, and demo acceptance (sections 3 and 4 above). Being able to say "I would grade partial progress and tag the failure mode, not just pass or fail" signals you already get the job.
5. Treat the trial shift as the interview. Many labs run a hands-on assessment. They are watching whether you follow the rubric exactly and stay steady when the task is dull. Steadiness is the whole test.
6. Build a record of reliable evaluations. Keep examples of clear failure reports and consistent scoring. More specialized work may require further technical training; this guide does not establish a seniority pay ladder.

If evaluation feels like a fit, its closest sibling on-ramps are teleoperation (piloting the robot) and embodied-AI data annotation (labeling what the robot perceives). Many people move between all three.

9. Who this work suits, and who it does not

It suits you if you are patient, detail-obsessed, and genuinely fine with repetition: someone who can watch a task fail forty times and still tag failure number forty accurately. It rewards honesty under pressure, because the temptation to nudge a struggling robot or round a score up is real, and resisting it is the job. If you like being close to cutting-edge robots without needing to be an engineer, this is one of the shortest paths in.

It does not suit you if you need variety hour to hour, dislike strict protocols, or want fully remote work at US on-site pay (that combination mostly does not exist yet). Robot evaluation and QA is legitimate, growing, and one of the better-paid entry points into embodied AI, and it is also disciplined shift work that values reliability over flash. If that trade reads as fair to you, start with the employers above and the live roles on the RobotTraining.jobs evaluation and QA hub.