RANKING
Best to worst
In a 2022 study, labellers were shown between 4 and 9 replies to the same prompt and ranked them. A separate model then learned to predict which reply they would prefer.
See the stages
WHAT THE RESEARCH SAYS
Behind every reply,
a ranking.
Chatbots are not only trained on text. They are trained by people.
After a language model has learned from text, human trainers write example answers and rank its replies from best to worst. Those judgements teach it to follow instructions, and the people making them are often remote platform workers.
4 TO 9 replies ranked at a time
See the stages- People write examples
- People rank replies
- A model learns the ranking
- Human judgement, at scale
THE SHORT ANSWER
People teach the model
what a good answer is.
The technique is called reinforcement learning from human feedback.
Here is what it involves, and who does it.
WRITING
Show, then tell
Before any ranking, labellers wrote example answers showing the behaviour they wanted. The model was first fine-tuned to copy those demonstrations.
How it works
THE WORK
Remote, often unseen
Oxford researchers say AI developers rely on online platforms for data annotation, labelling, image tagging and content moderation, done by workers mostly in the Global South.
See the numbers
HUMAN FEEDBACK IN FOUR STEPS
From example answers
to a better reply.
This is the method set out in the 2022 paper. Later systems vary the details, but the core idea is the same: turn human judgements into a training signal.
Write the examples
Trainers write good answers to a set of prompts, and the model is fine-tuned to imitate them.
Rank the replies
The model answers each prompt several times, and trainers put the replies in order from best to worst.
Model the judges
A reward model learns from those rankings to predict which reply people would prefer.
Tune against it
Reinforcement learning nudges the model towards replies the reward model scores highly.
TWO NUMBERS WORTH KNOWING
Human feedback works,
the work pays little.
One figure on what the feedback achieved,
one on what online platform workers earn.
100× fewer parameters
Smaller model, preferred by people
In human evaluations, replies from a 1.3 billion parameter model trained with human feedback were preferred to those of a 175 billion parameter model without it.
THE STUDY · 2022
1 in 2
Online platform workers earning under US$2 an hour
The International Labour Organization, drawing on surveys and interviews with some 12,000 workers, found that half of online platform workers earn less than US$2 per hour.
PLATFORM WORK · ILO 2021
The ILO also found that digital labour platforms increased five-fold worldwide in the decade to 2021, and that working hours can often be long and unpredictable.
A QUICK STAGE CHECK
Where do people
come in?
Pick a stage of the training method described in a 2022 research paper to see what happens there. The bar shows how directly human trainers are involved.
Based on Ouyang et al. (2022), Training language models to follow instructions with human feedback.
WHO DOES THE WORK
A planetary
labour market.
The Fairwork project at the Oxford Internet Institute cites estimates that 163 million people are active on freelance and microwork platforms. In 2023 it scored 15 such platforms against basic fair-work principles, and none met its minimum standards.
It found problems of low pay and non-payment, long hours, opaque automated management and harm from dangerous tasks, such as exposure to distressing or violent content. Most platforms could not show they had policies to make sure workers earned at least the local minimum wage.
Questions
Questions people ask
What is an AI trainer?
Here, a person who teaches an AI system by example and judgement: writing model answers, ranking replies, or labelling and tagging data. The work is often done remotely through online platforms.
What does reinforcement learning from human feedback mean?
A training method in which people rank a model's replies, a reward model learns to predict their preferences, and the main model is then tuned to score well against it.
Does human feedback make a real difference?
In one 2022 study, people preferred replies from a model with 100 times fewer parameters that had been trained with human feedback over a much larger model without it. The feedback also improved truthfulness and cut toxic output, though simple mistakes remained.
How many people trained the model in that study?
About 40 contractors, chosen with a screening test. The authors note that this small group was not representative of everyone who would use the model, and that most comparisons were labelled by one person.
What is the work like?
Research is mixed on the benefits but clear on risks. The ILO found half of online platform workers earn under US$2 an hour, and Oxford's Fairwork project found no platform it rated met its minimum fair-work standards.
Sources
- Ouyang et al. (2022), Training language models to follow instructions with human feedback (arXiv)
- ILO: Rapid growth of digital economy calls for coherent policy response (WESO 2021)
- ILO: World Employment and Social Outlook 2021, The role of digital labour platforms
- Oxford Internet Institute: Fairwork Cloudwork Ratings 2023
Updated