Running a time study

A time study measures how long one task takes the people who do it, with the AI and without it. Its result becomes the figures a formula compares, such as agent minutes per ticket before and after. It's how to measure a before-figure that no system records, and how to reconstruct one once the AI is already in use. See Baselines for an initiative that's already live.

Time studies live under Method → Time Studies. An Owner or Admin, or the initiative's owner, can start one.

Setting one up

Choose New Time Study. Pick the initiative and the metric it measures. Only metrics counted in minutes are offered.
Describe the task as participants will read it. Say what's in and what's out: "One ticket, from first reading it to marking it solved. Count only the minutes you spend on it, including checking what the AI drafted."
Choose what it times: with the AI and without it, or one of the two. With both, Roiva draws each participant's days without the AI at random, so nobody chooses which days go into which arm.
Set the window, the time zone its days run in, the confidence of its interval (90% or 95%), and how long its result stands before the task is due to be timed again (3, 6 or 12 months).
Set the precision you want and how much the task varies. The study's page then says how many timings each arm needs, at least 30. Once timings arrive, it says how many more reach that precision. Aim for at least five people in each arm. Three is the fewest a result can be published from, and with so few the interval understates how much people differ. From the third day, if an arm has fewer timings than its share of that number for the days gone, the study shows in the Inbox of whoever manages it, once a week while it stays behind.

Who times the task

Add the people who do the work.

A member of the account records in the app, signed in. Their Inbox shows the study each day, with that day's arm.
Anyone else gets a link of their own, under a label such as "Agent A" rather than their name. Roiva never emails the link: copy it and send it yourself. It's shown once, stored only in a form that can't be turned back into the link, and stops working a week after the window ends, or as soon as you revoke it or issue a new one. The page it opens shows the task, the study's name and dates and the account's name, and nothing else in the account.

Each person times one task at a time with the page's timer, pausing it when they step away, or types the minutes. The study says which arm each day is. They can't change a timing once it's recorded. If one is wrong, tell the study's owner, who can leave it out of the result with a reason; it stays on the record.

Importing timings from another tool

If your team already tracks time in Zendesk, Freshdesk, Harvest, Toggl or Jira, the study's owner can choose Import Timings on the study and upload a CSV of its export. It needs a header row naming these columns, one task to a row:

participant: a member's email, or a label of up to 60 characters for someone outside the account. Many tools export full names: replace them with labels first if you don't want names kept.
arm: with the AI, or without it.
minutes: hands-on minutes, such as 12, 12.5 or 12:30.
date: YYYY-MM-DD, inside the study's window.
task (optional): a ticket or task ID.

If any row can't be read, nothing is imported, and the rows are listed. A row already imported is skipped, so importing the same file twice records it once. Roiva doesn't keep the file.

A tracker workbook's Time studies and Timings sheets bring in whole studies instead, each new, with their timings marked imported in the same way: importing the workbook again creates them again. Its minutes are numbers, such as 12.5. See Importing a tracker workbook.

Roiva didn't draw the days of imported timings, so whoever recorded them chose which went into which arm. The result says how many timings were imported, and a study with any publishes its without-AI figure as a sample, never as measured before go-live.

The result

The study's owner sees the result as timings arrive. Whoever manages the study sees it once it's closed with enough people to publish, and everyone in the account sees it once it's published.

For each arm it shows:

the mean minutes per task, with an interval;
how many timings it rests on, from how many people;
the median and the middle half, so a few very long tasks are seen.

The interval is worked out by person. Twenty timings from one fast person are one person's pace, so the interval narrows only as more people are timed. With both arms, it also shows the difference between them and its interval. When the initiative has a formula comparing the metric before and after, and last month has every figure it reads, it also shows what that range is worth a month at last month's volume and rates, and whether that's cash or freed capacity.

The figure published is the mean, not the median: a formula multiplies minutes by volume, and only the mean times the volume is the total.

Publishing

Close the study, then choose Publish result. Each arm needs timings from at least three people.

The without-AI mean becomes the initiative's baseline for the metric. It's labeled measured (a time study before go-live) only when nothing in the study used the AI. Otherwise it's a without-AI sample, a reconstructed baseline.
The with-AI mean becomes a reading of the metric.

Both stand from the month the window started until the study's re-study date. Roiva recalculates each month's figures from them; anything already approved stays as it was approved.

A study of agent minutes per ticket also prices the initiative's deflected tickets, from the help desk's next sync, if they were on Roiva's default price: its without-AI minutes multiplied by your labor rate. See How deflection value is booked. A published study takes no more exclusions, and its figures can't be edited: to change them, time the task again in a new study.

Whoever published the result, the study's owner, whoever left timings out, whoever imported timings and every member who timed tasks for it count as authors of the figures, as whoever types a reading does. Your approval settings then decide whether they may approve value calculated from them. See Approving value and initiatives.

Timing it again

From a month before a study's re-study date, whoever manages it (its owner, the initiative's owner, and the account's Owners and Admins) sees it in their Inbox. After that date, value calculated from its figures is marked Re-study due and held for a person to approve. Publishing a new study of the same task ends that. The months before the new study's window keep the figures they were calculated from.

What a time study shows, and what it doesn't

It shows how long the timed tasks took the people timed, in that window, with an interval for how much it could differ had other people been timed.

It doesn't show:

that the tasks timed stand for every task;
that people work at the same pace when they aren't being timed;
that typed minutes are exact;
what any minutes freed were used for;
that the AI caused a change in volume or outcomes.

It's a measured estimate of the time a task takes, not an experiment. A measurement charter can name the study a baseline came from. See Measurement charters.

Baselines for an initiative that's already live: the other ways to set a before-figure
Metrics, baselines and observations: how formulas read figures
Aggregate figures only: a note for your risk team: what a study keeps about a person outside the account