Checking a Video Analytics Vendor's \"95% Accurate\" on Your Own Footage

"95% accurate" is not a measurement until you know what was counted. Ask for two numbers, precision (how many alerts were real) and recall (how many real events were caught), counted per event on your own cameras, split by day, night and dusty zones. Build a small ground-truth sample yourself and write the pass marks into the pilot contract before the pilot starts.

By the Mama Editorial Team. Mama scopes camera and floor-visibility projects for Indian factories. A buyer's guide for owners and plant heads with a vendor accuracy figure in hand and a pilot about to start: how to check it on your own floor in about two weeks, without a data scientist.

Key takeaways

• Accuracy is the wrong headline number for rare events. If machines run most of the time, a system that always says "running" scores high accuracy and catches nothing.
• Ask for precision and recall, per event. Precision is the share of alerts that were real; recall is the share of real events the system caught. These are the scikit-learn definitions.
• The unit matters as much as the metric. Per-minute, per-frame or per-hour scoring flatters a system; per-event scoring is what your supervisors live with.
• Small samples are noisy. On 20-odd events, a 73% recall could really be anywhere from about 52% to 87%. Size the sample before you argue about decimals.
• Write the pass marks into the pilot, with the definitions, the unit and the sample plan, before day one.

What "95% accurate" can actually mean

A vendor's accuracy figure usually comes from one of four places, and each says something different:

1. Object accuracy on a benchmark dataset. Says nothing about your 30-foot roof, sodium lamps or lint.
2. Per-minute or per-hour classification. Each minute of each machine is labelled "running" or "stopped", and accuracy is the share of minutes labelled correctly. Because most minutes are running minutes, this number is high almost by default.
3. Per-event detection on another customer's site. Closer, but someone else's angles and lighting.
4. Per-event detection on your site. The only number that predicts what your supervisors will see.

In Introduction to Information Retrieval, Manning, Raghavan and Schütze explain that accuracy is a poor measure when data is skewed: in their case over 99.9% of documents are non-relevant, so a system can look strong by calling everything non-relevant while being useless. Google's machine-learning course agrees: on 99%-negative data, always predicting negative scores 99% accuracy despite being useless. On a shop floor, "stopped", "wrong-zone material" and "blocked aisle" are the rare class.

So ask: accurate at what, counted how, on whose footage?

Precision and recall, in shop-floor words

The definitions are short. From the scikit-learn documentation:

• Precision = TP / (TP + FP). Of everything the system flagged, how much was real.
• Recall = TP / (TP + FN). Of everything that really happened, how much the system flagged.

Here TP is a true positive (system flagged it, it happened), FP a false positive or false alarm (flagged, didn't happen) and FN a false negative or miss (happened, not flagged). scikit-learn also defines the F1 score as the harmonic mean of the two (in the example below, 75.6% precision and 85.0% recall give F1 = 80.0%). Don't accept it alone: it hides which way the system errs.

The two numbers pull against each other. Google's course notes that raising a detection threshold tends to cut false positives, and that precision and recall often move in opposite directions. A vendor can tune either up for a demo, so ask for both at one frozen setting.

What each failure costs you is different. A miss on a machine stoppage is an hour you never learn about. A false alarm costs a supervisor's walk and, repeated often, his trust in the system. Decide which hurts more for each use case before you set pass marks.

Per event or per minute: agree the unit first

Before any counting, write down three definitions with the vendor:

• What an event is. For example: "a machine stoppage is a continuous period of 5 minutes or more with the spindle not turning during a scheduled shift."
• What counts as a match. For example: "a system flag matches a true stoppage if it starts within 3 minutes of the true start." Two flags for one stoppage count as one TP and one FP.
• What is excluded. Planned breaks, scheduled changeovers, camera offline periods. Agree up front whether an NVR gap during a power cut counts as a miss (we would say it is a separate uptime number, reported on its own).

If the dashboard reports minutes, ask for an event export: start, end, camera, machine.

Build a ground-truth sample from your own cameras

Ground truth is a human-checked record of what really happened. You do not need to label everything; you need a fair sample.

Choose the cameras. Pick three to five cameras that represent your floor honestly: one clean, well-lit view, one at night under IR, and one hard case such as a grinding bay, a cotton-dust area or a camera with a column in the way. You pick, not the vendor.

Choose the time. Sample whole shifts, not highlight clips, and include at least two night shifts and one shift change. Pick the shifts by date in advance so nobody cherry-picks after seeing results.

Label blind. Two people, ideally a supervisor and a maintenance hand, watch the recorded footage at 2x–4x speed and log every event against your written definition, without seeing the system's output. Where they disagree, a third person decides.

Then compare. Line up the human log with the system's event export and mark each item TP, FP or FN. A spreadsheet is enough.

Mind the people in the frame. You are labelling machine states and zones, not individuals. Keep the review footage inside the plant, restrict who watches it, and make sure your worker CCTV notice covers analytics testing (template here).

A worked ground-truth example (assumptions, not results)

Every figure below is an assumption chosen to illustrate the method. None is a measurement of any vendor or of Mama.

Assume a machining shop tests machine-stoppage detection on 48 machine-shifts (six machines, eight shifts, 8 hours each = 384 machine-hours, or 23,040 machine-minutes). The labellers find 80 true stoppages of 5 minutes or more, totalling 2,400 stopped minutes. The system flags 90 stoppages.

Condition (assumed)	True stoppages	System flags	TP	FP (false alarms)	FN (misses)	Precision	Recall
Day, clear view	48	52	44	8	4	84.6%	91.7%
Night, IR	22	26	16	10	6	61.5%	72.7%
Dusty / partly blocked	10	12	8	4	2	66.7%	80.0%
All	80	90	68	22	12	75.6%	85.0%

Now look at how the same system could be described:

• Per-minute accuracy. Assume the system mislabels 600 of the 23,040 minutes in total. Accuracy = 22,440 / 23,040 = 97.4%. That is the "97% accurate" on the slide.
• A system that does nothing. Label every minute "running" and accuracy is 20,640 / 23,040 = 89.6%, with zero stoppages caught. That is the floor for "accuracy".
• Per event. Precision 75.6%: one alert in four is a wasted walk. Recall 85.0%: 12 real stoppages went unseen. At night, recall drops to 72.7% and precision to 61.5%.

What it means in rupees (assumptions). Assume each missed stoppage averages 25 minutes and the machine-hour rate is ₹900 (use your own costed rate). The 12 misses hide 300 minutes, or ₹4,500 of machine time, in just 48 machine-shifts. Assume each false alarm takes a supervisor 5 minutes to check: 22 false alarms cost 110 minutes of supervision.

How big a sample is big enough

Percentages on small counts swing a lot. The NIST/SEMATECH e-Handbook of Statistical Methods recommends the Wilson score interval for a proportion, noting that it suits virtually all combinations of sample size and proportion and that its lower limit cannot be negative, unlike the simple normal-approximation formula. Applying it at 95% confidence to the assumed numbers above:

• Overall recall 68/80 = 85.0%, plausible range about 75.6%–91.2%.
• Night recall 16/22 = 72.7%, plausible range about 51.8%–86.8%.
• Day recall 44/48 = 91.7%, plausible range about 80.4%–96.7%.

The practical lessons:

• Rule of thumb (ours, not a statistical standard): aim for at least 50 true events per condition you care about. Even then the range is wide: 43 caught out of 50 (86%) still spans about 73.8%–93.0%. Under about 20 events, you can tell "terrible" from "fine" but not "85%" from "92%"; if you need to separate close numbers, you need hundreds of events, not dozens.
• Rare events need more footage, not more cameras pointed at the same thing. If a blocked fire aisle happens twice a month, two weeks of footage will not score it. Either extend the window or accept that you are scoring the common events only.

Night, dust, glare and other Indian conditions to test on purpose

The blended number looked fine; the night row is where month three disappoints. Sample these deliberately:

• Night under IR. Colour disappears, reflective surfaces bloom, and moving machine parts blur. See how your cameras behave in the night and low-light guide.
• Dust, lint, mist and steam. Foundries, spinning and weaving sheds, rice mills and wet processing areas put particles in front of the lens; IR light reflects off them at night.
• Flicker and glare. LED lighting and skylights can produce rolling bands or sudden exposure jumps that confuse motion-based logic. Check the 50 Hz flicker fix before blaming the analytics.
• Occlusion. Stacked WIP, a parked forklift, a monsoon tarpaulin.
• Stream quality. Many systems analyse the low-resolution substream, not the recording you watch. If the substream is set to 1 fps or a tiny resolution, accuracy falls; confirm the substream settings are the ones that will run in production.

What to write into the pilot contract

Turn all of the above into a short acceptance schedule. Paste-ready wording to adapt with your lawyer:

1. Definitions. "Event", "match window", exclusions and the unit of scoring (per event) as agreed above.
2. Metrics. "Precision = TP/(TP+FP); recall = TP/(TP+FN), as defined in standard references, computed per event."
3. Pass marks per condition, for example: day precision ≥ 80% and recall ≥ 85%; night precision ≥ 65% and recall ≥ 75%. These are illustrative; set your own from what a miss and a false alarm cost you.
4. Sample plan. Cameras and shift dates chosen by the factory before the pilot; a minimum number of true events per scored condition (we suggest 50 as a rule of thumb) where they occur.
5. Frozen settings. No threshold or model changes during the scoring window; tuning happens before it, and is logged.
6. Ground truth by the factory. The factory's labels decide; the vendor may review disagreements, not overrule them.
7. Uptime reported separately. Camera, network and NVR availability as its own number, so outages are neither hidden nor double-counted.
8. Exit. If pass marks are not met, the factory may end the pilot at no further charge.

These clauses slot into the 30-day pilot plan and the video analytics RFP template.

What this method cannot tell you

• It measures detection, not value. A system can score well and still flag things nobody acts on.
• Your labels are imperfect too.
• A two-week sample is a snapshot. Re-score a small sample every quarter.
• Cameras cannot see what is out of frame, inside an enclosure, or behind a guard. No metric fixes coverage.
• It does not judge people. Keep the scoring at machine and zone level; the method is not a tool for ranking operators.

Where Mama fits

Mama reads a factory's existing cameras, or cameras added where coverage is thin, and sends the owner a short morning note on where machine hours and money leaked the day before. It works on machines, flows and aggregate counts, not named individuals. Detection quality depends on your cameras and conditions, so the method above is the right way to judge any system, Mama included. Mama can draw a camera plan for the machines and events you intend to score, so the ground-truth sample has the views it needs. Leave an email or phone in the form below.

FAQ

Is 95% accuracy good for factory video analytics? It depends entirely on what was counted. Per-minute accuracy on mostly-running machines can be over 90% for a system that catches nothing. Ask for per-event precision and recall on your own footage.

Which matters more, precision or recall? For costly, rare events (a stopped bottleneck machine, a blocked exit) recall usually matters more, because a miss is invisible. For frequent, low-value events, precision matters more, because false alarms train people to ignore the system.

How many hours of footage do I need to label? As a rule of thumb (not a standard), enough to contain roughly 50 true events per condition you score. For common events like machine stoppages, a few shifts per camera can be enough; for rare events, weeks.

Can the vendor build the ground truth for us? They can help set it up, but your own people should label blind and settle disagreements, or the vendor grades its own homework.

Should we accept F1 score as the single pass mark? Use it to compare vendors, but write separate pass marks for precision and recall in the contract, so a system that is weak on one cannot hide behind the other.