Approach
One camera never moves, so we learn only what has to be learned, which is where the road users are. Everything else is explicit geometry and kinematics you can read, test and tune: when a car counts as stopped, which way a lane flows, how close two vehicles came.
Two parts, one perception stack
Part A: events
Reads the whole file, so it can smooth tracks in both directions. Frames are decoded in order and every second one goes to the detector at 960 px in FP16 batches of 16. ByteTrack links boxes into tracks. Kinematics turn tracks into speed, heading and acceleration. Fourteen rules turn those into segments, and post-processing makes the output valid.
Part B: risk
A separate copy of the same detector and tracker receives frames one at a time and never sees the future, the file or Part A's output. Every third frame it measures time to collision, braking, swerving, wrong-way motion and pedestrians on the road. A logistic combiner, a causal moving average and a one-second peak hold turn those measurements into a 0 to 1 score.
What is learned and what is a rule
| Stage | Kind | Details |
|---|---|---|
| Object detection | Learned | YOLO11m, COCO weights, FP16; 960 px for events, 640 px for risk. Keeps person, bicycle, car, motorcycle, bus and truck, plus animals and luggage for obstacles. YOLO11n on CPU. |
| Tracking | Algorithm | ByteTrack with a 30-frame buffer. Tracks are split at physically impossible jumps and short gaps are interpolated. |
| Kinematics | Rule | Savitzky-Golay smoothing (Part A) or least-squares windows (Part B). Speeds are normalised by the typical vehicle size at that image row, so one threshold works near and far. |
| Scene | Hand-drawn, with automatic fall-back | Lanes, stop lines, crosswalks and zones come from scene.json. When absent, the road surface and lane flow are learned from the video's own tracks. |
| Events | Rule | One class per file in roadsight/events/rules/; every threshold lives in configs/default.yaml. |
| Risk | Hand-set logistic, refittable | Weights in weights/risk.json; tools/fit_risk.py refits them on labelled accidents. |
The fourteen rules
A class that is predicted but never occurs adds a zero to the macro average, so each class ships only behind a confidence gate. Classes that need scene geometry stay silent until it is drawn.
| Class | Starts when | Ends when | Main signal | Needs |
|---|---|---|---|---|
| stopped vehicle | Vehicle stops | Moves again or leaves | Below 0.1 BL/s for 10 s while same-direction traffic keeps passing | auto road |
| congestion | Queue stops moving | Queue clears | Median speed per travel direction below 0.2 BL/s with many vehicles | auto road |
| jaywalking | Pedestrian steps onto the road | Leaves it | Foot point inside the carriageway, away from crosswalks and parked cars | auto road |
| wrong way | Enters against the flow | Back in a correct lane or gone | Heading against the lane flow for 1.5 s over at least one car length | auto flow |
| accident | First contact | Everyone involved stops | Box contact, speed collapse, then standing still or people appearing | none |
| near miss | Evasive action starts | Pair separates | Time to collision under 1 s plus braking or swerving above the video's own noise | none |
| red light | Front crosses the stop line on red | Clears the junction | Signal colour in a drawn box plus line crossing | stop line, signal |
| stop line | Stops past the line on red | Signal turns green | Stationary beyond the line during red | stop line, signal |
| failure to yield | Vehicle enters the crosswalk | Leaves it | Pedestrian on the same crosswalk nearby | auto crosswalk |
| solid line crossing | Wheels reach the line | Fully across | Footprint straddles a drawn solid line, then stays on the other side | solid lines |
| illegal U-turn | Turn starts | Heading settles | Heading change over 150° inside a no-U-turn zone | zone |
| illegal turn | Turn starts | Heading settles | Entry and exit arms not in the allowed-movement table | arms, table |
| road obstacle | Object appears | Removed | Animal or luggage on the carriageway for several seconds | auto road |
| fire or smoke | Smoke appears | Clears | Flame colour with flicker, persisting 3 s | off by default |
BL/s is body lengths per second: pixel speed divided by the typical vehicle size at that image row.
Constraints we designed around
- Offline. All weights ship in the repository and every download path is disabled; a test runs the pipeline with sockets blocked.
- Deterministic. Fixed seeds, deterministic cuDNN, sequential decoding and a tracker with per-instance IDs. A test runs each video twice and compares the output byte for byte.
- Causal risk. The risk model keeps its own state and only past frames. A test checks that truncating the video never changes earlier scores.
- Never crash. Every rule runs inside its own guard; a failing rule drops its class for that video, not the video.
- Time. Part A and Part B together take 0.7x to 1.0x the video length on a laptop GPU, against a 3x limit. The stride rises automatically only when a machine is far slower.
Models, data and licences
| Component | Licence | Use |
|---|---|---|
| Ultralytics YOLO11 (m, s, n; COCO weights) | AGPL-3.0 | Detection, unchanged COCO weights |
| ByteTrack (Ultralytics implementation) | MIT / AGPL-3.0 | Multi-object tracking |
| COCO | CC BY 4.0 | Training data of the detector weights |
| Organizer sample videos | Competition terms | Development labels and tuning |