Tracking lacrosse players from game film
Earlier attempts
I've been interested in machine vision for awhile. Back in 2024 I fine-tuned a YOLO model on a few hundred labelled lacrosse images of on-field identities (goalie, longpole, player, referee) and had it predict those identities as overlays back onto game film.
It worked OK -- reasonably rejecting sideline players, having good fidelity on referees and goalies for the most part -- but it was surprisingly fragile to different angles, lighting, etc.
New promising ideas
Recently, I've seen a flurry of posts, like this one from @skalskip92 in a basketball scenario, demonstrating pipelines of detecting, tracking, re-identifying players based on OCR of jersey numbers, and using keypoints on court to project screen locations to court locations for a radar inlay.
The other novel idea here was how to use GPT Astra to auto-annotate player IDs at $0.08 per frame, but instead of using it for every frame, just syncing the IDs to the tracks produced by the cheaper existing workflows. What I wanted to figure out was whether these workflows could work on lacrosse game film too.
Player Identification
Lacrosse is genuinely hard! Helmets erase faces, so some of the appearance cues that carry soccer or basketball ReID are gone. Players numbers are often occluded by each other or their own sticks, and bodies frequently crash together into scrums that leave multi-second gaps in clean trajectories.
The tech stack
RF-DETR, zero-shot, for people. No lacrosse training data at all.
BoT-SORT with camera-motion compensation for tracking, because the camera pans and zooms constantly and every track lives in a moving frame.
VLM calls per tracklet for assignment of team, jersey number, and some plain text info on stick characteristics and player pose.
Homography interpolated between keyframes to transpose pixel screen positions onto field positions
Kalman filter and RTS smoother to stitch together tracklets and VLM anchor points with ellipsoid position uncertainty estimates and physics movement constraints into continuous estimated player positions per frame
Where it landed
One Torrey Pines offensive possession against Bishop's. Forty-eight seconds, thirteen players, no gaps in time and no gaps in attribution — every player accounted for, every frame, with a jersey number above his head and a marker at his feet, and the same thirteen simultaneously on a field radar in real yards.
It is one possession, at one end of one field, in one game, and I checked a lot of it by hand. But it's pretty promising all-in-all. Some ideas for things to try next to build on this:
Different approaches to identifying the player possessing the ball, either directly in frames where the ball is visible in his stick, or by inference by tracking the ball as it's passed between players in the air, or through more complex discriminators on pose, movement, positioning of players.
Normalizing pipeline (after the first attempt took a lot of circular detours to address edge cases). Testing robustness on other posessions, during transition, during substitutions, faceoffs, etc.
Additional Links
The work that started this:
Tools:
RF-DETR (Apache-2.0)
supervision (MIT)
Comments