Fencing Visualized
A real-time AR system that detects and visualizes fencing sword-tip trajectories — making the sport’s fastest, near-invisible movements legible to audiences.
In fencing, the decisive action happens in a few hundredths of a second — the tip of the blade moves faster than the eye can follow. Fencing Visualized is a real-time augmented-reality system that detects those sword tips and overlays their trajectories live, so an audience can finally see what the athletes are doing. It was developed at Rhizomatiks Research in collaboration with Dentsu Lab Tokyo.
Origin
The project began from a simple problem posed by Olympic fencing medalist Yuki Ota: fencing is thrilling to compete in but notoriously hard to watch, because spectators can’t track the blade. The goal became making the sport “easier to understand and more appealing for the audience” — a sport everyone can enjoy because they can actually follow what is happening in the match.
How it works
The core challenge is brutal for computer vision: a sword tip is only a few pixels wide even in 4K footage, and it moves at extreme speed against a busy background.
Early versions (2013–2014) relied on optical motion capture with retro-reflective markers attached to the blades. From 2016 onward the system moved to a marker-free approach driven by deep learning — detecting the bare sword tip directly from camera images, so athletes could compete normally.
The detector is a deep neural network for object detection using cascaded classifiers based on YOLO v3, paired with custom hardware. An array of 24 cameras lines both sides of the piste (each covering roughly 8 meters), and the pipeline was upgraded from 2D position estimation to full 3D estimation, all running in real time so the AR trajectory is synthesized instantly during the bout.
Training the model
Detecting something this small and fast demands an enormous, carefully labeled dataset. In 2019 the team shot dedicated footage at Kiramesse Numazu using 8 cameras and 12 fencers, then annotated over 200,000 images. To push robustness further, more than a million CG datasets were generated for data augmentation — synthetic blades, lighting, and motion the real shoot could never cover.
Timeline
- 2012 — Initial AR motion-capture experiments visualizing dancers’ movement.
- 2013 — The Fencing Visualized Project is officially launched.
- 2014 — Real-time AR demonstrated at the Yuki Ota Cup demo match.
- 2017 — Real-time AR live-streamed at NTT DOCOMO Future Experiment Vol. 2.
- 2018 — Exhibition demonstration at the 71st All Japan Fencing Championships.
- 2019 — Used in actual matches at the 72nd All Japan Fencing Championships, and deployed at the H.I.H. Prince Takamado Trophy JAL Presents Fencing World Cup.
In competition
By 2019 the system had crossed from demo to live officiating-grade visualization, running in real competition at the All Japan Fencing Championships and the Fencing World Cup — broadcasting blade trajectories to spectators as the bouts happened.
Recognition
The work was presented as Fencing Visualized at ACM SIGGRAPH Asia 2021 (Emerging Technologies) — Shimizu, K., et al. (2021).
Project documentation: Rhizomatiks Research — Fencing Tracking.
More Images
