
Project name
Fencing Visualized
Description
Enable detection for real-time AR synthesis in fencing
Role
Researcher / Engineer
Updated Date
2025.01.20
Project Types
Rhizomatiks’ Fencing Tracking and Visualization system uses AR technology to visualize the tips of swords in motion. Building on various development processes since 2012, the system has been updated to utilize deep learning to visualize sword tips without markers. The system enables detection of sword tips, which human eyes cannot follow, and real-time AR synthesis to instantly visualize the trajectory.
We developed the “Fencing tracking and visualization system.” It detects the tips of sabers (fencing swords) to visualize the trajectory of the sabers in real time, which doesn’t require any markers but works only with the input of the images from cameras. This is the only fencing visualization technology that has been used in actual international fencing matches, such as the H.I.H. Prince Takamado Trophy JAL Presents Fencing World Cup 2019.

Fencing sabre, especially its tip, moves quite fast, and its flexibility results in a large distortion in its shape. Additionally the tip is the size of only a few pixels when captured even by a 4K camera so that it is too small to detect with image recognition techniques. We developed a multi-stage deep learning network for general object detection based on YOLO v3 [Redmon and Farhadi 2017, 2018], starting from the hardware selection of a camera for analysis. Since a single camera can only cover about 8 meters, we eventually installed 24 4K cameras on the both sides of the piste to cover the entire match area and improved the robustness of the sabre tip detection. We also developed a system to estimate the 3D position of the tips from the detection results of multiple cameras.
Credits
(Rhizomatiks) Planning, Creative Direction : Daito Manabe
(Rhizomatiks) Planning, Technical Direction, Hardware Engineering : Motoi Ishibashi
Software Engineering: Kye Shimizu, anno lab (Kisaku Tanaka, Sadam Fujioka, Kyle Mc-Donald (IYOIYO)
Dataset System Engineering: Tatsuya Ishii (Rhizomatiks), ZIKU Technologies, Inc. (Yoshihisa Hashimoto, Hideyuki Kasuga, Seiji Nanase, Daisetsu Ido)
Dataset System Engineering : Ignis Imageworks Corp. (Tetsuya Kobayashi, Katsunori Kiuchi, Kanako Saito, Hayato Abe, Ryosuke Akazawa, Yuya Nagura, Shigeru Ohata, Ayano Takimoto, Kanami Kawamura, Yoko Konno)
Visual Programming : Satoshi Horii, Futa Kera (Rhizomatiks)
Videographer : Muryo Homma (Rhizomatiks)
Hardware Engineering & Videographer Support : Toshitaka Mochizuki (Rhizomatiks)
Hardware Engineering : Yuta Asai, Kyohei Mouri, Saki Ishikawa
Technical Support : Shintaro Kamijyo (Rhizomatiks)
Project Management : Kahori Takemura (Rhizomatiks)
Project Management, Produce : Takao Inoue (Rhizomatiks)
This work was conducted with assistance from Dentsu Lab Tokyo
Exhibitions
H.I.H. Prince Takamado Trophy JAL Presents Fencing World Cup 2019 (2019) / Tokyo, Japan / Stanford
エイブルPresents第72回全日本フェンシング選手権大会 (2019) / Palo Alto, California / Stanford CCRMA