The iPhone is a pretty impressive piece of equipment as we mentioned in an earlier post about IMUs (inertial measurement units) and how it can potentially be used to track sports movements. However, few people would want to strap their expensive iPhone on themselves while playing basketball or soccer, especially if it has a plus-sized screen.
On the other hand, apart from the in-built IMUs, the iPhone has a set of components that allow it to track motion without needing it strapped onto the athlete. The components/features we are talking about here are the camera sensors, its powerful processor, the iOS Vision framework (supported by the Core machine learning framework) and the Augmented Reality Kit (ARKit). Basically, stuff that was released since iOS 11.
Some of the commonly known applications that rely on the Vision framework include barcode or QR code reading, document scanning with words or numbers recognition, face detection and tracking, and of course face filters on Instagram. There is also this measuring app called MeasureKit that allows users to measure real-life objects by pointing and moving the camera. Other than measuring things like a ruler, it also does angles, a person’s height, calculate the 3D space an object takes up, etc.
Those who have updated to iOS 12, may have noticed a new Measure app added to their array of apps. The Measure app is like a very basic version of the MeasureKit as it just measures objects like a ruler. But its free and one thing I like about it is that I can take a snapshot of the object and its measurements so I don’t forget. Accuracy wise, I reckon its pretty good and the error is maybe less than 5%.
Dribble Up is a company that created an app for tracking dribbling of a soccer ball. The app requires the user to purchase the Dribble Up branded soccer ball (or Smart Soccer Ball as they call it) and the app will only start tracking the user’s activity when it recognises or has “scanned” the ball. Then there are 3 main activities that the app tracks: Ball Control, Shooting, and Juggling.
Setting up: To use the app, a smartphone stand is recommended to hold the iPhone upright in the landscape orientation. For tracking Ball Control or Juggling, the user has to position the iPhone about 5-10cm off the ground with the screen facing themselves and the ball. Plugging the iPhone to a bigger screen (eg a TV) can give the user much better visual feedback. For tracking shooting into a goal, the user needs to place the iPhone about 3m behind the goal post with the front-facing camera aimed at the goal post. They also need another iPad or iPhone to review the shooting results.
The “smarts”: With Ball Control and Juggling activities, the app recognises and tracks the movement of the Smart Ball using the Front facing camera. The screen provides a visual feedback of exactly what the user is doing with an overlay of metrics and tools (like virtual cones) that help set parameters for each activity.
With the Shooting activity, the user needs to identify the four corners of the goal post by tapping on the screen of the iPhone (that’s placed at the back of the goal post). Then when the activity starts (using the other connected iPhone/iPad), it tracks where the ball lands on the goal. Landing the ball at the top 2 corners are the hardest and therefore gets the user the highest points and the points get lower as the ball lands closer towards the middle of the goal post.
myKicks is an app created by Aussie company Formalytics, that films and tracks penalty kicks. It tracks any size 5 soccer ball and can capture metrics such as the speed of the ball, the trajectory and where the ball landed in the goal. It works with a standard goal post or it could also work if you are kicking against a wall.
Setting up: To use the app, a tripod is recommended to hold the iPhone and it should be placed 14m away from the middle of the goal post or 3m away from the penalty kick position and 6m to the left or right. Unlike the DribbleUp app, myKicks uses the back facing camera of the iPhone to capture the kick. But similar to the DribbleUp app, the user needs to drag a rectangle on the screen to demarcate the goal area.
The “smarts”: Once the iPhone and ball are in place, the user just needs to say “Ready” and the app will call out a countdown for the user to kick the ball. Then when the ball is kicked and hits the goal net, the app recognises that the penalty kick is complete and starts processing the image data. That takes a few seconds and once it is done, it gives a audio feedback of the user’s results. The user can also review details of the kick on the iPhone.
Moving away from soccer, we now look at an app for basketball. HomeCourt is an app that tracks a person’s shooting metrics on an indoor or outdoor basketball court. There is a free portion of the app that allows the user to track up to 300 shots per month and the metrics tracked include where (on the court) the user shoots (attempts) and how many shots they make. They also have a subscription for tracking unlimited shots and a program called “Shot Science” which analyses more metrics of individual shots.
Setting up: There is the option of leaving the iPhone at ground level and leaning it on an object like a water bottle, and the user will be using the front camera. The iPhone should be placed somewhere around half-court and the camera should have a clear view of the court including the hoop and player. There is also the recommended option of securing the iPhone on a tripod at a height of 1.5m. If the ground option is picked, a little calibration procedure is required which is to shoot a free throw. For the tripod set up, the iPhone should also be around the half-court line and have a clear view of the backboard, player and court lines. If the lines are not clear or faded, the user will be asked to drag and position a court template over the camera footage to assist with identification.
The “smarts”: Once the tracking starts, the user can start shooting and the app tracks every time a shot is attempted and identifies the shots that make it through the hoop. It logs the position where each shot is attempted and made. That is the basic free (300 shots) tracking. For the Shot Science subscribers, they get access to shot analysis such as release angle, release time, leg angle, speed, vertical and types of shots. There are hardware requirements for real-time feedback of Shot Science metrics – users need to have the iPhone XS or XR with the A12 processor. Else with iPhones 6s and above, the Shot Science analysis is only available post workout.
Next up is a sprint tracking iPhone app called MySprint. It measures and calculates the complete power-force-velocity profile of an athlete doing a 30m sprint using a method developed and published in the International Journal of Sports Physiology and Performance. They have even validated the app with a radar gun and timing photocells (which is the gold standard) and published it in the European Journal of Sports Science.
Setting up: To ensure accurate measurement, the sprinting track needs to be measured and marked up at 5m intervals up to 30m using 6 vertical markers. The iPhone should be mounted on a tripod and placed 10m away from the sprint track at the 15m mark. The back camera is used to track the athlete. Due to the parallax effect with reference to the iPhone position, the vertical markers’ positions need to be corrected slightly.
The “smarts”: When recording a sprint, a second person is required to point the iPhone at the sprinter and rotate it to ensure that the athlete is in the middle of the screen when recording. After the sprint is recorded, a bit of manual tagging is required which involves selecting the frame when the athlete started the sprint and each time the athlete’s chest or hip touches/crosses the vertical marker. Once that is done, the app processes the data and creates a power-force-velocity profile of the athlete’s sprint.
Three out of the four apps (except MySprint) have a competitive or gamified part to it. They allow users to track their own performance as well as their friends’ performance, and users are ranked with all other users which motivates them to train more. The app also sends out reminders and tips to users to keep them going.
Summary – What the apps all need
[Points of References] One thing these apps all need is/are point/s of references. DribbleUp needs the DribbleUp smart ball and the soccer goal post for the shooting activity, myKicks needs to know the location of soccer goal post corners, HomeCourt looks for the lines on the court and the hoop, and MySprint requires careful placement of the vertical markers and the iPhone.
[Ideal Settings] There are also ideal settings that make the camera tracking more accurate. For example, good lighting, standard equipment sizes, and having no other moving objects (or people) in the background/scene of capture.
[Hardware] They all run on iPhones but some of them require the newer iPhones. DribbleUp needs an iPhone that runs on at least iOS 10 (iPhone 5 or newer), MySprint requires at least the iPhone 5s (with 240fps recording), myKicks and HomeCourt requires the iPhone 6s or newer, and HomeCourt’s Shot Science real-time feedback needs the iPhone XS or XR. The apps also recommend a tripod (with different heights) for mounting the iPhones.
Here’s a table that sums it all up:
When Apple acquired 3D sensor company, Primesense, it was thought that 3D skeleton tracking like the first version of Kinect would be available on new iPhones. Although it didn’t quite go down that path, vision algorithms and more powerful processors still took camera tracking to the next level as we see in the above apps. There might be some extra equipment that’s required and may even need specifically measured placements for accuracy, but what we get is highly accurate and useful sports tracking using a device that many people already own – an iPhone.
In the next couple of months, we will be trying to do more detailed testing and review of the above products. If you are keen to know more, do keep a lookout or subscribe (on the side) to be notified when the reviews are out. With that thanks for reading!