Hudl Assist offers users the ability to submit their game film to Hudl to have expert analysts break down their game stats. Though Hudl's Football Assist offering has largely had positive feedback, we were regularly hearing negative feedback regarding tagging offensive formations. Formations are a tricky breakdown component to get right, as each team has their own methodology and mental model of offensive formations. Currently, analysts tag a formation by plotting points on the video, which is matched to a database of known formations (which do not include backfield or strength), and the default formation is returned to the coach. The coach is them prompted to name the returned diagram, and that name is used when that formation is tagged through the season.
At the end of Football season, Assist Operations conducted multiple surveys to determine the success of Assist, which informed prioritizing offensive formation improvements. When asked which columns users would like to see added, Strength and Backfield were the most frequently requested. Therefore, we knew entering the project that supporting Strength and Backfield would be a key part of our improvements.
Key Research Questions
How do coaches decide what to name a formation? Does this vary between staff members?
What information are coaches communicating via formation names?
Why are formations important to coaches, and what information is dependent upon formations?
How do coaches use formation data for their own team, compared to how they use it when scouting opponents?
Are coaches willing and able to alter how they name formations to get the most out of Hudl Assist?
Most coaches need formation, backfield, and strength broken out individually in order for formations tagged by Assist to be useful, and are willing to do the work or adapt their breakdown techniques to name each separately.
Coaches think about strength, backfield, and formation when thinking of “formation,” even if they don't break those components out explicitly.
Formation naming is highly variable, and how the coach communicates those three data points varies greatly -- but most coaches are communicating strength, backfield, and formation, and the mental model of a singular formation contains that information.
1. Internal and External User Interviews
Neither myself nor the other designer on the project have played or coached Football, and neither of us felt we had enough in-depth knowledge around formations to speak confidently about them to our end users. Therefore, we started our research by interviewing three of our coworkers who are also football coaches. This allowed us to evaluate and iterate on our questions, ensure we’re using the correct language with coaches to uncover the insights we need, and to gain a greater understanding of formations to ensure we could effectively lead external user interviews.
Our external interviews were largely informed by the research objectives, however, we wanted to conduct external user interviews before sending a survey, as we wanted to ensure that the survey confirmed any assumptions we validated during the comparatively small set of user interviews.
The goal of the survey is to capture a large amount of data regarding coach's mental model of formations, and how they install formations on their team. While user interviews help us understand the issue in depth, how teams use formations is highly variable. We wanted a large amount of data to back up our assumptions and key takeaways from user interviews, as interviewing enough users to feel confident about established tendencies among our tens of thousands of users was outside of scope.
3. Data Analysis
As Hudl supports 98% of High School Football teams in the country, we have a large amount of existing data that we can manipulate and use to inform our assumptions and conclusions. I ran multiple queries to better understand some trends within the existing data, and help inform our research objectives.
I compared the amount of Assist formation names that included an indicator of strength (left/L/LT, right/R/RT, Strong, Weak) to the number of non-Assist formation names that included an indicator of strength over the same time period. As Assist users were explicitly told not to include Strength in their formation names, I was seeking to determine how amenable coaches are to changing their workflows around formation naming.
Of the 23,217 Assist Formations named between 1 Sept 2017 and 15 Sept 2017, 7,545 or 32.5% had a strength indicator.
Of the 563,359 non-Assist Offensive Formation column data values filled out between 1 Sept 2017 and 15 Sept 2017, 263719, or 46.8% had a strength indicator.
Number of formation names that included common strength indicators (left/L/LT, right/R/RT, Strong, Weak), to determine the percentage of coaches that consider strength a part of the formation name, as opposed to a separate column (as Hudl has columns available for offensive formation, strength, and backfield).
As mentioned above, 46.8% of formations included strength indicators.
Number of coaches who use the offensive formation column alone, offensive formation + strength column, offensive formation + backfield column, or all three.
Based on our interviews, our assumptions that coaches are amenable to changing their workflows when the benefits of doing so is laid out for them was proven to be correct. Though each coach has their own mental model of a formation name that may or may not include the strength in the name itself, most coaches don’t have specific reasons for the mental model they use (“that’s how my first coaching job approached it” was a common sentiment), and are used to having to change their workflows due to other circumstances anyway, such as implementing a new offense or coaching staff changes.
The data analysis corroborates the interviews, as 14.3% fewer formations included possible strength indicators when a user was told not to include them. However, that was without any education or training regarding the benefits of not including those indicators — future testing will include that education to validate the assumption that including it will increase adoption rates of the Hudl formation naming schemes. Therefore, we feel confident approaching Assist formation naming from the perspective to provide naming schemes that provide the most value, rather than trying to fit into highly variable existing naming schemes.