Without continual growth and progress, such words as improvement, achievement, and success have no meaning.
Benjamin Franklin
Tracking studies fielded over iterative waves will always have their place as a “tried and true” core research methodology. In addition to its perceived importance, time is a dimension that is unique to tracking studies. A “one off” study of attitudes and usage reflects respondents’ perspectives at a specific point in time. Furthermore, if we ask respondents to recall historical behavior so that we may evaluate trends in this behavior, chances are the respondent cannot recall the behavior, or even worse, provides an inaccurate recollection in his or her responses. If we want to look at a trend, the accuracy of the trend is critical… and tracking studies provide a mechanism for ensuring the validity of key indicators over time.
However, with the value of research being scrutinized by Marketing Executives more than ever before, the value of tracking studies has been under increasing fire. And, who can blame the Execs??? If we as researchers miss noteworthy trends in key tracking measure indicators over repeated waves, we have no one to blame but ourselves. In such a scenario, non-research stakeholders’ interpretation will be either a) indicators are unchanged, therefore calling into question the value of repeated measures for the study, or b) fluctuations may be due to “noise” in the data (instead of a statistically significant trend), which undermines the validity of the overall research methodology…
I feel strongly that better recognition of trends over time must take center stage in tracking studies. So what does that really mean, you ask? Is it simply a case of throwing together more line graphs to graphically depict key indicator changes from quarter to quarter?
Hardly. Typically, researchers will show quarter to quarter changes in this fashion, but it is not enough. We must take the next step, and provide key insight into what, if any, trends are conclusive from the measured time-series data. Establishing how confident we are in observed positive/negative trends is no joke – it is part of being a responsible Researcher/Analyst/Marketing Scientist. We do not want our clients or internal customers to observe a slightly upward trending pattern in time-series data, and mistakenly conclude that a trend definitively exists.
So, without getting too technical (of course… since I want you to keep reading my blog), there are two key steps to providing a robust time-series analysis of tracking data:
1. Request that an Analyst/Marketing Scientist build a time-series model that provides an appropriate fit of the repeated measure indicator data across at least 4 waves.
2. Request that the same Analyst/Marketing Scientist compute the confidence level % that reflects how confident a stakeholder should be that a positive (or negative) trend is being observed in repeated measure indicator data.
The Analyst/Marketing Scientist needs to provide a model that, minimally, generates a linear trend line that passes centrally through the repeated measure indicator data (this “linear” approach is the simplest form of time-series trend estimation). An example of a linear trend line generated by a time series model is depicted below:
The dashed trend line in this graph has a slope value associated with it. A positive slope value describes an upward trend line, and a negative slope value describes a downward trend line. The Analyst/ Marketing Scientist needs to test whether this slope value is significantly different from a value of zero (e.g., representing a “flat trend line”), and generate a level of confidence representing how confident we should be that a positive (or negative) trend does indeed exist. This second step test and confidence computation is critical to properly interpret the trend. Typically, a 95% confidence level is used to determine whether a difference is cited, but I am proposing that we observe and consider the computed confidence level value itself, and not constrain our interpretations to whether a 95% confidence level threshold is reached or not.
Also, both Researchers and Analysts need to be cognizant of seasonal and non-seasonal cyclical components to time series data, as these factors will impact the precision of the estimated trend line. If such factors are suspected, the analyst can build component estimates for these factors into the time series model. These factors typically become apparent, if they exist, beginning after 12 months of data collection, and truly take form typically after 24 – 36 months of data collection. The lengthier the time frame across which data has been collected… the more obvious both seasonal and non-seasonal cyclical components will be.
To support your efforts, the details of conducting an appropriate times series analysis and confidence test are readily available in elementary statistics textbooks and various sites across the web – if should want a more detailed take on this approach. The link below serves as a great primer on time-series analysis for Analysts, covering the different approaches to modeling times series, from a straight-forward least-squares linear model, to more sophisticated models that address auto-regressive processes (Yikes!!!).
//statistik.mathematik.uni-wuerzburg.de/timeseries/index.php?id=book
But for the purposes of this blog, the point is clear – if you want to secure budget for either a brand new tracking study or another year of a current tracking study, you need to uncover unique insight that speaks across the time horizon of the study. A time-series analysis and confidence level computation to support trend-based statements is a great place to start!
What’s your take on adding value to tracking studies? How much value do you feel applying time-series analysis to examine respondent trends over iterative waves of a study brings to stakeholders? We’d love to hear from you – click the “Comment” button for this entry and let us know what you think!
~ Marketing Workshop