Analysts don’t like the “why” questions. They are tough to answer. For instance, in a help desk analysis, it is easy to show which tickets are resolved faster. But it is difficult to say why. In my practice in Sopra Steria I found that one tool is especially robust to deal with practical “why” questions: the Chi-Square test for independence.

Answering the “why”: the traditional way

Even for a statistician, it is not immediately clear how a Chi-Square test (which I will cover later) could help to answer the “why” questions. To demonstrate how I use it, I need to first explain the alternative way. Here is an example.

I want to know why some help desk tickets take long time, while some others take short time. This is depicted in the histogram below. I am interested in the spike peak (a spike, a hump) to the very left, representing a group of tasks that are being solved below 1 minute, which is surprisingly quick. I want to know why.

To discover the reason for that peak, I should divide the data into two sets: records within the peak (blue below), and all the remaining records (orange below). Then I will test some ideas that come to mind. Perhaps some agents (service desk employees) solve problems quicker, because they are more experienced or more effective? Let’s visualize the distribution of tasks sorted by the agent name within the peak, and outside the peak:

The outcome is interesting. When it comes to the number of tasks closed below 1 minute, the agent Anton is clearly the leader. This way we quickly identified a hint, which can lead to the potential reason for the peak in data.

Why the traditional way does not work

Unfortunately in real life, such reasoning is not that easy. The “agent name” is an example of the data feature. The real data will have hundreds of features. Each of those features will have hundreds or thousands of possible values. Guessing which feature is responsible for the peak is tedious and impractical. One would need to browse through hundreds of charts looking like this:

And here is where the Chi-Square test for independence (also known as chi2, described in the previous article) can shine. Here is rendering of a Jupyter notebook demonstrating the power of chi2 on a different example.

The reason I use Jupyter notebooks is that the technical readers can read the source code to know exactly what I was doing. Non-technical readers can simply skip the source code sections, and still understand the concept.

import pandas as pd, numpy as np, chi2_util, loghist # https://github.com/altanova/stuff
import matplotlib.pyplot as plt, seaborn as sns
from sklearn.preprocessing import LabelEncoder
sns.set()
df = pd.read_csv("sample.csv")

In [2]:

# this field represents minutes
df['minutes'] = df['G']
fig, ax = plt.subplots(1,1, figsize = (15,3))
loghist.lhist(xticks = 'minutes',axis = ax, df = df, field = 'G', annotate = 'statistics')
plt.tight_layout()
plt.show()

# G represents minutes
df['hours'] = df.minutes / 60
df['group'] = df.apply(lambda r: (r.hours >= 143.5) & (r.hours <= 148), axis = 1)
df['group'] = df['group'].astype(int)

In [4]:

fig, ax = plt.subplots(1,1, figsize = (15,3))
loghist.lhist(xticks = 'minutes',axis = ax, df = df, field = 'minutes', annotate = 'statistics', 
              group_field = 'group')
plt.tight_layout()
plt.show()

orig = df.copy()
features = ['A', 'B', 'C', 'D', 'E', 'F', 'H', 'I', 'J']
dfcn = df[features].fillna('EMPTY')
dfce = dfcn.apply(LabelEncoder().fit_transform)
dfchi = dfce.merge(orig['group'], 
           left_index = True, 
           right_index = True, 
           how = 'inner')

# run chi2. For technical details, see the previous articles on ondata.blog, 
# or https://github.com/altanova/stuff
chi2_util.chi2_score(dfchi, 
                     features =  features, 
                     target = 'group', 
                     alpha = 0.05,
                     deep = True,
                    verbose = False)[:3]

piv = df.pivot_table(columns = 'group', 
               index = 'J',   # feature that interests us
               values = 'A',  # any feature
               aggfunc = len).sort_values(by = 1, ascending = False)
piv[:5]

# hypothesis: the reason for the spike is the 'phased out' code.
top_values = piv.index[0:1].tolist()
df['reason'] = df.apply(lambda r: int(r['J']) in top_values, axis = 1)

In [10]:

fig, ax = plt.subplots(1,1, figsize = (15,4))
loghist.lhist(xticks = 'minutes',axis = ax, df = df, field = 'minutes', annotate = 'statistics', 
              group_field = 'reason')
plt.tight_layout()
plt.show()

Summary

The Chi-Square test does not directly answer the why question, but it provides hints speeding the reasoning.

In our case, we found the following hint: the spike consists mainly of tickets, whose “code” field has value “duplicate”. What does this really mean?

The business interpretation may be as follows. Certain tasks are marked duplicate very early in their lifetime. Then, after a certain retention period of inactivity, an automatic process transfers tasks to the closed state. So in fact, we are not seing a spike in activity after 6 days, but a disproportional number of fake tasks that are automatically closed after 6 days. Quite an important insight for the business owner of the project.

With the help of chi2 test, this conclusion could be reached quickly. We could efficiently establish the feature that correlated with the spike: the task code. Remember it was one of 300 possible choices. An example of another possible choice (feature), mentioned earlier in the article, would be the agent name. This time however, if we tried to see which agent is responsible for the spike, we would fail because the spike was simply not related to that feature. We would then need to repeat the same for 100+ other features, wasting hours of work. The chi2 test made this quicker.

Notes and credits

Final note: the reasoning above is simplified. The chi square test is not universal. It only works on categorical features, on certain conditions. Also, it is part of a larger family of statistical tests for independence, not to be discussed here. To read more: [don’t trust the data], [unmasking data in camouflage], [isolating the spikes]. The subject is quite broad. The goal of this material is to demonstrate how an advanced statistical tool, such as the chi square test for independence, can be used to efficiently address a practical problem.

Credits: creating this material was possible thanks to my Data Science work for Sopra Steria. We always look for talent. If interested, [click here to contact our recruitment].