Žan Pušenjak
Studies / Research

Generalized Linear Models

By Žan Pušenjak

Machine Learning for Data Science, FRI, University of Ljubljana


machine-learning
View PDF

Implementation

We implemented both models by minimizing their log-likelihood with the help of the scipyscipy function fmin_l_bfgs_bfmin\_l\_bfgs\_b1.

To achieve accurate interpretability, we used the first class as reference class. With that we achieved the right number of parameters to optimize:

  • Multinomial (M−1)K(M-1)K

  • Ordinal (M−2)+K(M-2) + K

Where MM is the number of classes and KK is the number of features.

Application of the Multinomial Regression

We used the provided data set of basketball shot types2 and our model implementation to provide some insight into the relationship between the ShotType feature and other features.

The values of beta parameters on the provided dataset

We additionally prepared the data by one hot encoding the categorical features Competition, PlayerType and Movement and normalizing the numerical features Distance and Angle. Finally we removed highly correlated features (features with correlation coefficient >0.7>0.7), which allowed us to better interpret the final parameters.

We used the β\beta parameter matrix as the main instrument for our findings. A β\beta parameter represents the log-odds of observing the class as opposed to observing the reference class. So a higher β\beta value will increase those odds for the class and a lower value will decrease them. Our reference class was above-head

Observing Figure 1 we selected the highest and lowest β\beta and tried to find any explainable relations.

Competition U14

We can observe that two β\beta parameters for the Competition_U14 feature are negative, values being:

  • dunk: β=−19.88±3.98\beta = -19.88 \pm 3.98

  • tip-in: β=−17.32±3.41\beta = -17.32 \pm 3.41

So if the shot was recorded in a U14 basketball game the chances of observing a dunk or a tip-in rather than an above-head shot are significantly decreased. This is a logical explanation since the players at that age are usually not tall enough to perform a shot of that type and mostly throw above-head.

Movement

The Movement_drive and Movement_no features positively influence the hook-shot, layup and other types, with the exact parameter values being:

  1. for Movement_drive

    • layup: β=11.27±1.67\beta = 11.27 \pm 1.67

    • other: β=11.70±1.66\beta = 11.70 \pm 1.66

  2. for Movement_no

    • dunk: β=9.06±1.94\beta = 9.06 \pm 1.94

    • hook-shot: β=11.02±2.33\beta = 11.02 \pm 2.33

    • tip-in: β=8.31±1.67\beta = 8.31 \pm 1.67

This tells us that the chances of observing a layup or other shot type instead of above-head will increase if there is drive movement present. The unexpected results that we could not interpret were that if there is no movement present a dunk, hook-shot or tip-in will occur more regularly.

On the other hand

Distance

It is apparent the Distance feature was the most important feature on some classes. This is expected, as the further the player is from the rim, the less likely he is to perform a tip-in, dunk or layup, in that order, and that reflects on the β\beta parameters.

  • tip-in: β=−37.34±3.86\beta = -37.34 \pm 3.86

  • dunk: β=−25.97±2.97\beta = -25.97 \pm 2.97

  • layup: β=−16.12±1.15\beta = -16.12 \pm 1.15

Application of the Ordinal Regression

We made some synthetic data using a data generating process (DGP) of ordinal value to show that the ordinal regression model will outperform the normal multinomial regression with suck DGP. We used accuracy as the metric for comparison. The performance of the models can be observed on Figure 2 where we observe that the ordinal regression clearly outperforms the multinomial one. We can also see that increasing the data set does not attribute to the results.

Performance on an ordinal dataset at different data set sizes.

Even when increasing the data set size to 1000010000 the ordinal regression still achieved a better accuracy seen in Table 1.

Accuracy of ordinal vs multinomial regression with ordinal data set of size 1000010000.
1-2 Model Accuracy
Ordinal reg. 0.59±0.010.59 \pm 0.01
Miltinomial reg. 0.64±0.010.64 \pm 0.01

  1. https://docs.scipy.org/doc/scipy/reference/generated/
    scipy.optimize.fmin_l_bfgs_b.html↩︎

  2. https://journals.plos.org/plosone/article?id=10.1371/
    journal.pone.0128885↩︎