Kernel Methods

Implementation

We implemented the kernelized ridge regression (KRR) and Support Vector Regression (SVR). To go along with the models we implemented three different kernels:

• Polynomial kernel

• RBF kernel

• Linear kernel

We used the implementations on sine data. We could observe that the choice of the parameters greatly affected the fit. With SVR we had to be careful to manage the number of support vectors. We used parameters in Table1 and were able to achieve results seen on Figure6

Where $M$ is the degree of the polynomial and $c$ is the coefficient in the polynomial.

We observed that the RBF kernel achieved an excellent fit to the data, while the Polynomial kernel struggled to capture the underlying pattern effectively.

Fitting the housing data

We trained our model to fit the housing data. The combinations of the kernels and methods stayed the same, but this time we used nested-cross-validation (NCV) to determine the best parameter $\lambda$. We also fixed $\lambda=1$ and compared the two results. To determine the $\epsilon$ parameter for SVR, we made tried to fit the model on some samples and looked at the number of the support vectors. Observing the results in 2 we can see that the parameter did not change the number so we fixed it at $\epsilon = 0.5$

Because computation of lots of parameter values was not possible we evaluated the models with a simple train-test split to get a feel for what values of parameters to explore more deeply. The results can be observed on Figure 1 for KRR and Figure 2 for SVR.

Thanks to this simple exploration we were able to confidently chose parameters $M \in [1,10]$, $\sigma \in \{80,100,120\}$ and $\lambda \in \{0.01, 0.1, 1, 10\}$. We started with leave-one-out NCV but because of computational reasons we abandoned that approach and used a 10-fold NCV.

The results can be observed on Figures 3, 4 and 5. We can see that the NCV approach outperforms the fixed $\lambda$ model, and would likely perform even better if we could check even more values for $\lambda$. We can see that with polynomial kernel, the number of support vectors increases as the degree increases where with RBF kernel the sigmas do not have any effect, but the fixed lambda yields less vectors.