Introduction to Open Data Science

University of Helsinki, spring 2017

• Tuomo Nieminen and Emma Kämäräinen with
• Adjunct professor Kimmo Vehkalahti

Powered by Rpresentation. The code for this presentation is here

1. Regression and model validation
   
2. Logistic regression
   
3. Clustering and classification
   
4. Dimensionality reduction techniques
   

For IODS by Tuomo Nieminen

Powered by Rpresentation. The code for this presentation is here.

A statistical model:

• Embodies a set of assumptions and describes the generation of a sample from a population
  
• Represents the data generating process
  
• The uncertainty related to a sample of data is described using probability distributions
  

Linear regression is an approach for modeling the relationship between a dependent variable \( \boldsymbol{y} \) and one or more explanatory variables \( \boldsymbol{X} \).

There are many applications for linear models such as

• Prediction or forecasting
  
• Quantifying the strength of the relationship between \( \boldsymbol{y} \) and \( \boldsymbol{x} \)
  

In linear regression, it is assumed that the relationship between the target variable \( \boldsymbol{y} \) and the parameters (\( \alpha \), \( \boldsymbol{\beta} \)) is linear:

\[ \boldsymbol{y} = \boldsymbol{\alpha} + \boldsymbol{X} \boldsymbol{\beta} + \boldsymbol{\epsilon} \]

• The goal is to estimate the parameters \( \alpha \) and \( \boldsymbol{\beta} \), which describe the relationship with the explanatory variables \( \boldsymbol{X} \)
  
• An unobservable random variable (\( \boldsymbol{\epsilon} \)) is assumed to add noise to the observations
  
• Often it is reasonable to assume \( \boldsymbol{\epsilon} \sim N(0, \sigma^2) \)
  

In the simple linear equation \( \boldsymbol{y} = \alpha + \beta \boldsymbol{x} + \boldsymbol{\epsilon} \)

• \( \boldsymbol{y} \) is the target variable: we wish to predict the values of \( \boldsymbol{y} \) using the values of \( \boldsymbol{x} \).
  
• \( \alpha + \beta \boldsymbol{x} \) is the systematic part of the model.
  
• \( \beta \) quantifies the relationship between \( \boldsymbol{y} \) and \( \boldsymbol{x} \).
  
• \( \boldsymbol{\epsilon} \) describes the errors (or the uncertainty) of the model
  

When the model is \[ \boldsymbol{y} = \alpha + \beta_1 \boldsymbol{x}_1 + \beta_2 \boldsymbol{x}_2 + \boldsymbol{\epsilon} \]

• The main interest is to estimate the \( \boldsymbol{\beta} \) parameters
  
• Interpretation of an estimate \( \hat{\beta_1} = 2 \):
  
  • When \( x_1 \) increases by one unit, the average change in \( y \) is 2 units, given that the other variables (here \( x_2 \)) do not change.
    

A statistical model always includes several assumptions which describe the data generating process.

• How well the model describes the phenomenom of interest, depends on how well the assumptions fit reality.
  
• In a linear regression model an obvious assumption is linearity: The target variable is modelled as a linear combination of the model parameters.
  
• Usually it is assumed that the errors are normally distributed.
  

Analyzing the residuals of the model provides a method to explore the validity of the model assumptions. A lot of interesting assumptions are included in the expression

\[ \boldsymbol{\epsilon} \sim N(0, \sigma^2) \]

• The errors are normally distributed
  
• The errors are not correlated
  
• The errors have constant variance, \( \sigma^2 \)
  
• The size of a given error does not depend on the explanatory variables
  

Leverage measures how much impact a single observation has on the model.

• Residuals vs leverage plot can help identify which observations have an unusually high impact.
• The next two slides show four examples.
  
• Each row of two plots defines a data - model validation pair.

For IODS by Tuomo Nieminen

Powered by Rpresentation. The code for this presentation is here.

In regression analysis, the target variable \( \boldsymbol{Y} \) is modelled as a linear combination of the model parameters and the explanatory variables \( \boldsymbol{X} \)

\[ \boldsymbol{Y} = \boldsymbol{\alpha} + \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon} \]

Another way to express this is to use conditional expectation

\[ E[\boldsymbol{Y} \mid \boldsymbol{X}] = \boldsymbol{\alpha} + \boldsymbol{X}\boldsymbol{\beta} \]

So, linear regression is a model for the (conditional) expected value of Y.

If the target variable \( Y \) is binary, taking only the values

• 0 (“failure”)
• 1 (“success”)

with probability \( p \), then \( E[Y] = p \).

• The goal in logistic regression is to define a linear model for the probability of “success” (\( p \), the expected value of \( Y \))
• The problem is that \( p \) only takes on values between 0 and 1
• A possible predictor can take on any value. There is no way to use multiplication and addition to restrict the predictors values to the range of \( p \). What to do?

The ratio of two odds is called the odds ratio. It can be computed by the following steps:

1. Compute the odds of “success” (\( Y = 1 \)) for individuals who have the property \( X \).
2. Compute the odds of “success” (\( Y = 1 \)) for individuals who do not have property \( X \).
3. Divide the odds from step 1 by the odds from step 2 to obtain the odds ratio (OR).

Odds ratio can be used to quantify the relationship between \( X \) and \( Y \). Odds higher than 1 mean that \( X \) is positively associated with “success”.

From the fact that the computational target variable in the logistic regression model is the log of odds, it follows that applying the exponent function to the fitted values gives the odds:

\[ \exp \left( log\left( \frac{\hat{p}}{1 - \hat{p}} \right) \right) = \frac{\hat{p}}{1 - \hat{p}}. \]

The exponents of the coefficients can be interpret as odds ratios between a unit change (vs no change) in the corresponding explanatory variable.

\[ exp(\hat{\beta_x}) = Odds(Y \mid x + 1) / Odds(Y \mid x) \]

https://prateekvjoshi.files.wordpress.com/2013/06/cross-validation.png

A statistical model can be used to make predictions. An intuitive way of comparing different models is to test their predictive power on unseen data.

• The available data can be split into training and testing sets
• Only the training data is used to find the model
• The testing data is then used to make predictions and evaluate the model performance

In order to assert model performance, we need to measure it somehow.

• Depending on the nature of the target variable, different measures might make sense.
• If the task is binary classification such as in logistic regression, it is straight forward to calculate the proportion of correctly classified observations
• The proportion of incorrectly classified observations is the error (penalty, loss)

Cross-validation is a powerful general technique for assessing how the results of a statistical analysis will generalize to an independent data set.

• Utilizes the idea of training and testing sets effectively
• Mainly used in settings where one wants to estimate how accurately a predictive model will perform in practice
• Gives a reasonable measure of performance on unseen data
• Can be used to compare different models to choose the best performing one

One round of cross-validation involves

• Partitioning a sample of data into complementary subsets
• Performing the analysis on one subset (the training set, larger)
• Validating the analysis on the other subset (the testing set, smaller).

This process is repeated so that eventually all of the data is used for both training and testing.

For IODS by Emma Kämäräinen

Powered by Rpresentation. The code for this presentation is here.

• Linear discriminant analysis
• Distance measures
• K-means

Linear discriminant analysis (LDA) is a classification method. It can be used to model binary variables, like in logistic regression, or multiple class variables. The target variable needs to be categorical.

It can be used to

• Find the variables that discriminate/separate the classes best
• Predict the classes of new data
• Dimension reduction (not covered here)

This is a good and simple blog post about LDA. R-Bloggers also have a post about LDA, see it here.

Linear discriminant analysis produces results based on the assumptions that

• variables are normally distributed (on condition of the classes)
• the normal distributions for each class share the same covariance matrix

Because of the assumptions, the data might need scaling before fitting the model. The variables also need to be continuous.

Let's see an example next to wrap our heads around what LDA is really doing.

Classifying new observations:

• Based on the trained model LDA calculates the probabilities for the new observation for belonging in each of the classes
• The observation is classified to the class of the highest probability
• The math behind the probabilities can be seen here for those who are interested. Bayes theorem is used to estimate the probabilities.
• You'll see how the predicting is done in the DataCamp exercises.

How to determine if observations are similar or dissimilar with each others?

• Euklidean distance
• Manhattan/Taxicab distance (axis-aligned directions)
• Jaccard index (binary/categorical distance measure)
• Hamming distance (distance measure for words/strings)

• K-means is possibly the oldest and used clustering method in many fields of study
• Pro: Easy to use and often finds a solution
• Con: Small change in the dataset can produce very different results
• Many variations of k-means: k-means++, k-medoids, k-medians…

1. Choose the number of clusters you want to have and pick initial cluster centroids.
2. Calculate distances between centroids and datapoints.
3. For all the data points: Assign data point to cluster based on which centroid is closest.
4. Update centroids: within each cluster, calculate new centroid
5. Update clusters: Calculate distances between data points and updated centroids. If some other centroid is closer than the cluster centroid where the data point belongs, the data point changes cluster.

Continue updating steps until the centroids or the clusters do not change

Remarks about k-means:

• Distance measure in the algorithm:
  • Different distance measures produce different output
  • Deciding the best distance is not always easy
• Number of clusters as input
  • Many ways to find the optimal number of clusters
  • One way is to look at the total within cluster sum of squares (see next slide)
  • Other ways: hierarchical clustering, silhouette method, cross validation …

Total within sum of squares is calculated by adding the within cluster sum of squares (WCSS) of every cluster together. The WCSS can be calculated with the pattern

\( WCSS = \sum_i^N (x_i - centroid)^2 \)

So you are searching for the number of clusters, where the observations are closest to the cluster center.

For IODS by Tuomo Nieminen & Emma Kämäräinen

Powered by Rpresentation. The code for this presentation is here

In statistical analysis, one can think of dimensionality as the number of variables (features) related to each observation in the data.

• If each observation is measured by \( d \) number of features, then the data is \( d \) dimensional. Each observation needs \( d \) points to define it's location in a mathematical space.
• If there are a lot of features, some of them can relate to the same underlying dimensions (not directly measured)
• Some dimensions may be stronger and some weaker, they are not equally important

The original variables of high dimensional data might contain “too much” information (and noise or some other random error) for representing the underlying phenomenom of interest.

• A solution is to reduce the number of dimensions and focus only on the most essential dimensions extracted from the data
• In practise we can transform the data and use only a few principal components for visualisation and/or analysis
• Hope is that the variance along a small number of principal components provides a reasonable characterization of the complete data set

On the linear algebra level, Singular Value Decomposition (SVD) is the most important tool for reducing the number of dimensions in multivariate data.

• The SVD literally decomposes a matrix into a product of smaller matrices and reveals the most important components
• Principal Component Analysis (PCA) is a statistical procedure which does the same thing
• Correspondence Analysis (CA) or Multiple CA (MCA) can be used if the data consists of categorical variables
• The classification method LDA can also be considered as a dimensionality reduction technique

In PCA, the data is first transformed to a new space with equal or less number of dimensions (new features). These new features are called the principal components. They always have the following properties:

• The 1st principal component captures the maximum amount of variance from the features in the original data
• The 2nd principal component is orthogonal to the first and it captures the maximum amount of variability left
• The same is true for each principal component. They are all uncorreleated and each is less important than the previous one, in terms of captured variance.

Unlike LDA, PCA has no criteria or target variable. PCA may therefore be called an unsupervised method.

• PCA is sensitive to the relative scaling of the original features and assumes that features with larger variance are more important than features with smaller variance.
  
• Standardization of the features before PCA is often a good idea.
• PCA is powerful at encapsulating correlations between the original features into a smaller number of uncorrelated dimensions

PCA is a mathematical tool, not a statistical model, which is why linear algebra (SVD) is enough.

• There is no statistical model for separating the sources of variance. All variance is thought to be from the same - although multidimensional - source.
• It is also possible to model the dimensionality using underlying latent variables with for example Factor Analysis
• These advanced methods of multivariate analysis are not part of this course

A biplot is a way of visualizing two representations of the same data. The biplot displays:

(1) The observations in a lower (2-)dimensional representation

• A scatter plot is drawn where the observations are placed on x and y coordinates defined by two principal components (PC's)

(2) The original features and their relationships with both each other and the principal components

• Arrows and/or labels are drawn to visualize the connections between the original features and the PC's.

In a biplot, the following connections hold:

• The angle between arrows representing the original features can be interpret as the correlation between the features. Small angle = high positive correlation.
• The angle between a feature and a PC axis can be interpret as the correlation between the two. Small angle = high positive correlation.
• The length of the arrows are proportional to the standard deviations of the features

Biplots can be used to visualize the results of dimensionality reduction methods such as LDA, PCA, Correspondence Analysis (CA) and Multiple CA.