Predicting Student Academic Performance Using Regression Models: A Comparative Study on the Role of Prior Grades

Abstract

Problem Statement & Objectives

1.1 Problem Statement

1.2 Objectives

The objectives of this project were:

• Load and explore the UCI Student Performance Dataset (Portuguese subset) to understand its structure, distribution, and quality.
• Clean and preprocess the data, handle categorical features using Label Encoding and scale the features using StandardScaler.
• Build and train three regression models: Linear Regression, Decision Tree Regressor, and Random Forest Regressor.
• Evaluate model performance using Root Mean Square Error (RMSE) and R².
• Analyse feature importance to understand which variables are most influential in predicting final grades.
• Perform 5-fold cross-validation to assess whether the model generalises well across different subsets of the data.
• Conduct a second experiment by removing G1 and G2 from the feature set, and compare results to evaluate how much the previous grades contribute to predictive performance.

Conceptual Explanations

2.1 Key Regression Models Used

Three regression algorithms were used in this project. A brief explanation of each is provided below.

2.1.1 Linear Regression

Linear Regression is one of the simplest and most interpretable machine learning models. It assumes that the output variable can be modelled as a weighted sum of the input features plus a bias term. Despite its simplicity, linear regression can perform very well when the relationship between inputs and outputs is approximately linear. In this project, because G1 and G2 are strongly linearly correlated with G3, linear regression turned out to be competitive with more complex models.

2.1.2 Decision Tree Regressor

Decision Tree Regressor works by recursively partitioning the feature space into smaller regions, assigning a predicted value (usually the mean of training samples in that region) to each leaf. While decision trees can model nonlinear relationships, they are prone to overfitting, especially when the data is noisy or the tree is allowed to grow deep without regularisation. This characteristic was visible in this project, where the Decision Tree consistently underperformed relative to the other two models.

2.1.3 Random Forest Regressor

Random Forest is an ensemble method that builds many decision trees on bootstrapped subsets of the training data, using a random subset of features at each split. The predictions of all trees are averaged to produce the final output. This process reduces the variance that individual decision trees suffer from, generally making Random Forest more robust. An additional benefit is that Random Forest provides feature importance scores, which were used in this project to identify which variables contributed most to predicting G3.

2.2 Evaluation Metrics

Two metrics were used to evaluate the models.

• Root Mean Square Error (RMSE) measures the average magnitude of prediction errors. It is expressed in the same units as the target variable (grade points), making it easy to interpret. A lower RMSE indicates that predictions are closer to the actual values.
• R² (R-squared) measures the proportion of variance in the target variable that the model explains. A value of 1.0 means perfect prediction. A value of 0 means the model is no better than predicting the mean grade for every student. A negative value, as was observed for the Decision Tree in the second experiment, means the model is actually worse than that baseline.

Dataset Description

3.1 Source

The Portuguese language dataset (student-por.csv) was selected for this project because it contains 649 records, compared to 395 in the Mathematics dataset. A larger sample size allows the models to learn from more examples, which generally leads to more reliable evaluation results.

3.2 Target Variable

The target variable for all models is G3, the final grade for the academic year. Grades in this dataset are on a scale from 0 to 20. The dataset also contains G1 (first period grade) and G2 (second period grade), which represent earlier assessments during the same academic year.

3.3 Feature Overview

The dataset contains 33 features in total covering a wide range of student attributes.

Implementation

4.1 Development Environment

The project was implemented in Python using Jupyter Notebook. The following libraries were used:

• Pandas, for data loading, exploration, and manipulation
• NumPy, for numerical computation
• Matplotlib, for data visualisation
• Seaborn, for statistical visualisations including correlation heatmaps
• Scikit-Learn, for preprocessing, model training, evaluation, and cross-validation

4.2 Overall Project Pipeline

The project followed a standard supervised learning pipeline:

• Data loading using pandas read_csv
• Data exploration using head(), info(), describe(), shape, isnull().sum() to understand the data structure and check for missing values
• Exploratory Data Analysis (EDA) through visualisations
• Preprocessing: Label Encoding for categorical variables followed by StandardScaler for feature scaling
• Experiment 1: Training and evaluating all three models using all features including G1 and G2
• Feature importance analysis using the Random Forest model
• Cross-validation using 5-fold CV on the Random Forest model
• Experiment 2: Repeating training and evaluation after dropping G1 and G2 from the feature set
• Comparative analysis of results from both experiments

4.3 Preprocessing Choices

The data was split into training and test sets using an 80:20 ratio. A random_state of 50 was used for both Experiment 1 and Experiment 2.

Data Exploration & EDA

5.1 Basic Exploration

Initial exploration of the dataset was performed using standard pandas methods. The dataset contains 649 rows and 33 columns. The isnull().sum() check confirmed that there are no missing values in any of the columns. This meant no imputation or row-dropping was necessary before preprocessing.

The describe() output showed G3 ranging from 0 to 19 and an average score of 11.91. Most students scored between 10 and 14, indicating moderate overall academic performance. Student absences ranged from 0 to 32, with an average of 3.66 absences, and the majority of students had relatively low absence counts. The data also shows that most students had no previous academic failures, suggesting that severe academic difficulties were uncommon within the dataset.

5.2 Distribution of Final Grades

The histogram of G3 values shows how final grades are distributed across the 649 students. The majority of students scored in the range of 9 to 15, with the highest concentration around 9 to 11. A notable feature of the distribution is the presence of a small cluster of students who scored 0, indicating a small number of students who received a final grade of zero.

The distribution is broadly unimodal, with most grades concentrated in the middle range and fewer students at the extreme ends. Very few students scored below 5 or above 18, suggesting that most students achieved moderate academic performance.

5.3 Study Time vs Final Grade

Looking at the spread of points in each category, the relationship is not particularly strong. Students in all four study time groups achieved grades across a wide range of values. There is a slight tendency for students in higher study time categories to achieve higher grades more frequently, but the pattern is not strong enough to conclude that study time alone determines academic performance. The scatter plot suggests that study time has only a limited influence on final grade when considered by itself.

5.4 Absences vs Final Grade

The absences scatter plot shows a wide spread of data points, with most students having between 0 and 15 absences and only a few students recording more than 20 absences. Students with both high and low grades can be found throughout the lower absence range.

While there is a slight tendency for grades to decrease as absences increase, the relationship is weak and contains many exceptions. Some students with very few absences scored poorly, while others with relatively high absences still achieved good grades. This suggests that absences alone are not a strong predictor of academic performance.

5.5 Correlation Heatmap (All Features)

The full correlation heatmap shows pairwise Pearson correlations between all numerical variables in the dataset. The most visually striking feature of the heatmap is the strong positive correlation cluster in the bottom-right corner, corresponding to G1, G2, and G3. These three grade variables are clearly highly correlated with one another.

Other notable patterns include a moderate negative correlation between failures and grades, meaning students with more past failures tended to achieve lower final grades. Parental education levels show a mild positive correlation with grades, while social variables exhibit weak negative relationships with academic performance.

5.6 Correlation Between G1, G2, and G3

A focused heatmap was created showing only the correlation between the three grade variables. The annotated values are:

These are very high correlations. G2, being the most recent grade before the final assessment, has the strongest relationship with G3 at 0.92. This observation was important because it shaped the design of the two experiments and became central to interpreting the feature importance results.

Model Building

6.1 Experiment 1: Prediction Using All Features

Experiment 1

In the first experiment, the feature set X consisted of all columns except G3. The label Y was G3. After preprocessing with StandardScaler, the data was split 80:20 for training and testing (random_state=50). Three models were trained.

6.2 Experiment 2: Prediction Without G1 and G2

Experiment 2

In the second experiment, G1 and G2 were dropped from the feature set in addition to G3. The same preprocessing and split procedure was applied (random_state=50). This experiment was designed to simulate an early prediction scenario where prior period grades are not yet available.

Experiment 1 Results

7.1 Model Performance

7.2 RMSE Comparison

The bar chart below shows the RMSE values for all three models. Linear Regression and Random Forest have very similar RMSE values (approximately 1.31 and 1.35 respectively), while the Decision Tree has a noticeably higher RMSE of approximately 1.86. A lower RMSE is better, so Linear Regression was the most accurate model by this measure.

7.3 R² Comparison

The R² bar chart confirms the same ranking. Linear Regression achieved the highest R² of approximately 0.808, followed closely by Random Forest at 0.798. The Decision Tree achieved 0.615, which, while not bad in absolute terms, is notably lower than the other two.

7.4 Actual vs Predicted (Linear Regression)

The scatter plot of actual versus predicted grades for Linear Regression shows a generally strong alignment along the diagonal. Points that lie close to or on the diagonal line represent accurate predictions. The spread of points suggests that the model performs well across most of the grade range, with slightly more variation at the lower end (including the cluster of students who scored 0, which the model does not fully capture).

7.5 Discussion of Experiment 1 Results

One of the more interesting observations from Experiment 1 was that Linear Regression slightly outperformed Random Forest. Normally, ensemble methods like Random Forest are expected to perform better than simple linear models because they can capture non-linear relationships in the data.

The explanation here lies in the nature of the prediction task. Because G2 has a correlation of 0.92 with G3 and G1 has a correlation of 0.83, the relationship between the most informative features and the target is largely linear. Linear Regression is well-suited to exactly this kind of task. Random Forest, while slightly less efficient here, still performed comparably because it also captured the strong signal from G2 and G1.

The Decision Tree's weaker performance is consistent with its known tendency to overfit. Without any constraint on tree depth, the Decision Tree may have memorised aspects of the training data that do not generalise to the test set.

Feature Importance Analysis

8.1 Results

Feature importance scores were extracted from the trained Random Forest model. The top 10 features and their importance scores are shown below.

8.2 Interpretation

The most interesting result in the feature importance analysis is the overwhelming dominance of G2. It received an importance score of approximately 0.834, meaning roughly 83% of the predictive signal used by the Random Forest model came from the second period grade alone.

A surprising finding was that G1 received a relatively small importance score of approximately 0.013 despite being strongly correlated with G3. This is not a sign that G1 is unimportant in reality. Instead, it reflects a phenomenon known as feature redundancy. Because G2 is even more strongly correlated with G3 than G1 is, and because G1 and G2 are themselves correlated (0.86), the model effectively finds that once it has used G2, there is very little additional predictive information remaining in G1. The Random Forest therefore relies almost entirely on G2 and assigns most of its importance there.

Among the non-grade features, absences ranked second with an importance score of 0.043. All remaining features had scores below 0.01, indicating that while they contribute marginally, their individual effects are quite small compared to the grade variables.

Cross-Validation

9.1 Setup and Results

5-fold cross-validation was performed on the Random Forest model using the full feature set (all features including G1 and G2). In 5-fold cross-validation, the dataset is divided into 5 equal parts. The model is trained on 4 parts and evaluated on the remaining part, rotating through all 5 combinations. This gives a more reliable estimate of how the model will perform on unseen data compared to a single train-test split.

The R² scores obtained across the 5 folds were:

9.2 Interpretation

The average R² score across the 5 folds was approximately 0.779. This is very close to the single train-test split R² of 0.798 obtained for Random Forest in Experiment 1. It suggests the model is not heavily dependent on any particular way the data was divided. The performance is consistent across different subsets.

Fold 2 produced the lowest score (0.691) while Fold 4 produced the highest (0.846). This variation is expected and is partly due to the natural variation in which students end up in the test portion for each fold. The range of scores is not unusually large, which further supports the conclusion that the model generalises reasonably well.

Experiment 2 Results

10.1 Model Performance Without G1 and G2

After removing G1 and G2 from the feature set, all three models were retrained and re-evaluated. The results are shown in the table below.

10.2 Interpretation

The drop in performance after removing G1 and G2 was dramatic for all three models. Linear Regression fell from an R² of 0.808 to just 0.172. Random Forest dropped from 0.798 to 0.195. The Decision Tree reached a negative R² of −0.421.

A negative R² for the Decision Tree does not indicate an error in the code. R² can be negative when a model performs worse than the simplest possible prediction, which is always predicting the average G3 value. The Decision Tree, once deprived of G1 and G2, was making predictions so noisy that they were less useful than simply guessing the class average for every student. This suggests the tree found no coherent structure in the remaining features that it could reliably exploit.

The small positive R² values for Linear Regression and Random Forest (0.172 and 0.195) confirm that the remaining features do contain some information about G3. However, that information is weak. The models explain only about 17–20% of the variance in final grades using demographic, family, and lifestyle features alone.

Comparative Analysis

The two experiments together tell a clear story. When previous grades are available, regression models can predict final grades reasonably well. When they are not, performance falls off sharply.

The comparison below summarises the change in R² for each model between the two experiments.

The approximately 75–80% reduction in R² after removing just two features illustrates how central G1 and G2 were to the models' predictive ability. It tells us that for early prediction systems (where no prior period grades exist), building a reliable grade predictor based on demographic and lifestyle information alone is challenging.

Between the two remaining models in Experiment 2, Random Forest marginally outperformed Linear Regression (R² of 0.195 vs 0.172). This is a reversal from Experiment 1, where Linear Regression led. The reason is that without the dominant linear signal from G2, the remaining weaker features may have a slightly non-linear relationship with G3 that Random Forest captures marginally better.

Conclusion