Projects |

Sandbox

Fri, 07 Aug 2026 00:00:00 +0000

Not everything fits neatly into a single write-up. This page is a sandbox: a place to point to smaller, standalone projects that live outside this site, and that will keep growing as I add to them.

🎮

Work in progress

Edu

Text-based games exploring governance and decision-making. More scenarios and content are on the way.

A small, self-contained proof-of-concept — a sample of how I like to think through and work a problem.

Take a look →

Both are actively evolving — check back here for updates, or click through to see them as they stand today.

Multiple Linear Regression Analysis of Temperature Data in Albany and Sacramento

Fri, 29 Dec 2023 00:00:00 +0000

Table of Contents

Introduction

The study of weather patterns has been of great interest to scientists, researchers and the general public for many years. In recent times, there has been a growing concern about the impacts of global warming and climate change on weather patterns and the environment. One area of particular interest is the relationship between temperature and other meteorological variables, such as humidity, precipitation, and wind speed. Understanding this relationship is important for improving our understanding of climate patterns and predicting future weather patterns. This study aims to investigate the relationship between temperature and the other meteorological variables in the United States, and to explore how this relationship varies across a specific time range.

The dataset pertains to two prominent cities located on the opposite coasts of the United States: Albany, the capital city of New York State, and Sacramento, the capital city of California State. The data were respectively collected from Albany International Airport and Sacramento Metropolitan Airport.

The data was extracted from the (NOAA) website on 5 May 2023 and is a subset of the Local Climatological Data (LCD) dataset. The original data ranges back to the 1940s and 1970s for the two selected stations, but for the purpose of this report, we will be analyzing measurements from January 1st, 2000, to December 31st, 2022.

The data was stored in two seperate csv files. It is worth noting that the Daily Humidity values in Sacramento are only available from January 1st, 2005. The data was parsed, cleaned and slightly pre-processed in Excel. It is worth noting the units of measurement employed for the various variables under consideration. Temperature is expressed in Celsius, wind speed in meters per second (m/s), precipitation in millimeters (mm), and humidity in percentage (%).

The study was conducted utilizing Python 3 and Jupyter Notebook as the programming and development environment. To ensure clarity, comprehensibility, and reproducibility, the report incorporates the code implementation and its corresponding outputs. This approach facilitates a transparent presentation of the analysis, enabling us to follow the methodology and reproduce the results beyond the final report.

Exploratory data analysis: Understanding the data

# Importing libraries
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import seaborn as sns
from matplotlib.dates import MonthLocator, DateFormatter
import statsmodels.api as sm

# Reading the datasets
albany = pd.read_csv('albany.csv')
sacramento = pd.read_csv('sacramento.csv')

# Datetime formatting
sacramento['date'] = pd.to_datetime(sacramento['date'], format='%d/%m/%Y')
albany['date'] = pd.to_datetime(albany['date'], format='%d/%m/%Y')

The tail() method in the code below is used to retrieve the last rows of the Sacramento DataFrame. This feature facilitates rapid examination of the end of DataFrame, enabling data verification. We confirm the presence of 8333 rows and 7 columns, which include the date variable.

print (sacramento.tail(), "\n The size of the DataFrame is", sacramento.shape)

 date averagetemperature averagewindspeed maximumtemperature \
8329 2022-12-27 10.56 6.84 12.78
8330 2022-12-28 6.11 1.48 10.00
8331 2022-12-29 10.00 3.98 12.22
8332 2022-12-30 13.89 8.18 16.67
8333 2022-12-31 13.33 7.82 15.56

 minimumtemperature precipitation humidity
8329 8.33 10.41 85.33
8330 2.22 0.00 90.75
8331 7.22 16.26 85.04
8332 11.11 1.52 87.71
8333 10.56 47.75 82.71
 The size of the DataFrame is (8334, 7)

We proceed to do the same with the Albany DataFrame, and confirm the presence of 8375 rows and 7 columns.

print (albany.tail(), "\n The size of the DataFrame is", albany.shape)

 date averagetemperature averagewindspeed maximumtemperature \
8370 2022-12-27 -2.22 2.59 1.11
8371 2022-12-28 0.00 3.04 5.00
8372 2022-12-29 1.67 2.59 8.89
8373 2022-12-30 10.00 3.53 13.89
8374 2022-12-31 9.44 3.80 11.67

 minimumtemperature precipitation humidity
8370 -5.56 0.00 56.88
8371 -5.56 0.00 63.63
8372 -5.56 0.00 61.33
8373 6.11 0.00 52.79
8374 6.67 2.03 81.17
 The size of the DataFrame is (8375, 7)

In the following, the describe() method, particularly useful for exploratory data analysis, allows for a comprehensive understanding of the data’s summary statistics at a glance. These statistics provide insights into the central tendency, variability, and distribution of the data in each column of the DataFrame.

Sacramento

sacramento.describe()

	averagetemperature	averagewindspeed	maximumtemperature	minimumtemperature	precipitation	humidity
count	8334.000000	8334.000000	8334.000000	8334.000000	8334.000000	6572.000000
mean	16.758523	3.378572	24.091746	9.145821	1.173276	63.278203
std	6.681999	1.715114	8.657389	5.392560	4.735319	16.252343
min	0.560000	0.040000	3.890000	-11.110000	0.000000	16.000000
25%	11.110000	2.100000	16.670000	5.000000	0.000000	51.040000
50%	16.670000	3.170000	23.890000	9.440000	0.000000	62.130000
75%	22.780000	4.430000	31.670000	13.330000	0.000000	76.250000
max	35.560000	13.500000	53.890000	27.220000	104.650000	100.000000

In Sacramento, California, the average annual temperature is around 16.76 degrees Celsius, with a standard deviation of approximately 6.68. The temperature range is substantial, indicating potential fluctuations in weather conditions. Notably, the highest recorded temperature of 53.89 degrees Celsius occurred on March 19, 2018, while the lowest temperature of -11.11 degrees Celsius was recorded on February 11, 2004. Likewise, the average precipitation amount in Sacramento is approximately 1.17 mm. The precipitation values demonstrate a standard deviation of about 4.74 units. This aligns with the prevailing climate in California, characterized by hot, arid summers and short, cold, wet winters, resulting in partly cloudy conditions. The number of days with precipitations below the average overall precipitation amount of 1.17 mm is 7368 days, over the study period of 8334 days. The descriptive measures of central tendency and variability correspond to the anticipated weather patterns in the area.

max_temp_index = sacramento['maximumtemperature'].idxmax()
date_max_temp = sacramento.loc[max_temp_index, 'date']
min_temp_index = sacramento['minimumtemperature'].idxmin()
date_min_temp = sacramento.loc[min_temp_index, 'date']
print("Date with the highest maximum temperature:", date_max_temp)
print("Date with the lowest minimum temperature:", date_min_temp)

Date with the highest maximum temperature: 2018-03-19 00:00:00
Date with the lowest minimum temperature: 2004-02-11 00:00:00

count_below_threshold = sacramento[sacramento['precipitation'] < 1.17].shape[0]
print("Number of days with precipitations below the average precipitation amount of 1.17 mm is :", count_below_threshold)
count_exceeding_threshold = sacramento[sacramento['averagetemperature'] > 16.76].shape[0]
print("Number of days with daily average temperature exceeding the overall average temperature of 9.92 C is :", count_exceeding_threshold)

Number of days with precipitations below the average precipitation amount of 1.17 mm is : 7368
Number of days with daily average temperature exceeding the overall average temperature of 9.92 C is : 4028

columns_to_include = ['averagetemperature', 'averagewindspeed', 'precipitation', 'humidity']
subset_sacramento = sacramento[columns_to_include]
# pair plots between numerical columns
sns.pairplot(subset_sacramento)
plt.show()

The grid figure presented illustrates the pair plots, providing insights into the relationships, distributions, and interactions among the selected variables. These visualizations allow for a better understanding of the anticipated effects of each variable on the target temperature variable. To enhance the visual clarity of the plots, the minimum and maximum temperatures were excluded from the analysis.

An evident observation is the strong negative correlation between humidity and average temperature, with a limited number of noticeable outliers. In contrast, the relationship between precipitation and temperature does not exhibit a clear pattern, but rather displays more outlier values. Notably, low precipitation values are observed across various temperature ranges, whereas higher precipitation values tend to occur within a specific temperature range. Furthermore, the influence of wind speed on temperature is not evident for lower wind speed values. However, as the wind speed increases, the temperature tends to stabilize within a specific temperature range. This finding suggests that wind speed may serve as a more reliable predictor of temperature within this specific range.

Overall, the pair plots provide valuable insights into the relationships and patterns among the variables, shedding light on their potential impact on the studied temperature variable.

correlation_matrix = sacramento.corr()
sns.heatmap(correlation_matrix, annot=True, cmap="coolwarm")
plt.title("Sacramento Correlation Matrix Heatmap")
plt.show()

The heatmap figure provides a visual representation of the correlation coefficients between pairs of variables. Each cell in the heatmap represents the correlation coefficient, with color gradients indicating the magnitude and direction of the correlation. Warmer colors, such as shades of red, indicate a positive correlation, while cooler colors, such as shades of blue, indicate a negative correlation. The heatmap reinforces the observations made from the pair plots mentioned earlier. It visually confirms the relationships and patterns observed between variables. Furthermore, in the process of building our model, we have made the decision to only include the minimum temperature variable and exclude the maximum temperature. This decision is based on the redundancy of their influence on the average temperature and the potential issue of multicollinearity that could arise. By excluding the maximum temperature, we aim to prevent any redundant or collinear effects in our analysis.

Albany

albany.describe()

	averagetemperature	averagewindspeed	maximumtemperature	minimumtemperature	precipitation	humidity
count	8330.000000	8330.000000	8330.000000	8330.000000	8331.000000	8375.000000
mean	9.916845	3.286505	15.062441	4.494155	2.862368	68.329931
std	10.551006	1.709147	11.239033	10.239078	7.307641	12.755930
min	-21.670000	0.000000	-17.780000	-26.670000	0.000000	24.710000
25%	1.670000	2.010000	5.560000	-2.780000	0.000000	59.960000
50%	10.560000	3.080000	16.110000	4.440000	0.000000	68.250000
75%	19.440000	4.290000	25.000000	13.330000	1.780000	77.500000
max	31.110000	10.590000	37.220000	25.000000	119.130000	100.000000

In Albany, New York, the average overall temperature is around 9.92 degrees Celsius, with a standard deviation of approximately 10.55. The temperature range is as substantial as in Sacramento (around 64 degrees Celsius between the lowest and highest values in both), indicating potential fluctuations in weather conditions. Notably, the highest recorded temperature of 37.22 degrees Celsius occurred on July 21, 2011, while the lowest temperature of -26.67 degrees Celsius was recorded on January 24, 2005. Likewise, the average precipitation amount in Albany is approximately 2.86 mm. The precipitation values demonstrate a standard deviation of about 7.3 units. This aligns with the prevailing climate in New York, characterized by humid continental with warm to hot summers and freezing cold snowy winters. The number of days with precipitations below the average overall precipitation amount of 2.86 mm is 6603 days, over the study data of 8330 days. While the number of days with daily average temperatures exceeding the overall average temperature of 9.92 C was 4318 days. The descriptive measures of central tendency and variability correspond to the anticipated weather patterns in the area.

These findings from both areas suggest that in comparison, most precipitations in Sacramento occured in fewer days that had an excessive amount of pecipitations. In contrast, the precipitations in Albany were better spread over more days.

max_temp_index2 = albany['maximumtemperature'].idxmax()
date_max_temp2 = albany.loc[max_temp_index2, 'date']
min_temp_index2 = albany['minimumtemperature'].idxmin()
date_min_temp2 = albany.loc[min_temp_index2, 'date']
print("Date with the highest maximum temperature:", date_max_temp2)
print("Date with the lowest minimum temperature:", date_min_temp2)

Date with the highest maximum temperature: 2011-07-21 00:00:00
Date with the lowest minimum temperature: 2005-01-24 00:00:00

count_below_threshold = albany[albany['precipitation'] < 2.86].shape[0]
print("Number of days with precipitations below the average precipitation amount of 2.86 mm is :", count_below_threshold)
count_exceeding_threshold = albany[albany['averagetemperature'] > 9.92].shape[0]
print("Number of days with daily average temperature exceeding the overall average temperature of 9.92 C is :", count_exceeding_threshold)

Number of days with precipitations below the average precipitation amount of 2.86 mm is : 6603
Number of days with daily average temperature exceeding the overall average temperature of 9.92 C is : 4318

subset_albany = albany[columns_to_include]
# pair plots between numerical columns
sns.pairplot(subset_albany)
plt.show()

correlation_matrix2 = albany.corr()
sns.heatmap(correlation_matrix2, annot=True, cmap="coolwarm")
plt.title("Albany Correlation Matrix Heatmap")
plt.show()

The pair plots and heatmap figures for Albany, New York, reveal distinct patterns and relationships among the variables. A notable observation is the prevalence of low correlation coefficients and cooler colors in the heatmap, indicating weaker associations between the variables. In particular, the maximum and minimum temperatures exhibit a strong positive correlation with the average temperature, suggesting a consistent relationship between these variables. However, for humidity and windspeed, there are no discernible patterns or clear correlations with the average temperature. Additionally, there are several outliers observed in the precipitation variable, indicating instances of extreme or unusual precipitation values.

The absence of clear patterns and the presence of non-linear relationships between humidity, precipitation, windspeed, and average temperature in Albany, New York, can be attributed to the specific nature of the local climate category. This climate category is described as humid continental, characterized by warm to hot summers and freezing cold snowy winters. Within this climate category, the intricate interplay among various atmospheric conditions, such as prevailing winds and moisture sources, gives rise to diverse and dynamic weather patterns. The variability in temperature, humidity, precipitation, and windspeed is influenced by numerous factors and regional weather phenomena, including blizzards. Furthermore, the seasonal temperature extremes and the potential impact of the proximity to large bodies of water, such as the Atlantic Ocean, introduce additional variability and non-linear relationships among the variables under study.

Considering the distinct characteristics of the locality, it is reasonable to expect that the relationships may not conform to a simple linear pattern. The complex interplay of factors such as temperature inversions, air masses, and local topography contributes to the observed complexity in these relationships.

Multiple linear regression

Sacramento

sacramento.isnull().sum()

date 0
averagetemperature 0
averagewindspeed 0
maximumtemperature 0
minimumtemperature 0
precipitation 0
humidity 1762
dtype: int64

Prior to constructing the linear regression model, we examine the dataset for missing values. As a result, we identify 1762 instances of missing data specifically pertaining to the humidity variable. Instead of discarding these unavailable data points, which represent a substantial portion of the entire dataset, we opted to impute the missing values using the mean.

mean_humidity = sacramento['humidity'].mean()
sacramento['humidity'].fillna(mean_humidity, inplace=True)

# Adding a constant column to the independent variables:
# in the context of using statsmodels for multiple linear regression, it represents the intercept term in the linear regression equation
X = sacramento[['averagewindspeed', 'minimumtemperature', 'precipitation', 'humidity']]
X = sm.add_constant(X)

# Defining the dependent variable
y = sacramento['averagetemperature']

# Fitting the OLS model
model = sm.OLS(y, X)
results = model.fit()

# Print the regression summary
print(results.summary())

 OLS Regression Results
==============================================================================
Dep. Variable: averagetemperature R-squared: 0.926
Model: OLS Adj. R-squared: 0.926
Method: Least Squares F-statistic: 2.601e+04
Date: Sun, 18 Jun 2023 Prob (F-statistic): 0.00
Time: 23:27:19 Log-Likelihood: -16811.
No. Observations: 8334 AIC: 3.363e+04
Df Residuals: 8329 BIC: 3.367e+04
Df Model: 4
Covariance Type: nonrobust
======================================================================================
 coef std err t P>|t| [0.025 0.975]
--------------------------------------------------------------------------------------
const 16.2143 0.133 121.559 0.000 15.953 16.476
averagewindspeed -0.5312 0.013 -40.981 0.000 -0.557 -0.506
minimumtemperature 1.0532 0.004 245.554 0.000 1.045 1.062
precipitation -0.0711 0.005 -15.089 0.000 -0.080 -0.062
humidity -0.1139 0.002 -69.456 0.000 -0.117 -0.111
==============================================================================
Omnibus: 663.002 Durbin-Watson: 0.920
Prob(Omnibus): 0.000 Jarque-Bera (JB): 3431.099
Skew: -0.189 Prob(JB): 0.00
Kurtosis: 6.121 Cond. No. 439.
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

The OLS Regression Results provide insights into the linear regression model that was applied to the dataset. The R-squared value of 0.926 indicates that approximately 92.6% of the variability in the average temperature (dependent variable) can be accounted for by the independent variables included in the model. This suggests a robust relationship between the independent variables (averagewindspeed, minimumtemperature, precipitation, humidity) and the average temperature. The F-statistic is 2.601e+04, with a remarkably low probability (p < 0.001). This implies that the overall regression model is statistically significant, indicating that at least one of the independent variables exhibits a significant association with the average temperature. The coefficients represent the estimated impact of each independent variable on the average temperature while holding other variables constant. Here are the interpretations of the coefficients:

average windspeed: With each unit increase in average windspeed, the average temperature is estimated to decrease by approximately 0.5312 units.
minimum temperature: With each unit increase in minimum temperature, the average temperature is estimated to increase by approximately 1.0532 units.
precipitation: With each unit increase in precipitation, the average temperature is estimated to decrease by approximately 0.0711 units.
humidity: With each unit increase in humidity, the average temperature is estimated to decrease by approximately 0.1139 units.

The p-values associated with each coefficient are very low (p < 0.001), indicating that all independent variables have a statistically significant relationship with the average temperature. The 95% confidence intervals provide a range within which the true value of each coefficient is likely to fall. For example, the confidence interval for the average windspeed coefficient is (-0.557, -0.506), suggesting that the true effect of average windspeed on average temperature lies within this range with 95% confidence.

Overall, the findings support the idea that the independent variables (averagewindspeed, minimumtemperature, precipitation, humidity) are significant predictors of the average temperature in the Sacramento dataset. The regression model has a high R-squared value, indicating a good fit, and the coefficients and p-values suggest that all independent variables have a meaningful impact on the average temperature.

The regression equation is:

Sacramento temperature = 16.2143 − 0.5312 × averagewindspeed + 1.0532 × minimumtemperature − 0.0711 × precipitation − 0.1139 × humidity

# Obtain the predicted values
y_pred = results.predict(X)

# Calculate the residuals
residuals = y - y_pred

# Plot the Residuals vs. Fitted values
sns.residplot(x=y_pred, y=residuals, lowess=True, line_kws={'color': 'red'})
plt.xlabel('Fitted Values')
plt.ylabel('Residuals')
plt.title('Residuals vs. Fitted Values')
plt.show()

The (Residuals vs. Fitted) plot shows that the residuals are fairly randomly scattered around the 0 residual line. Also, the residuals form a seemingly horizontal band around the residual = 0 line, which suggests that the variances of the error terms can be considered equal.

The graph shows a number of outliers that deviate significantly from the general pattern of the residuals and could be investigated further. The number of these individual points shouldn’t greatly impact the regression coefficients and overall model fit.

# Generate the normal QQ plot
sm.qqplot(residuals, line='s')
plt.title('Normal QQ Plot of Residuals')
plt.show()

The (Normal Q-Q) plot shows if residuals are normally distributed (with a small tail). The relationship between the theoretical quantiles and the standardized residuals is approximately linear for most points. We can say that the error terms are indeed normally distributed. The presence of some outliers is confirmed and pronounced here as well.

Albany

albany.isnull().sum()

date 0
averagetemperature 45
averagewindspeed 45
maximumtemperature 45
minimumtemperature 45
precipitation 44
humidity 0
dtype: int64

Similar to Sacramento, we examine the dataset for missing values. We identify at most 45 instances of missing data pertaining to all variables except for humidity. We can proceed to discarding these unavailable data points, which do not represent a substantial portion of the entire dataset.

albany.dropna(inplace=True)

X2 = albany[['averagewindspeed', 'minimumtemperature', 'precipitation', 'humidity']]
X2 = sm.add_constant(X2)

# Defining the dependent variable
y2 = albany['averagetemperature']

# Fitting the OLS model
model2 = sm.OLS(y2, X2)
results2 = model2.fit()

# Print the regression summary
print(results2.summary())

 OLS Regression Results
==============================================================================
Dep. Variable: averagetemperature R-squared: 0.972
Model: OLS Adj. R-squared: 0.972
Method: Least Squares F-statistic: 7.188e+04
Date: Sun, 18 Jun 2023 Prob (F-statistic): 0.00
Time: 23:27:28 Log-Likelihood: -16575.
No. Observations: 8330 AIC: 3.316e+04
Df Residuals: 8325 BIC: 3.320e+04
Df Model: 4
Covariance Type: nonrobust
======================================================================================
 coef std err t P>|t| [0.025 0.975]
--------------------------------------------------------------------------------------
const 12.0641 0.134 89.777 0.000 11.801 12.328
averagewindspeed -0.3560 0.012 -29.811 0.000 -0.379 -0.333
minimumtemperature 1.0241 0.002 519.140 0.000 1.020 1.028
precipitation -0.0116 0.003 -3.895 0.000 -0.018 -0.006
humidity -0.0812 0.002 -45.519 0.000 -0.085 -0.078
==============================================================================
Omnibus: 805.064 Durbin-Watson: 1.529
Prob(Omnibus): 0.000 Jarque-Bera (JB): 1700.446
Skew: 0.618 Prob(JB): 0.00
Kurtosis: 4.836 Cond. No. 484.
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

The R-squared value of 0.972 indicates that approximately 97.2% of the variability in the average temperature (dependent variable) can be accounted for by the independent variables included in the model. This suggests a strong relationship between the independent variables (averagewindspeed, minimumtemperature, precipitation, humidity) and the average temperature. The F-statistic is 7.188e+04, with a remarkably low probability (p < 0.001). This implies that the overall regression model is highly statistically significant, indicating that at least one of the independent variables exhibits a significant association with the average temperature.

The coefficients represent the estimated impact of each independent variable on the average temperature while holding other variables constant. Here are the interpretations of the coefficients:

average windspeed: With each unit increase in average windspeed, the average temperature is estimated to decrease by approximately 0.3560 units.
minimum temperature: With each unit increase in minimum temperature, the average temperature is estimated to increase by approximately 1.0241 units.
precipitation: With each unit increase in precipitation, the average temperature is estimated to decrease by approximately 0.0116 units.
humidity: With each unit increase in humidity, the average temperature is estimated to decrease by approximately 0.0812 units.

The p-values associated with each coefficient are very low (p < 0.001), indicating that all independent variables have a statistically significant relationship with the average temperature.

The 95% confidence intervals provide a range within which the true value of each coefficient is likely to fall. For example, the confidence interval for the average windspeed coefficient is (-0.379, -0.333), suggesting that the true effect of average windspeed on average temperature lies within this range with 95% confidence.

The OLS Regression Results show that the regression model has a high R-squared value of 0.972, indicating that approximately 97.2% of the variability in the average temperature (dependent variable) can be explained by the independent variables included in the model. The F-statistic of 7.188e+04 is highly significant (p < 0.001), indicating that the overall regression model is statistically significant.

The regression equation can be expressed as:

Albany temperature = 12.0641 − 0.3560 × averagewindspeed + 1.0241 × minimumtemperature − 0.0116 × precipitation − 0.0812 × humidity

# Obtain the predicted values
y_pred2 = results2.predict(X2)

# Calculate the residuals
residuals2 = y2 - y_pred2

# Plot the Residuals vs. Fitted values
sns.residplot(x=y_pred2, y=residuals2, lowess=True, line_kws={'color': 'red'})
plt.xlabel('Fitted Values')
plt.ylabel('Residuals')
plt.title('Residuals vs. Fitted Values')
plt.show()

Also in this case, the (Residuals vs. Fitted) plot shows that the residuals are randomly scattered around the 0 residual line. The residuals form a horizontal band around the residual = 0 line, which suggests that the variances of the error terms are considered equal.

The graph shows a number of outliers that deviate significantly from the general pattern of the residuals and could be investigated further. The number of these individual points wouldn’t greatly impact the regression coefficients and overall model fit.

# Generate the normal QQ plot
sm.qqplot(residuals2, line='s')
plt.title('Normal QQ Plot of Residuals')
plt.show()

The (Normal Q-Q) plot shows that the relationship between the theoretical quantiles and the standardized residuals is approximately linear for most points. However, it is in fact skewed right, meaning that most of the data is distributed on the left side with a long “tail” of data extending out to the right. The points depart upward from the straight red line as you follow the quantiles from left to right. The red line shows where the points would fall if the dataset were normally distributed. The point’s trend upward shows that the actual quantiles are much greater than the theoretical quantiles, meaning that there is a greater concentration of data beyond the right side of a Gaussian distribution.

Overall, the findings derived from the linear regression analysis provide evidence supporting the existence of a relationship between temperature and meteorological variables in the United States, with a specific focus on Albany and Sacramento. Both cities exhibit statistically significant associations between temperature and the meteorological variables of average windspeed, minimum temperature, precipitation, and humidity. By understanding and analyzing these relationships, we can enhance our understanding of weather patterns, contribute to climate studies, and improve weather predictions. These findings highlight the significance of considering multiple meteorological factors when examining temperature variations and provide valuable insights into the impacts of these variables on weather patterns in the United States.

Plotting climate data using pandas

Sat, 20 Mar 2021 00:00:00 +0000

Table of Contents

Introduction

The data for this comes from a subset of The National Centers for Environmental Information (NCEI) Daily Global Historical Climatology Network (GHCN-Daily). The GHCN-Daily is comprised of daily climate records from thousands of land surface stations across the globe.

The data (stored in a ) is comprised of daily climate records over the period 2005-2015, from land surface stations near Ann Arbor, Michigan, United States. Each row in the datafile corresponds to a single observation.

The provided variables are :

id : station identification code
date : date in YYYY-MM-DD format
element : indicator of element type
- TMAX : Maximum temperature (tenths of degrees C)
- TMIN : Minimum temperature (tenths of degrees C)
value : data value for element (in tenths of degrees C)

For the purpose of this notebook, we are going to plot a line chart of the record high and record low temperatures by day of the year over the period 2005-2014. Then, we overlay a scatter of the 2015 data for any points (highs and lows) for which the ten-year record (2005-2014) was broken in 2015.

Importing libraries and reading data

import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import seaborn as sns
from matplotlib.dates import MonthLocator, DateFormatter

df = pd.read_csv('data.csv')

Data processing and time series manipulation

Since the temparture values are in the tenths of degree Celsius, we need to convert them to °C.

df['Data_Value'] = df['Data_Value']/10 #convert temperatures to °C

Next, we would ensure that the Date values are interpreted as date type, and sort the entire DataFrame by date.

df['Date'] = pd.to_datetime(df['Date'], infer_datetime_format=True) #convert to date type
df = df.sort_values('Date') #sort DataFrame by date

We use the head() function to get the first n rows (5 by default), and the shape attribute for a tuple representing the dimensionality of the resulting DataFrame:

print (df.head(), "\n The size of the DataFrame is", df.shape)

 ID Date Element Data_Value
60995 USW00004848 2005-01-01 TMIN 0.0
17153 USC00207320 2005-01-01 TMAX 15.0
17155 USC00207320 2005-01-01 TMIN -1.1
10079 USW00014833 2005-01-01 TMIN -4.4
10073 USW00014833 2005-01-01 TMAX 3.3
 The size of the DataFrame is (165085, 4)

For clarity and ease, we are going to create two distinct DataFrames to hold TMAX and TMIN values.

dftmax = df[df['Element']=='TMAX'] #get dataframe of only TMAX values
dftmin = df[df['Element']=='TMIN'] #get dataframe of only TMIN values

To have a better overview of the entire data, I decided to keep all values including those of leap years. In this cell, we use numpy.arange() to get all possible dates in such a year (2008).

observation_dates = np.arange('2008-01-01', '2009-01-01', dtype='datetime64[D]')

Now, we would get the maximum temperatures for each day in the period from 2005 to 2014. We should remember that there are several registered values for any given day for that period. The resulting DataFrame dftmax14 has a length of 3652 (maximum values for each day in a 10-year span).

Then, we’ll create a pandas series comprised of all TMAX values in a one-year span (tmax1y). For any single day value (1 June for example), we would get the maximum TMAX for that day for the period 2005-2014.

#extract of TMAX of each day from 2005 to 2014
dftmax14 = dftmax[dftmax['Date']<'2015-01-01'].groupby('Date')['Data_Value'].max()

#set index to month-day format instead of year-m-d
dftmax14.index = dftmax14.index.strftime('%m-%d')

#TMAX in a one year span resume for data from 2005 to 2014
tmax1y = dftmax14.groupby(dftmax14.index).max()

We repeat the same steps to get minimum temperatures for each day in the period from 2005 to 2014.

#extract of TMIN of each day from 2005 to 2014
dftmin14 = dftmin[dftmin['Date']<'2015-01-01'].groupby('Date')['Data_Value'].min()

#set index to month-day format instead of year-m-d
dftmin14.index = dftmin14.index.strftime('%m-%d')

#TMIN in a one year span resume for data from 2005 to 2014
tmin1y = dftmin14.groupby(dftmin14.index).min()

Similarly, we create a DataFrame for TMAX values in 2015 (dftmax15). Then, we extract the dates for which TMAX values in 2015 were higher than the values of TMAX over the period 2005-2014 (observation_dates_tmax15). Finally, we update dftmax15 to have a DataFrame of those record breaking TMAX values.

#extract of TMAX of each day in 2015
dftmax15 = dftmax[dftmax['Date'] >= '2015-01-01'].groupby('Date')['Data_Value'].max()
#set index to month-day format instead of year-m-d
dftmax15.index = dftmax15.index.strftime('%m-%d')

dftmax15.loc['02-29'] = tmax1y['02-29']
dftmax15.sort_index(inplace=True)

observation_dates_tmax15 = observation_dates[dftmax15.values > tmax1y.values]

dftmax15 = dftmax15[dftmax15.values > tmax1y.values]

We can check the size of dftmax15, and infer that there are 37 days in 2015 that broke all TMAX values registered from 2005 to 2014.

dftmax15.size

The same previous steps to get a DataFrame dftmin15 of those record breaking TMIN values.

#extract of TMIN of each day in 2015
dftmin15 = dftmin[dftmin['Date'] >= '2015-01-01'].groupby('Date')['Data_Value'].min()
#set index to month-day format instead of year-m-d
dftmin15.index = dftmin15.index.strftime('%m-%d')

dftmin15.loc['02-29'] = tmin1y['02-29']
dftmin15.sort_index(inplace=True)

observation_dates_tmin15 = observation_dates[dftmin15.values < tmin1y.values]

dftmin15 = dftmin15[dftmin15.values < tmin1y.values]

By checking the size of dftmin15, we infer that there are 32 days in 2015 that broke all TMIN values registered from 2005 to 2014.

dftmin15.size

Plotting and styling using matplotlib and seaborn

In this cell, we use matplotlib and seaborn to create a figure, and plot line charts of the record high and record low temperatures by day of the year over the period 2005-2014, and a scatter of the 2015 data for any points (highs and lows) for which the ten-year record (2005-2014) was broken in 2015. I made sure the visual was nice, with appropriate legends and labels, and reduced chart junk.

months = MonthLocator(range(1, 13), bymonthday=1, interval=1)
monthsFmt = DateFormatter("%b")

%matplotlib inline
sns.set_style("white") # Set the aesthetic style of the plot

fig = plt.figure(figsize=(10,5)) # Create a new figure with determined figsize parameter
ax = plt.gca() # Get the current Axes instance on the current figure

# Plots and Scatter plots of y vs. x with determined parameters
ax.scatter(observation_dates_tmax15 , dftmax15.values, s=35,
 c=sns.dark_palette("purple", n_colors=37), alpha=0.5, label='2005-2014 record high broken' )
ax.scatter(observation_dates_tmin15 , dftmin15.values, s=25,
 c=sns.dark_palette((260, 75, 60), input="husl", n_colors=32), alpha=0.9, label='2005-2014 record low broken' )

ax.plot(observation_dates, tmax1y.values, '-', color= sns.color_palette("Reds")[-2], label='High', alpha=0.7, linewidth=1)
ax.plot(observation_dates, tmin1y.values, '-', color= sns.color_palette("Blues")[-2], label='Low', alpha=0.7, linewidth=1)

ax.set_xlabel('Months')
ax.set_ylabel('Degrees (C)')

ax.set_title('Highest and lowest temperatures by day of the year over the period 2005-2014 \n and records broken in 2015 for Ann Arbor, Michigan, United States')

ax.xaxis.set_major_locator(months)
ax.xaxis.set_major_formatter(monthsFmt)

plt.xticks(alpha=0.5)
plt.yticks(alpha=0.5)
sns.despine()
ax.legend()

maxv=tmax1y.values
minv=tmin1y.values
#To shade the area between the record high and record low temperatures for each day
ax.fill_between(observation_dates,maxv, minv, facecolor=sns.light_palette("lightgrey")[4], alpha=0.2)

#fig.savefig("temp.png", dpi=1200) #comment out to save the figure in png format (reduce dpi to get smaller files)

This is not the only way to represent the data as you may opt to tweak the different parameters, and experience more with the plotting libraries. You can also treat other questions, and have fun with the data and figures.

Furthermore, there are several ways to handle the data and the various DataFrames. I tried to be more explicit and went for simplicity, all whilst incorporating the .

If you want to learn more about data science through the python programming language, I highly recommend on Coursera.

Analysing Olympic Games medal table

Thu, 18 Mar 2021 00:00:00 +0000

Table of Contents

Introduction

The following uses the , which was derived from the Wikipedia entry on , as of the 2016 Summer Olympics and 2018 Winter Olympics. All changes in medal standings due to doping cases and medal redistributions up to and including 25 November 2020 are taken into account. Data queried on March 18th, 2021.

Using the power of pandas, read the csv file containing the dataset (the result is a DataFrame). Then perform data cleaning and preprocessing operations to get a more readable and practical format of the DataFrame, to be used later.

import pandas as pd
df = pd.read_csv("olympic_games_medal_table.csv",index_col=0,skiprows=1, encoding='latin_1')

for col in df.columns: # clean the lables of the raw data
 if col[:2]=="01":
 df.rename(columns={col:"Gold"+col[4:]}, inplace=True)
 if col[:2]=="02":
 df.rename(columns={col:"Silver"+col[4:]}, inplace=True)
 if col[:2]=="03":
 df.rename(columns={col:"Bronze"+col[4:]}, inplace=True)
 if col[:1]=="№":
 df.rename(columns={col:"#"+col[1:]}, inplace=True)

names_ids = df.index.str.split('\s\(') # split the index by '('

df.index = names_ids.str[0] # the [0] element is the country name (will be the new index)

df['ID'] = names_ids.str[1].str[:3] # the [1] element is the abbreviation or ID (take first 3 characters)

spare_df = df.copy()
df = df.drop('Totals') # remove the row with label 'Totals'

Get the top ten countries that have the most medals in the summer and winter games.

most_medals = df['Combined total'].nlargest(10).index

Stacked bar chart

In this cell, we use to visualize stacked bar charts representing the top ten countries in terms of total number of medals in the winter and summer games.

import matplotlib.pyplot as plt

%matplotlib inline

# get necessary pandas series
summer_medals = df.loc[most_medals, ['Total']]['Total']
winter_medals = df.loc[most_medals, ['Total.1']]['Total.1']

fig = plt.figure(figsize=(20, 10)) # create a figure object

# context manager for temporary styling
with plt.style.context(('seaborn-poster', {'xtick.labelsize' : 14, 'axes.labelpad':20 , 'axes.titlepad' : 20,
 'axes.spines.top' : False, 'axes.spines.right' : False, 'axes.spines.left' : False} )):
 # create the bar plots
 bar_list = [plt.bar(range(len(most_medals)), summer_medals, width = 0.5 ,color='#d69728',
 tick_label = most_medals, label = 'Summer'),
 plt.bar(range(len(most_medals)), winter_medals, width = 0.5 , bottom = summer_medals ,
 tick_label = most_medals, label = 'winter')]
 ax = plt.gca()
 ax.set_ylabel('Number of Medals')
 ax.set_xlabel('Countries')
 ax.set_title('Number of total medals for the top ten countries')
 ax.tick_params(bottom=False, left=False, labelleft=False)

 # attach text label for each bar displaying its value
 for i in bar_list:
 for bar in i:
 height = bar.get_height()
 bottom = bar.get_y()
 if height < 50:
 y = 0.5*height+bottom*1.20
 else:
 y = 0.5*height+bottom
 plt.gca().text(bar.get_x() + bar.get_width()/2, y, str(int(height)),
 ha='center', color='black', fontsize=20)
 plt.legend()

# save the plot as a png file. you can change the file format to pdf or any supported extension (comment out to use)
#fig.savefig("totalmedals.png", dpi=150)

Bubble chart

This chart is an example of a visualization that can be created to help understand the data. This is a bubble chart showing the value of adjusted gold medals (#total gold/ #total games) Vs. the rank with reference to the number of total medals won.

The size of the bubble corresponds to an adjusted value of total medals (#total medals/ #total games) won, and the color corresponds to the geolocation (European or non-European) or current status (red: no longer exists).

top_medals = spare_df['Combined total'].nlargest(11).index
bubble_df = spare_df.loc[top_medals].drop('ID', axis=1) # dataframe for top 11 winners

# to eliminate overlapping medals' count
sum_topcoun = bubble_df[1:][bubble_df.columns[~bubble_df.columns.str.contains('#')]].sum()
for col in bubble_df.columns:
 if col in sum_topcoun.index:
 bubble_df.loc['Totals'][col] -= sum_topcoun[col]

bubble_df = bubble_df.rename(index={'Totals':'Rest of the World'})

# create 2 new columns with their respective data
bubble_df['adjusted_cgold'] = (bubble_df['Gold.2'].div(bubble_df['# Combined Games'])).apply(lambda x: float('%.1f'%x))
bubble_df['Rank'] = range(1,len(bubble_df.index)+1)

# chart creation and styling
with plt.style.context(('seaborn-poster', {'xtick.labelsize' :12, 'ytick.labelsize':12,'axes.labelpad':20 ,
 'axes.titlepad' : 20,'axes.labelsize':15} )):
 ax2 = bubble_df.plot(x='Rank', y='adjusted_cgold', kind='scatter',
 c=['#e4aa1a','#377eb8','#e41a1c','#4daf4a','#4daf4a','#4daf4a','#4daf4a','#4daf4a',
 '#377eb8','#4daf4a','#4daf4a'], linewidths=2 ,
 xticks=range(1,len(bubble_df.index)+1),
 s=(bubble_df['Combined total'].div(bubble_df['# Combined Games']))*100, alpha=.55, figsize=[15,7])
 ax2.set_ylim(0,65)
 ax2.set_title('Adjusted total gold medals by total medals won')
 ax2.set_ylabel('Adjusted gold medals')
 for i, txt in enumerate(bubble_df.index): # add labels inside each bubble
 ax2.annotate(txt, [bubble_df['Rank'][i], bubble_df['adjusted_cgold'][i]], ha='center',fontsize=11)

#plt.savefig("cgold.png", dpi=150)

This chart shows that the United States has the most number of total medals in summer and winter games as indicated by the x-axis (Rank). Norway has the least number of combined medals among the top ten.

Based on the values of adjusted gold medals represented by the y-axis, the Soviet Union won the most gold medals relative to the number of olympic games in which they participated, followed by the US and Russia respectively.

The sizes of bubbles suggest that the top 3 countries that won most medals relative to the number of games they were part of, are respectively, the Soviet Union, the US and Russia.

To put it in perspective by comparing France and China, the latter won less total medals overall (position on x-axis). But taking into account the number of olympic games played, China did win more gold medals (position on y-axis) and total medals (bubble-size).

Governance of extractive industry in Tunisia

Wed, 03 Mar 2021 00:00:00 +0000

Governance of the extractive industry is important to optimize resource utilization, and to ensure that the outcomes from natural resources exploitation contribute to the sustainable development of the country.

Simultaneously obtaining higher economic revenues and better social impacts is not a simple task, and may be impeded by several practical and organizational obstacles, that focus on the present gains rather than on sustainable development.

Governance can be improved with the right legal, institutional and administrative measures, and by applying certain best practices. They must unfold in terms of multiple reforms carried out judiciously.

The key question addressed by this report is: how can we improve the governance of the extractive industry in Tunisia?

This would be done as follows:

Chapter 1 would address the legal framework governing the extractive industry in Tunisia. An analysis of the different legal texts, and the specific details would help contextualize and put into perspective the status quo.
Chapter 2 addresses the institutional and organizational frameworks of the sector. It would present the main entities that shape the public interventions and strategies. It would also try to analyse and build on the governance of the sector from this perspective.
Chapter 3 gives a broad but important idea about the current health of the sector in Tunisia, and showcases the importance of natural resources and the need for a better governance, with a focus on sustainable development.
Chapter 4 deals with the challenges and opportunities in line with the sustainable development of the sector and the country. It uses different approaches and tools to enhance the structures and regulations, and the governance of the whole sector along the decision chain.

This publication was written and submitted as part of my graduation work and requirements, from the National School of Administration of Tunis in 2020.