Blog |

Problem Solving Using Computational Thinking - Course Review

Fri, 07 May 2021 00:00:00 +0000

Table of Contents

For their 9th birthday, Coursera celebrated by offering their learners a collection of 9 coursers to pick from, and enrol to earn a free certificate (special offer was available through 30 April 2021). I chose Problem Solving Using Computational Thinking from the University of Michigan.

In this article, I will share some insights about the course, and what you can expect if you decide to take it on.

Course overview and structure

First, the course is taught entirely in English, but there are subtitles for other languages as well (currently French, Portuguese (European), Russian and Spanish).

In week 1, you will learn about the foundations of Computational Thinking from Associate Professor Chris Quintana, from the University of Michigan School of Education. Then, you will have the opportunity to see Computational Thinking through real world and hypothetical examples shared by three experts in weeks 2, 3 and 4.

These experts are, respectively, Associate Director Mariana Carrasco-Teja from the Michigan Institute for Computational Discovery and Engineering (airport surveillance and image analysis case study); Associate Professor Rafael Meza (epidemiology case study); and Instructional and Program Design Coordinator Darin Stockdil from the Center for Education Design, Evaluation, and Research (human trafficking case study).

The learning objectives are:

To define Computational Thinking components including abstraction, problem identification, decomposition, pattern recognition, algorithms, and evaluating solutions.
To recognize Computational Thinking concepts in practice through a series of real-world case examples.
And to develop solutions through the application of Computational Thinking concepts to real world problems (peer-graded assignment).

The course is structured in 5 weeks, with the last being a peer-graded final project. To review the learning material, do the practice quizzes and quizzes, it should take you around 2 hours per week for the first 3 weeks, and 1h15 for the 4th week (excluding the time required for discussion prompts).

That would surely depend on your own pace and learning style, and you should always devote enough time, and work through the material appropriately. As for the last week, I find it the most challenging, and it would take you longer than indicated. Allow and plan for at least 3 hours of work in that week.

Course review

My overall remarks and opinions regarding the course are:

Videos are not too long or too short. They are just about the right length for you to follow and focus in every segment, take a break and get back to another video.
Some parts would require very careful attention, and perhaps repeated reviewing of the material. That is because of their complexity for a non-specialized audience.
The course introduces new concepts, ideas and technologies from a variety of fields and domains. It brings richness of content, and should broaden one’s knowledge beyond the computational thinking aspects. You learn different things in just one course.
The course is well suited and appropriate for various skill levels. Even advanced learners can consolidate their knowledge and learn something new.
There are enough practice quizzes and quizzes for a learner to test their understanding. However, some questions require the student to fill-in their answers, and that might not always be the best method.
Not a lot of in-video questions.
The videos look scripted, with prepared speeches in advance. However, some videos are not as fluid or comprehensible. If necessary, you can use the subtitles.
Estimated times for completion of quizzes are a bit off.
There are no reading materials and other resources.

Note that the case for the 4th week is optional. You will need to consent to be able to read and examine the course material. That is because the case study covers a rather delicate topic regarding hypothetical implications of Computational Thinking on the issue of Human Trafficking.

I have taken many courses from the University of Michigan. This course is true to their approach and methodology. It is well structured, with professional high quality videos and production.

The case studies are meticulously presented, and I think they bring the most value to the course. And when working on the last peer-graded assignment, you get the chance to apply and test your knowledge to the fullest. Perhaps the thing that can be improved, is the quality of discussion forums.

Overall, I do recommend taking , and investing the time to complete the course.

Happy learning everyone!

K-Means clustering and similarity visualization of constitutions

Fri, 30 Apr 2021 00:00:00 +0000

Table of Contents

Introduction

Constitutions hold the fundamental principles and rules that constitute the legal basis of a country. They determine the system of goverment, and the relationships between branches and institutions.

When written, these documents can be quite unique and distinct in many aspects, such as length and legal terminology. However, they can also share some other features, since they tend to have similar purposes.

A textual analysis of such data can be useful. We are going to apply some techniques to compare and cluster various constitutions. This work tries to see if constitutional text corpuses are indicative of the set and outlined systems of government.

To do so, we are going to use TF-IDF term weighting and K-Means clustering from scikit-learn. If you need a text analysis refresher, please check .

# importing the libraries
import os
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans

%matplotlib inline

There are 35 constitutions in our dataset. Most of the documents were queried from on April 2021, using a simple crawler that implements the Beautiful Soup .

If you intend on getting your data from the source above, and as a general good scraping etiquette, please be nice and avoid overburdening their servers with requests.

After importing the necessary libraries, we read our documents to a DataFrame.

# Construct an empty DataFrame with two columns
df = pd.DataFrame(columns=('document', 'content'))
# Go through the files in working directory
# If it's a text file, open it and append the content to the DataFrame
for filename in os.listdir(os.getcwd()):
 if filename.endswith('.txt'):
 df = df.append({"document": filename[:len(filename)-4],
 "content": open(filename, encoding='latin_1').read().replace("\n"," ")},
 ignore_index=True)

We should check the names of the documents. Since pandas DataFrame columns are Series, we can pull them out and call .tolist() to turn them into a Python list.

print(df['document'].tolist())

['algeria_constitution', 'australia_constitution', 'austria_constitution', 'belgium_constitution', 'brazil_constitution', 'burkinafaso_constitution', 'china_constitution', 'costarica_constitution', 'ecuador_constitution', 'france_constitution', 'germany_constitution', 'india_constitution', 'japan_constitution', 'korea_constitution', 'malaysia_constitution', 'mexico_constitution', 'morocco_constitution', 'netherlands_constitution', 'nigeria_constitution', 'norway_constitution', 'pakistan_constitution', 'peru_constitution', 'portugal_constitution', 'rwanda_constitution', 'senegal_constitution', 'singapore_constitution', 'southafrica_constitution', 'spain_constitution', 'sweden_constitution', 'switzerland_constitution', 'tunisia_constitution', 'turkey_constitution', 'us_constitution', 'vietnam_constitution', 'zambia_constitution']

As we can see, the constitutions pertain to different countries from around the world, with different systems of government. Some nations from the list are monarchies, others are republics. Some have a unitary government, while others are federal. And the differences extend to the legislature as well. It is, in fact, a diverse collection. But, we should keep in mind that most of these constitutions are translated from their respective native languages. The original meaning in each document may not be conveyed with the same degree of accuracy.

Text processing and exploratory analysis

Next, we begin the analysis. First, we choose a list of stopwords from the Natural Language Tolkit (nltk). These are high-frequency terms (like who and the), that we may want to filter out of documents before processing.

from nltk.corpus import stopwords
sw = stopwords.words('english')
sw.append('shall') # Add "shall" to stopwrods
print ("There are", len(sw), "words in this stopwords list. The first 10 are:", sw[:10])

There are 180 words in this stopwords list. The first 10 are: ['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're"]

Then, we are going to tokenize our texts. NLTK provides several types of tokenizers for that purpose. We will use a custom regular expression tokenizer, that detects words containing alphanumeric characters only.

from nltk import regexp_tokenize
patn = '\w+'
df['content'] = df['content'].apply(lambda w: " ".join(regexp_tokenize(w, patn)))

To have better insights from our text corpus, let us create 4 new columns:

df['number_words'] for the number of words in each document
df['unique_words'] for the number of unique words in each document
df['number_words_without_sw'] for the number of words that are not in the stopwords list
df['percentage'] for the percentage of stop words in each text corpus

df['unique_words'] = df['content'].apply(lambda x: len(set(x.split())))
df['number_words'] = df['content'].apply(lambda x: len(x.split()))
df['number_words_without_sw'] = df['content'].apply(lambda y: len([word for word in y.split() if word not in sw]))
df['percentage'] = 100 - (df['number_words_without_sw'] * 100 / df['number_words'])

Let us use .head() to preview the first 6 rows of our DataFrame.

df.head(6)

All of the percentage values are less than 50. In fact, we can surmise that a well written piece of legal document, should not have a lot of stopwords. However, that may not always be the case. Feel free to share your opinions regarding this, in the comments section below, or by email.

We can push the exploration further, and check which constitutions have the most and the least unique words. Such a measure can be an indicator of the richness of the used lexicon, and its complexity as well.

max_unique_words = df['unique_words'].max()
doc_max_unique_words = df['document'][df.index[df['unique_words'].idxmax()]]
min_unique_words = df['unique_words'].min()
doc_min_unique_words = df['document'][df.index[df['unique_words'].idxmin()]]

print ("The {} has the most unique words with {} words. And the {} has the fewest with only {}"
 .format(doc_max_unique_words, max_unique_words, doc_min_unique_words, min_unique_words))

The brazil_constitution has the most unique words with 5286 words. And the us_constitution has the fewest with only 1005

After the previous exploratory phase, we move to term weighting and tf-idf.

Like we did in the previous article linked above, we are going to use TfidfVectorizer from sklearn to convert the collection of documents to a matrix of TF-IDF features. That would allow us to take into account how often a term shows up. Again, tf-idf is a statistical measure that evaluates how relevant a word is to a document in a collection of documents. You can refer to the official sklearn for the complex mathematical explanation.

tfidf_vectorizer = TfidfVectorizer(stop_words=sw, use_idf=True)

x = tfidf_vectorizer.fit_transform(df.content)
tfidfcounts = pd.DataFrame(x.toarray(),index = df.document, columns = tfidf_vectorizer.get_feature_names())

K-Means clustering

Let us go over a brief explanation of clustering in general before delving into K-Means clustering that we will be using.

Clustering is the process of grouping a collection of objects, such that those in the same partition (or cluster) are more similar (in some sense) to each other, than to those in other groups (clusters). There are a lot of clustering algorithms that can be utilized, and their use is modulated by specific conditions in the use cases.

As for the K-Means , it clusters data by trying to separate samples in n groups of equal variance. It minimizes the squared distance between the cluster mean (centroid) and the points in the cluster. This algorithm requires the number of clusters to be specified.

Below, we set the desired number of clusters. That choice is not an easy task. There are a few ways to determine the optimal number of clusters, but for the sake of this demonstration, we will not be going through them.

# Specify number of clusters
number_of_clusters = 3
km = KMeans(n_clusters = number_of_clusters)
# Compute k-means clustering
km.fit(x)

KMeans(n_clusters=3)

After computing the k-means clustering, and getting a fitted estimator, we ought to see the top words in each cluster. In the code below, cluster_centers_ gets the coordinates of each centroid. Then, .argsort()[:, ::-1] converts each centroid into a descending sorted list of columns by their relevance. That gives the words most relevant, since in our vector representation, words are the features in the form of columns. We use .get_feature_names() to get a list of feature names mapped from feature integer indices. Finally, the for loop wraps up the work, and prints out the top words in each cluster.

order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = tfidf_vectorizer.get_feature_names()

for i in range(number_of_clusters):
 top_words = [terms[ind] for ind in order_centroids[i, :5]]
 print("Cluster {}: {}".format(i, ' '.join(top_words)))

Cluster 0: state may federal law section
Cluster 1: article law national state president
Cluster 2: federal art law confederation para

The top terms for each cluster are somewhat intriguing. In fact, certain words do pertain to specific systems of government. Let us check the full results, and see how the entire dataset was partitioned.

results = pd.DataFrame()
results['document'] = df.document
results['category'] = km.labels_
results

By observing the results and seeing the top words for each cluster, we can see that a lot of Federal countries were assigned to cluster 2. However, a few were put in other clusters. By the same, some non-Federal countries were grouped in cluster 2. The same goes for clusters 0 and 1, where similar systems of government are not always put together.

This suggests that word relevance on its own, just gives a broader perception of how well the text corpus reflects the system of government. The analysis can be improved by using other algorithms and techniques. However, K-Means clustering being fairly simple and easy to implement, can be a good starting point for further and deeper inspection.

Visualizing text corpus similarity

We can visualize the similarities based on the TF-IDF features. To do so, we start by constructing the vectorizer as usual. We specify max_features to build a vocabulary that considers only the top max_features ordered by term frequency across the dataset.

vectorizer = TfidfVectorizer(use_idf=True, max_features=10, stop_words=sw)
X = vectorizer.fit_transform(df.content)
print (vectorizer.get_feature_names())

['article', 'constitution', 'court', 'federal', 'law', 'may', 'national', 'president', 'public', 'state']

We can use a combination of DataFrame.plot and matplotlib to draw a scatter plot representing the distribution of two terms on x and y axes, and a colormap to showcase the rlevance of a third term. We can clearly see than only 10 data points had some value regarding the term federal, while the rest had a value of 0 or close.

df2 = pd.DataFrame(X.toarray(), columns=vectorizer.get_feature_names())
import matplotlib.pyplot as plt
fig = plt.figure()
axi = plt.gca()

ax = df2.plot(kind='scatter', x= 'federal', y= 'president', s= 250, alpha= 0.6,
 c='state', colormap='viridis',
 figsize= (12,10), ax= axi)

axi.set_title('President X Federal', fontsize=18)
axi.set_xlabel("Federal", fontsize=18)
axi.set_ylabel("President", fontsize=18)

Text(0, 0.5, 'President')

Advanced word analysis with TF-IDF

Wed, 21 Apr 2021 00:00:00 +0000

Table of Contents

Introduction and basic concepts

In a previous , we utilized CountVectorizer from scikit-learn to count words. We used bag of words analysis, where a text is represented as the bag of its words, disregarding grammar, and with no particular order. This model may capture the characteristics of the text or document.

However, there are some limitations with simple word count analysis. A better solution would be to use latent features, such as the frequency of words used in a document.

In fact, some terms will appear more often, carrying little useful knowledge about the document’s actual contents. Those very frequent words would shadow the frequencies of more uncommon yet more interesting terms. These problems can be tackled with . Tf means term-frequency while tf–idf means term-frequency times inverse document-frequency. It is a statistical measure that evaluates how relevant a word is to a document in a collection of documents. The TF–IDF value increases in relation to the number of times a word appears in a document, and is compensated by the number of documents in the corpus that contain the word, which helps to compensate for the fact that certain words appear more often than others.

import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
from nltk.stem.porter import PorterStemmer
import re
import requests

Our texts for this notebook are some constitutions. We use requests to make a request and get a response with the desired text.

tn_constitution = open("constitution.txt").read().replace("\n"," ") # Tunisian Constitution
us_constitution = requests.get("https://www.gutenberg.org/cache/epub/5/pg5.txt").text[2623:] # US Constitution
jp_constitution = requests.get("https://www.gutenberg.org/cache/epub/612/pg612.txt").text[610:] # Japanese Constitution
athen_constitution = requests.get("https://www.gutenberg.org/cache/epub/26095/pg26095.txt").text[610:] # Athenian Constitution
df = pd.DataFrame([
 { "document": "Tunisian Constitution", "content": tn_constitution},
 { "document": "United States Constitution", "content": us_constitution },
 { "document": "Japanese Constitution", "content": jp_constitution},
 { "document": "Athenian Constitution", "content": athen_constitution },])

In text analysis, the raw data cannot be fed directly to most algorithms, since these expect numerical feature vectors of a fixed size rather than raw text documents of variable length. In order to address this, there are ways to extract numerical features from text, namely:

Tokenizing : Word tokens are the basic units of text. When processing, the first step is to split strings into tokens and giving an integer id for each possible token.
Counting the occurrences of tokens in each document - how many times does a word appear in the text.
Normalizing and weighting with diminishing importance tokens that occur in the majority of documents.

We can specify a tokenizer when using CountVectorizer. Here, you find a stemming_tokenizer for reference. We will not be using it for this work.

Stemming is a text preprocessing task for transforming related or similar forms of a word to its base form (talking to talk, and cats to cat for example). We will use the Porter stemmer from nltk.

porter_stemmer = PorterStemmer()
def stemming_tokenizer(str_in):
 words = re.sub(r"[^A-Za-z0-9\-]", " ", str_in).lower().split()
 words = [porter_stemmer.stem(word) for word in words]
 return words

Let’s put it all together, and experiment with the CountVectorizer.

vectorizer = CountVectorizer(stop_words='english')

matrix = vectorizer.fit_transform(df.content)
counts = pd.DataFrame(matrix.toarray(), index = df.document, columns = vectorizer.get_feature_names())

Since our texts are all constitutions, we could have a look at some intriguing terms. But, what else should we be checking? Which words might be the most interesting? The idxmax pandas method would return the label of the column with the maximum value, for each row. That is, we’ll get the most frequent word for each document.

counts.idxmax(axis=1)

document
Tunisian Constitution article
United States Constitution shall
Japanese Constitution shall
Athenian Constitution council
dtype: object

Now, we look at this subset of words accross all documents.

counts[['people','constitution', 'rules', 'law', 'order', 'assembly', 'house', 'democracy','article','shall','council']]

Term Frequency

We’re going to take into account how often a term shows up by using the TfidfVectorizer in the same way as CountVectorizer. TfidfVectorizer converts a collection of documents to a matrix of TF-IDF features. It is equivalent to CountVectorizer followed by TfidfTransformer.

tfidf_vectorizer = TfidfVectorizer(stop_words='english', use_idf=False)

x = tfidf_vectorizer.fit_transform(df.content)
tfidfcounts = pd.DataFrame(x.toarray(),index = df.document, columns = tfidf_vectorizer.get_feature_names())

Let’s check the same words as we did before!

tfidfcounts[['people','constitution', 'rules', 'law', 'order', 'assembly', 'house', 'democracy','article','shall','council']]

Notice how our numbers have shifted a bit. These are supposedly better relative indicators for the use of words, and their importance in our documents.

Inverse document frequency

By looking at the previous DataFrame, it seems like the word (shall) shows up a lot. So, even though it’s not a stopword, it should be weighted a bit less.

This is inverse term frequency. The more frequent a term shows up across documents, the less important it can be in our matrix.

#use_idf bool, default=True (to highlight by comparison) Enable inverse-document-frequency reweighting
idf_vectorizer = TfidfVectorizer(stop_words='english', use_idf=True)

y = idf_vectorizer.fit_transform(df.content)
idfcounts = pd.DataFrame(y.toarray(), index = df.document, columns = idf_vectorizer.get_feature_names())

Again with the same subset of words accross all documents.

idfcounts[['people','constitution', 'rules', 'law', 'order', 'assembly', 'house', 'democracy','article','shall','council']]

Notice how (council) increased in value because it’s an infrequent term, and (people) decreased in value because it’s quite frequent.

It is beneficial to understand how TF-IDF functions in order to obtain a deeper understanding of how machine learning algorithms work. TF-IDF allows us to associate each word in a document with a numerical value or vector, that reflects its relevance in that document. In text analysis with machine learning, TF-IDF algorithms help extract keywords, and by determining similar documents, we are able to automatically sort them into clusters. Besides, given a query, variations of the TF-IDF weighting are also used by search engines in scoring and ranking a document’s relevance.

Counting words in Python with scikit-learn's CountVectorizer

Sat, 10 Apr 2021 00:00:00 +0000

Table of Contents

Introduction

In a previous , we used simple techniques to visualize and count words in a document. In this notebook, we will be using another technique. The from scikit-learn is more elaborate than the Counter tool. It converts a collection of text documents to a matrix of token counts.

from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd
import numpy as np

The text for our work would be an English version of the Tunisian constitution.

text = open("constitution.txt").read().replace("\n"," ")

Counting words with CountVectorizer

The vectoriser does the implementation that produces a sparse representation of the counts. The fit_transform() method learns the vocabulary dictionary and returns the document-term matrix, as shown below. This method is equivalent to using fit() followed by transform(), but more efficiently implemented.

vectorizer = CountVectorizer()

matrix = vectorizer.fit_transform([text])
matrix # notice the size of the matrix

<1x1741 sparse matrix of type '<class 'numpy.int64'>'
 with 1741 stored elements in Compressed Sparse Row format>

The numbers in the array below represent how many times a word showed up in the text.

matrix.toarray()

array([[2, 1, 1, ..., 1, 1, 3]], dtype=int64)

If we want to know which word is which, we can use get_feature_names() to get feature names from feature integer indices. The order of the words in this array matches the order of the numbers from the previous array. Here, we only output the last 10 words. The last one is youth, and according to the last output value from matrix.toarray(), it appeared 3 times in the text. The word younger appeared just once!

print (vectorizer.get_feature_names()[1731:])

['works', 'world', 'worship', 'writing', 'written', 'year', 'years', 'young', 'younger', 'youth']

We can use DataFrames to turn the results into a human-readable format.

counts_df = pd.DataFrame(matrix.toarray(), columns = vectorizer.get_feature_names())
counts_df

Even more, we can get a sorted list similar to the result given by Counter. We use some pandas magic to transpose index and columns, and the result is naturally a transposed DataFrame. In fact, the used property T is an accessor to the method transpose().

counts_df.T.sort_values(by=0, ascending=False).head(8)

As seen so far, the CountVectorizer is quite useful, and it can handle a lot of preprocessing for us. That would allow us to focus on the interpretation of data for example.

So, how many times did people appear in the text?

counts_df['people']

0 108
Name: people, dtype: int64

How about law and order?

print (counts_df['law'], '\n' ,counts_df['order'])

0 106
Name: law, dtype: int64
 0 9
Name: order, dtype: int64

Counting words in multiple documents

All of that is quite good and exciting. Now, we will see how is CountVectorizer with multiple text documents. We will be using the United States’ Constitution and the Athenian Constitution, by Aristotle in addition to our previous text.

To read a text file from a URL in Python, we make a request with Requests module to get a Response object. We can read the content of the server’s response by accessing .text.

import requests
US_constitution = requests.get("https://www.gutenberg.org/cache/epub/5/pg5.txt").text[2623:] # To slice out the unwanted text
Athenian_constitution = requests.get("https://www.gutenberg.org/cache/epub/26095/pg26095.txt").text[610:]

We construct a DataFrame with the content by passing the appropriate data.

df = pd.DataFrame([
 { "document": "Tunisian Constitution", "content": text},
 { "document": "United States Constitution", "content": US_constitution },
 { "document": "Athenian Constitution", "content": Athenian_constitution },])
df

Finally, we create an organized DataFrame of the words counted in each document. This time, we feed the entire content column the CountVectorizer instead of a single text variable.

vectorizer = CountVectorizer()

matrix = vectorizer.fit_transform(df.content)
counts = pd.DataFrame(matrix.toarray(), index = df.document, columns = vectorizer.get_feature_names())

counts

This is a nice feature where we can select serveral interesting words to check in all documents.

counts[['people','constitution', 'rules', 'law', 'order', 'assembly', 'house', 'democracy']]

In this notebook, we used CountVectorizer from sklearn to count words in multiple documents. It is more advanced than working with Counter and having to do all the text cleaning.

Visualization and analysis of legal texts

Thu, 08 Apr 2021 00:00:00 +0000

Table of Contents

While browsing the Internet, you have probably seen a picture of a cloud filled with words of varying sizes that reflect the frequency of each word within a given text. This is referred to as a Tag Cloud or a Word Cloud. In this tutorial (see the notebook ), we will learn how to make Word Clouds in Python. This tool is useful for a visual exploration of text data.

We will use legal texts for the purpose of this tutorial, namely the Tunisian Constitution and the Tunisian Hydrocarbons Code.

As usual, we start by importing the different libraries used. The NumPy library is used for handling large, multi-dimensional arrays and matrices. For visualization, matplotlib is a comprehensive plotting library. It enables other libraries, such as seaborn and wordcloud, to run on its base. The pillow library adds support for opening, manipulating, and saving many different image file formats.

from PIL import Image
from wordcloud import WordCloud, STOPWORDS
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import re
import seaborn as sns

Generating Word Clouds

Then, we read the Hydrocarbons Code stored in a .txt file. I chose this legal document, because I have worked on it and analysed it for a thesis I wrote. It is a complex legal text that pertains to a sensitive and important topic, that is natural resources.

We have to make some necessary text processing. First, we convert the entire text to lower case. This is important since Python strings are case sensitive. Afterwards, we use the magic of regular expressions to deal with apostrophes and other characters to be removed. Whenever in doubt regarding regular expressions syntax, you can use website or the official Python to have the desired outcome.

hydrocarbons_code = open("hydrocarbons_code.txt", encoding='latin_1').read().replace("\n"," ")
hydrocarbons_code = hydrocarbons_code.lower()
hydrocarbons_code = re.sub(".'|[^\w ]", " ", hydrocarbons_code)

Since the text is in French, we have to make our own list (technically it’s a Python set with curly brackets here) of the words to remove from the given text. These words would not show in the Word Cloud.

If you do not speak French, most of these words are the equivalent to English articles, pronouns and conjunctions. They would not help us much in understanding the text through visuals.

stop = {'de','du','des','et','est','la','le','les','en','ou','par','au','aux','dans','une','un','pour','sur','ce','ces',
 'ne','qui','que','son','ses','sa','il','ci','a'}

In order to create an image form for the Word Cloud, we need to use a PNG file as a mask. Here, I use the map of Tunisia, just for fun.

The mask argument in the WordCloud function takes an N dimensional array (ndarray). We use Image module to open the PNG file, and we transform it to the numpy array form. According to WordCloud documentation, all white entries will be considerd masked out, while other entries will be free to draw on. In the NumPy array, all white parts of the mask have a value of 255, whereas values of 1 are black.

map_mask = np.array(Image.open("map.png"))

Now, we have a proper mask and we can make a cloud with the desired shape. WordCloud takes several parameters, and you can create a personalised result by changing the optional arguments. Some of these are fairly self-explanatory. For the rest, you can always consult the relevant . Or, you can check out the docstring of the function and see the required and optional arguments, by typing and running ?WordCloud.

hydrocarbons_cloud = WordCloud(max_words=1000, mask=map_mask, stopwords=stop , min_word_length = 3, min_font_size = 8,
 margin=5, random_state=1, background_color="white", include_numbers=True).generate(hydrocarbons_code)

Finally, we can output the result and have an insightful and beautiful visualizations. Naturally, the most frequent words are hydrocarbons, code, article and holder, as expected in a legal document about hydrocarbons!

%matplotlib inline
plt.figure(figsize=(18,18))
ax = plt.gca()
ax.set_title("Tunisian Hydrocarbons Code Cloud",fontsize=22)
plt.imshow(hydrocarbons_cloud.recolor(colormap=sns.color_palette(palette='blend:red,brown',as_cmap=True), random_state=3),
 interpolation="bilinear")
plt.axis("off")
plt.show()
#hydrocarbons_cloud.to_file("Hydrocarbons code cloud.png") #comment out to save the figure in a PNG format

Now, let’s take a look at the Tunisian Constitution. We will use an English translation from .

As before, we start by constructing a mask. For this example, it will be the Tunisian flag. For the rest, the only difference is that we use the default built-in STOPWORDS list.

flag_mask = np.array(Image.open("Flag_of_Tunisia.png"))

tunisian_constitution = open("constitution.txt").read().replace("\n"," ").lower()
tunisian_constitution = re.sub("[^\w ]", " ", tunisian_constitution)

constitution_cloud = WordCloud(max_words=1000, mask=flag_mask, stopwords=STOPWORDS , min_word_length = 3, min_font_size = 8,
 margin=5, random_state=1, background_color="white", include_numbers=True).generate(tunisian_constitution)

Our exquisite result looks good, and presents us with useful visual anchor.

%matplotlib inline
plt.figure(figsize=(20,10))
ax = plt.gca()
ax.set_title("Tunisian Constitution Cloud",fontsize=22, y=1.04)
plt.imshow(constitution_cloud.recolor(colormap=sns.color_palette(palette='blend:red,brown',as_cmap=True)),
 interpolation="bilinear")
plt.axis("off")
plt.show()
#constitution_cloud.to_file("Tunisian Constitution Cloud.png") #comment out to save the figure in a PNG format

Counting words

However, text analysis goes beyond visualisations. We can count words using the collection. If we are only interested in the most common words, we can use .most_common() with or without an argument specifiying the number of words.

In the code below, we use list comprehension to store words appearing more than 30 times in the constitution and having at least 4 letters.

from collections import Counter
tunisian_words = [x for x in Counter(tunisian_constitution.split()).most_common(50) if x[1]>30 and len(x[0])>3 ]
tunisian_words

[('article', 173),
 ('shall', 173),
 ('assembly', 149),
 ('president', 110),
 ('people', 108),
 ('government', 100),
 ('representatives', 97),
 ('with', 97),
 ('republic', 96),
 ('state', 67),
 ('members', 62),
 ('court', 62),
 ('their', 50),
 ('draft', 48),
 ('head', 47),
 ('constitutional', 45),
 ('that', 43),
 ('laws', 42),
 ('within', 40),
 ('from', 39),
 ('right', 38),
 ('council', 37),
 ('judicial', 37),
 ('authorities', 34),
 ('local', 33),
 ('national', 32),
 ('after', 32),
 ('rights', 31),
 ('constitution', 31)]

Comparing different documents

We can compare the occurences of words in two documents as well. We will use the French constitution for comparison, despite the differences between the governance systems of the two countries as implemented in their respective constitutions.

We read and process the .txt file as we did before. Then, we store the words of each document in a list (tunisian_words and french_words).

french_constitution = open("french constitution.txt").read().replace("\n"," ").lower()
french_constitution = re.sub("[^\w ]", " ", french_constitution)

tunisian_words = tunisian_constitution.split(" ")
french_words = french_constitution.split(" ")

Afterwards, we construct a DataFrame by passing in the Counter collections as data entries, and setting the column labels. In this example, we chose to remove missing values or drop them by using .dropna() with a 0 argument. That means, we drop rows which contain missing values. That would give us a DataFrame of only the words that exist in both documents!

The result below shows the number of occurrences of each word, the total, and a percentage value indicating the prevalence of that word in the Tunisian constitution with reference to its total use in both documents.

df = pd.DataFrame({
 'Tunisian_constitution': Counter(tunisian_words),
 'French_constitution': Counter(french_words)
}).dropna(0)

df['Total'] = df.Tunisian_constitution + df.French_constitution
df['Tunisian_percentage'] = (df.Tunisian_constitution / df.Total) * 100

df.head(10)

All of these common words between the two documents were used differently in each. We can check how many times they appeared by returning the sum of the values over the desired axis.

df.sum(axis = 0)

Tunisian_constitution 11950.000000
French_constitution 11769.000000
Total 23719.000000
Tunisian_percentage 40472.171694
dtype: float64

Now, let’s look at words used ten or more times in the Tunisian constitution. We sort them by descending value. The for loop is used to go through df.index and remove the words having three characters or less.

for ind in df.index:
 if len(ind)<4:
 df.drop(ind, inplace = True)

df[df.Tunisian_constitution >= 10].sort_values(by='Tunisian_constitution', ascending=False)

We can use the df.sum() again to compute the frequency of those words.

df.sum(axis = 0)

Tunisian_constitution 5109.000000
French_constitution 5125.000000
Total 10234.000000
Tunisian_percentage 34301.487972
dtype: float64

In this notebook (using python 3.7 pandas 1.2.1 and matplotlib 3.3.2), we have learned how to draw a Word Cloud that would be helpful for visualization of any text. Besides, we used Counter to count words in documents. The tool worked well with pandas DataFrames, allowing us to make simple comparisons.

This might have been naive text analysis, but it is an important first step towards a more comprehensive and elaborate text analysis.

Les bases de Python

Sun, 12 Aug 2018 00:00:00 +0000

Table of Contents

Utiliser Python comme une calculatrice

Après avoir , vous pouvez ouvrir une console ou l’invite de commande, tapez python et un intepréteur Python s’ouvre. Pour l’instant, nous utilisons ce . Essayons quelques commandes Python simples

2+3 # Un commentaire

20 + 11 * 3

(20 - 11) / 4

Les nombres entiers (comme 2, 3 et 11) sont de type int, alors que les décimaux (comme 10.0 et 3.14) sont de type float.

19 / 3 # La division (/) donne toujours un nombre de type 'float'

19 // 3 # L'opérateur (//) effectue des divisions entières

19 % 3 # L'opérateur (%) donne le reste de la division entière.

Il est possible de calculer des puissances (x**y) avec l’opérateur **

3 ** 2
# 11 ** 12. # les opérations avec des types d'opérandes mélangés donnent un résultat en virgule flottante

Le signe égal ( = ) permet d’affecter une valeur à une variable

hauteur = 7
base = 9
aire_du_triangle = (hauteur * base) / 2
aire_du_triangl # parlons des erreurs dans l'affichage des résultats

Les chaînes de caractères

Les chaînes de caractères peuvent être exprimés de différentes manières :

'Bonjour' # guillemets simples
"Hello" # guillemets doubles

'de l\'art' # utiliser \' pour protéger les guillemets
# "de l'art" # ou utiliser les guillemets doubles

'"Le petit chat est mort. ", dit Agnès'
# "\"Le petit chat est mort. \", dit Agnès"

La fonction print() affiche les chaînes de caractères de manière plus lisible.

'"L\'art !" dit Michel.'
# print ('"L\'art !" dit Michel.')
# p = 'Première ligne.\nDeuxième ligne.' # \n signifie nouvelle ligne
# p
# print (p)

Utilisez les chaînes brutes (raw strings) en préfixant la chaîne d’un r, pour éviter que les caractères précédés d’un antislash ne soient interprétés comme étant spéciaux.

print('C:\Documents\nom')
# print(r'C:\Documents\nom') # les guillemets précédés par (r)

Utilisez des triples guillemets : '''abc''' ou """xyz""" pour écrire des chaînes de caractères qui s’étalent sur plusieurs lignes.

Empêcher le retour à la ligne en ajoutant \

print("""\
Définitions:
 -dictionnaire Une structure de donnée associant des clefs et des valeurs
 -fonction Une suite d'instructions qui renvoient une valeur à celui qui l'appelle
""")

L’opérateur + permet de coller (concaténer) plusieurs chaînes. L’opérateur * permet de répéter les chaînes.

'OUI ' * 3 + 'Hurrah!'
# prefix = 'Hello Wo'
# prefix + 'rld!'

Les caractères peuvent être accédés par leur position. Pour l’indexation des chaînes de caractères, le premier caractère est à la position 0.

phrase = "Je m'appelle Brian"
phrase[0]
# phrase[15]

Pour effectuer un décompte en partant de la droite, nous utilisons des indices négatifs (commencent par -1).

phrase[-1]
# phrase[-3]
# phrase[-18]

Pour obtenir une sous-chaîne :

phrase[0:2] # caractères de la position 0 (inclut) à 2 (exclu)
# phrase[13:18]
# phrase[:12]
# phrase[12:] # s[:i] + s[i:] = s
# phrase[-5:]
# phrase[20] # indice trop grand (hors bornes)
# phrase[5:20] # gérés silencieusement si utilisés dans des tranches

Les chaînes de caractères sont immutable : elles ne peuvent pas être modifiées.

phrase = "Je m'appelle Brian"
phrase[1] = 'j'
# phrase[13:] = 'Stewie'
# phrase[:13] + 'Stewie !'

La fonction len() donne la longueur d’une chaîne :

p = 'I have a dream'
len(p)

Les listes

Une suite d’éléments séparés par des virgules, placés entre crochets. Les éléments d’une liste ne sont pas obligatoirement du même type.

premiers = [2, 3, 5, 7, 11]
premiers

Les listes peuvent être indicées et découpées :

premiers[0]
# premiers[-1]
# premiers[2:10]

Les opérations de découpage (en tranches) renvoient une nouvelle liste contenant les éléments spécifiés.

premiers[:] # une copie de la liste

Les listes supportent des opérations comme pour les chaînes de caratcères.

premiers + [13, 17, 19]
# premiers * 2

Il est possible de changer le contenu des listes : elles sont mutables

nbres = [1, 2, 'c', 4]
# nbres[2] = 3
nbres

nbres.append(5) # méthode pour ajouter des éléments à la fin
nbres

nbres[2:4] = [10, 100] # affectation par tranches
# nbres[:] = [] # supprimer toutes les valeurs
nbres

Il est possible de créer des listes contenant d’autres listes :

a = [1, 2, 3]
b = ['f', 'r', 'a']
res = [a, b]
# res
# res[1]
# res[0][2]

La compréhension des listes (list comprehension) permet de construire des nouvelles listes où chaque élément est le résultat d’une opération appliquée à chaque élément d’une autre séquence; ou de créer une sous-séquence d’éléments satisfaisants une certaine condition. Elle consiste en deux crochets contenants une expression suivie par une clause for, puis par une ou plusieurs clauses for ou if.

[x for x in range(6) if x % 2 == 0]

[(x, y) for x in [1,2,3] for y in [3,1] if x != y]

Contrôle du flux

L’instruction `if` :

x = 11
if x % 2 == 0:
 print ('foo')
 s = 'nombre pair'
else:
 print ('bar') # x % 2 != 0

x = 20
if x % 5 == 0 and x % 2 == 0:
 print ('foobar')
elif x % 5 == 0:
 print ('foo')
else:
 print ('bar')

L’instruction `for` :

Elle permet d’itérer sur les éléments d’une séquence (une liste, une chaîne de caractères, etc.) par ordre.

pays = ['France', 'Canada', 'Belgique', 'Suisse']
for p in pays:
 print (p, len(p))

mot = '1 chat'
for c in mot:
 print (c)

for i in range(1, 4):
 print (i ** 2)

L’instruction `while` :

x = 1
while x <= 5:
 print (x, end=' ') # Le paramètre (end) sert à enlever le retour à la ligne, ou terminer par un autre caractère
 x += 2 # x = x + 2

Les fonctions :

Le mot-clé def définit la fonction. Il est suivi du nom de la fonction, et de ses paramètres entre parenthèses.

def add(n): # calculer la somme des chiffres de 1 à n
 somme = 0 # une variable locale
 for i in range(1, n+1):
 somme += i
 return somme
add(3)
# add(10)

# somme # n'est pas défini 'globalement'
# somme = 100 # variable globale
# somme

def fib(a, b, n):
 """ print une suite de fibnoacci à partir des termes a et b, jusqu'à n """
 while a < n:
 print(a, end=' ')
 a, b = b, a+b
fib(5, 8, 2000) '''{même les fonctions sans instruction
 return renvoient une valeur, quoique ennuyeuse. Cette valeur est appelée None }'''

Structures de données

Tuples

Une séquence d’éléments séparés par des virgules (et encadrés par des parenthèses si nécessaire)

t = 'python', 3.5, 101
t
# t[2]

# n = t, 'Guido', ('2.7', 2010), [0, 90] # tuples imbriqués
# n

n[1] = 'Guido van Rossum' # ils sont immutables
# n[3][1] = 100
n

vide = () # initier un tuple vide
# un = 'python',

Les ensembles (`sets`)

Une collection non ordonnée, sans élément dupliqué.

notes = {18, 15, 14, 11 ,18, 14}
notes
# fruits = {'orange', 'raisin', 'pomme' ,'kiwi' , 'orange', 'pomme'}
# fruits

a = set('abccba')
a
# a & set('atta') # supportent d'autres opérations (unions, intersections, différences, etc.)

Dictionnaires

Des ensembles non ordonnés de pairs clé : valeur (key : value pairs). Ils sont indexés par des clés (keys), qui peuvent être de n’importe quel type immuable : chaînes de caractères, nombres et tuples (s’ils ne contiennent que des immutables). Les clés doivent être uniques au sein d’un dictionnaire.

d = {} # créer un dictionnaire vide

# d = {'Marie':15, 'Jean':2, 'Victor':9}
# d['Marie']
# del d['Victor']
# d['Charles'] = 33
d

p = dict([('Anthony', 'a'), ('Guido', [0, 'g']), ('Marie', 'm')])
# p.keys()
# p.values()
# 'guido' in p
# 'Guido' in p

Création des dictionnaires par compréhension (dict comprehensions).

{x: x**3 for x in range(1, 6)}

Modules

Un module est un fichier contenant des définitions et des instructions. Le nom du fichier est celui du module, suffixé de .py.

import <module> # importer le module dans la table des symboles

<module>.<fonction> # accéder aux fonctions (ou constantes)

Pour importer les noms d’un <module_1> directement dans la table des symboles du module qui l’importe (<module_2>). De ce fait, le nom du <module_1> n’est pas défini à l’intérieur du <module_2>

from <module> import <fonction_1>, <fonction_2>, <fonction_n>

Ou bien

from <module> import * # importer tous les noms du module (déconseillé)

Exercice pratique de synthèse

Définir une fonction somme_impairs qui prend une liste de nombres comme argument, et renvoie la somme de tous les entiers impairs. Par exemple :

somme_impairs([5, -13, 3]) = -5
somme_impairs([2, 1, 1.01) = 1

Écrire un programme qui prend une valeur entrée par l’utilisateur (1 <= N <= 100), et affiche un message comme suit :

 - "Votre valeur X est plus grande. Essayer de nouveau !" # si la valeur entrée est plus grande à celle du programme
 - "Votre valeur X est plus petite. Essayer de nouveau !" # si la valeur entrée est plus petite à celle du programme
 - "Bravo ! C'était bien X" # si la valeur entrée est correcte

Avec X est la valeur entrée par l’utilisateur. Penser à utiliser la fonction randint du module random pour générer un entier aléatoirement.

from random import randint
n = randint(1, 100)

Pour plus de détails sur n’importe quel objet, fonction ou module, utiliser : help(<nom>)

Blog |

Problem Solving Using Computational Thinking - Course Review

Course overview and structure

Course review

K-Means clustering and similarity visualization of constitutions

Introduction

Text processing and exploratory analysis

K-Means clustering

Visualizing text corpus similarity

Advanced word analysis with TF-IDF

Introduction and basic concepts

Term Frequency

Inverse document frequency

Counting words in Python with scikit-learn's CountVectorizer

Introduction

Counting words with CountVectorizer

Counting words in multiple documents

Visualization and analysis of legal texts

Generating Word Clouds

Counting words

Comparing different documents

Les bases de Python

Utiliser Python comme une calculatrice

Les chaînes de caractères

Les listes

Contrôle du flux

L’instruction if :

L’instruction for :

L’instruction while :

Les fonctions :

Structures de données

Tuples

Les ensembles (sets)

Dictionnaires

Modules

Exercice pratique de synthèse

L’instruction `if` :

L’instruction `for` :

L’instruction `while` :

Les ensembles (`sets`)