Gaming

Posts about the games I have played.

,

Automated Twitter User Tagging Using Natural Language Processing

Introduction

Previously, some EDA and interactive data visualization have been implemented to Twitter data. In this post, I will talk about how to use unstructured text data from Twitter to build an automated user-tagging application. Since Twitter data is text-rich, the primary approach to handle Twitter data is Natural Language Processing (NLP). NLP teaches the computer about how to understand natural languages such as English. Combined with Machine Learning, it is possible to train a model that makes decisions based on the text input. In other words, based on NLP, we can train classifiers to do various tasks such as user clustering. For example, in this post, I will walk you through the process of building a binary classifier to predict if the Twitter user is a programmer or a gamer using that user’s profile as the input.

Data Ingestion

Following the steps in my previous post about streaming and storing tweets in a PostgreSQL database, I launched another round of Twitter streaming, using keywords in [python, programming, java, C++, fortnite]. The first 4 keywords capture users that are more likely to be programmers, and the last keyword, fortnite, on the other hand, captures users that are more likely to be gamers. The streaming last around 72 hours results in 299741 non-duplicate tweets tweeted from 195660 Twitter users.  To ensure the data purity, two additional conditions are applied to clean the raw samples:

  1. The user’s profile description is not empty and is written in English.
  2. The user’s tweets contain label information, in other words, the tweets under this user must include one of the five keywords specified, so the user can be correctly labeled.

After filtering, the total number of user reduce to 104303.  Figure 1 shows the first ten users and their profile description. 

Figure 1. First ten records of the user dataset.
Figure 1. First ten records of the user dataset.

Data Preprocessing

As mentioned in my last post, computational hardware can only handle numerical data, which is eventually converted into binary bytes. It is not possible that we simply feed the description column in Figure 1 to the machine and tell it, learn from this. The ultimate goal of text recognition is to convert the text data into numerical data and train our computer to be able to understand the underlying information. Natural Language Processing (NLP) does precisely this. NLP is a broad field of study that covers many cutting-edge challenges, such as sentiment analysis, topic recognition, and high-precision translation, etc. The model I talk about here belongs to topic-recognition, and there are several crucial steps to follow before building the machine learning model. 

Tokenization

Tokenization means to split the document into individual words. In our case, not only the whole description needs to be divided into a list of different words, but it also requires a fair amount of cleaning work, so all the symbols, emojis, and web links are removed. For example, define a cleaning function that takes the raw text as input and generates a cleaned list of words as output, as shown below: 

The steps followed to generate the processed data are the following:

  1. Lowercasing all the letters in the sentence.
  2. Tokenize the sentence (split the whole string into words).
  3. Keep only alphabetic words (remove other characters, e.g. emojis, Japanese symbols).
  4. Remove all stopwords (such as and, the).
  5. Lemmatize all tokens.

Lemmatization is the process of returning only the base of a word, which is called lemma of the word. For example, the lemma of words doggy, dogs is dog.

Data-Balancing

For any classification problem, if the dataset is skewed or imbalanced, the model would be biased towards the class that has more samples. The Twitter-user dataset is pre-labeled in the database by pattern matching their tweets. The distribution of the two classes is shown in Figure 2: 

Figure 2. The number of labels for all Twitter users.
Figure 2. The number of labels for all Twitter users.

Even though I used three other keywords to capture tweets from programmers, the number of programming users are still not even half of the number of gamers. There are possible approaches to balance the dataset: undersampling and oversampling. While either of them can eliminate the impact of the imbalanced dataset, they have different drawbacks and should be wisely chosen with different situations. In our case, since we do not have a massive amount of data, if we decide to undersample the gamer class, half of the samples would be dropped. For data analytics, reducing data is absolutely the least approach to go for. Hence here we choose to oversample the programmer class. The problem with this approach, though, is that the additional samples are just duplicates of random samples. 

To oversample the dataset, I use the imbalanced-learn library. Imblearn provides multiple strategies for re-sampling the dataset such as Naive Random Oversampling, Synthetic Minority Oversampling Technique (SMOTE), and Adaptive Synthetic sampling method for oversampling, Prototype Generation and Prototype Selection for undersampling. After applying Naive Random Oversampling to the original dataset, the amount of samples has been balanced to $67289$ for each class. The number is smaller since after preprocessing, some profiles are removed for being empty after removing all non-alphabetical words. 

Model Implementation

There are numerous machine learning models for binary classification, such as Logistic Regression, Random Forest, and Naive Bayes. To pick the best model for our application, we can write a python script to loop through the model pool and plot their performance to decide which is the best fit. With all default parameters for all the models and a small amount of training data, one can get a general idea about how the models would perform on a specific classification problem. From there, based on the initial performances, pick the best model to fine-tune its hyperparameters to achieve the best score.

To roughly train and evaluate all the models, two essential libraries are necessary: scikit-learn and yellowbrick, scikit-learn for building machine learning pipeline and yellowbrick for both model evaluation and report visualization. 

Implementing a machine learning pipeline is a great point to start. With the model as the last section in the pipeline, one can easily switch between different machine learning models with the same data configuration. The first section in the pipeline is CountVectorizer.

CountVectorizer

Since the preprocessing only tokenize all the descriptions, to enable the computer to understand the text content, the next step is to transform all the text information into numbers. First, similar to a hashtable, we give each word in the user’s description a unique index, and the number is decided by the appearances of that word among all the descriptions. For instance, as shown below, we first count the number of all the words in our dataset, then assign a number to each word. And the order of the indexing is the same with the descending order of the counts, which means 0 stands for the most frequently used word (http), and 1 stands for the second most used word (fortnite), and so on.  

It turns out that there are 89548 unique tokens in the entire dataset, which means the indices are in the range of [0, 89547]. Now we can use a vector that contains 89548 elements to represent a specific user description by putting 1s to the corresponding indices. For example, If fortnite is in the description and http is not, the first element in the vector would be 0, and the second element would be 1, and so on. However, it is not possible to store vectors with 89547 elements in memory as many as we want due to memory limit. In fact, the majority of the vector is composed of zeros, and only a small portion of it is ones, which is also called a sparse vector. Large-size sparse vectors are usually stored in sparse representations where only the indices and the corresponding non-zero values are stored. A simple example is shown below:

Here the dense vector [0, 0, 2, 1, 0] is transformed into a sparse matrix by scipy. In output 61, it says there are only 2 elements stored, which is the non-zero values of 2 and 1. And the printed information shows that it is stored with the indices (0, 2), (0, 3) (row 0, columns 2, 3) and the value 2, 1.

Fortunately, the CountVectorizer() module in scikit-learn does all the heavy lifting mentioned above, including counting, indexing, and saving the results as a sparse matrix. 

TfidfTransformer

While CountVectorizer handles all the work of converting text information into equivalent numerical values, which might work fine in most cases. However, it would be even better if a TfidfTransformer is attached next to the CountVectorizer. Tf stands for term-frequency, which is a normalization of the output from the CountVectorizer, and tf-idf means term-frequency times inverse document-frequency.

The formula for calculating tf-idf of a term t in a document set d is

tfidf(t, d) = tf(t, d) * idf(t)

where tf(t, d) calculates the term frequency of t in a document set, and

idf(t) = log[n/df(t)] + 1

computes the inverse ducoment frequency of term t, where df(t) is the document frequency of t. The document frequency is the number of documents in the set that contains the term t.

From the formula, when the document frequency of term t is approaching the number of entire document set n, the logarithmic would be approaching 0. And the weight for the term t would go towards 1. Whereas the term that has a lower document frequency would have a higher idf weight to magnify its term-frequency tf(t, d).

From the math behind tf-idf, one can tell that the goal of it is to scale down the impact of tokens that occur in the majority of the document set hence are empirically less informative than the tokens are occur in a smaller portion of the document set.

Similarly, TfidfTransformer module in scikit-learning receives the sparse matrix from the CountVectorizer and transforms it into an inverse-document-frequency weighted sparse matrix. 

Model Selection

So far, two sections of the pipeline are implemented, to finish up the pipeline, append a machine learning model after the TfidfTransformer. The next step is to pick a specific model that fits our classification problem the best. With a pre-defined model pool, a simple for loop would do the trick:

# pipeline
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
# classification models
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import MultinomialNB
from sklearn.svm import LinearSVC
# performance visualization tools
from yellowbrick.classifier import ClassPredictionError
from yellowbrick.classifier import ClassificationReport
from yellowbrick.classifier import ROCAUC
from yellowbrick.classifier import ConfusionMatrix
# train and visualize the results of each model individually
for model in [LogisticRegression, RandomForestClassifier, LinearSVC,
              KNeighborsClassifier, MultinomialNB]:
    # instantiate a pipeline
    pipeline = Pipeline([
    ('vect', CountVectorizer()),
    ('tfidf', TfidfTransformer()),
    ('model', model()),
    ])
    # print the current model in the pipeline
    print(model)
    # for four different metrics and visualization
    for viz in [ClassificationReport, ClassPredictionError,
                ROCAUC, ConfusionMatrix]:
        # print current model and visualization combination
        print(model, viz)
        # fit and evaluate the model
        viz1 = viz(pipeline)
        viz1.fit(X_train, y_train)
        viz1.score(X_test, y_test)
        plt.figure()
        viz1.poof()

 

It took around 5 mins to execute the for-loop for five models and four visualizations for each model with a single CPU core. And the results are shown in Figure 3- Figure 7.

Figure 3. Visualization of metrics for Logistic Regression Classifier.
Figure 3. Visualization of metrics for Logistic Regression Classifier.
Figure 4. Visualization of metrics for Random Forest Classifier.
Figure 4. Visualization of metrics for Random Forest Classifier.
Figure 5. Visualization of metrics for Linear SVC (Support Vector Classifier).
Figure 5. Visualization of metrics for Linear SVC (Support Vector Classifier).
Figure 6. Visualization of metrics for K-Neighbors (KNN) Classifier.
Figure 6. Visualization of metrics for K-Neighbors (KNN) Classifier.
Figure 7. Visualization of metrics for Naive Bayes Classifier.
Figure 7. Visualization of metrics for Naive Bayes Classifier.

As shown in the figures above, five models (from top to bottom: Logistic Regression, Random Forest, Linear SVC, K-Nearest Neighbor, and Multinomial Naive Bayes) are trained and evaluated individually (from left to right: prediction error, confusion matrix, precision-recall and f1-score, and roc-curve). In the prediction plots (first column), blue area denotes the number of predictions for class 0 (programmer), and the green area stands for predictions for class 1 (gamer).

It is clear that Random Forest, Linear SVC, and Naive Bayes classifiers can provide pretty good scores while K-Nearest Neighbors (KNN) classifier is not able to handle the task. Since the dimension of the dataset is too high (up to 80000 dimensions), the distance between two samples in the hyperspace can be tremendous thus impact the performance of the voting strategy of KNN algorithm.

Cross-Validation & Model Tuning

Figure 8 shows the benchmark profiling for all five models. Even though Random Forest Classifier has a pretty good performance, the training time is too long to be considered. Therefore, the model pool is shrunk into two models: Linear SVC and Naive Bayes. The next step is hyperparameter tuning and cross-validation.

Figure 8. Benchmark profiling for all the models.
Figure 8. Benchmark profiling for all the models.
GridSearchCV

GridSearchCV is a submodule provided by scikit-learn, which combines grid search and cross-validation together into one entity. It explicitly searches over a specified set of parameters. For each combination of parameters, it performs k-fold cross-validation automatically. Execute the following code to find the best hyperparameters for Naive Bayes classifier:

# instantiate the pipeline with some predefined parameters
pipeline = Pipeline([
    ('vect', CountVectorizer(ngram_range=(1, 2))),
    ('tfidf', TfidfTransformer(use_idf=True)),
    ('nb', MultinomialNB(fit_prior=False)),
])
# specify parameter grid to search over
parameters = {
    'vect__max_df': (0.25, 0.5),
    'nb__alpha': (0, 0.1, 0.2, 0.3, 0.4)
}
# initialize the GridSearchCV object
grid_search = GridSearchCV(pipeline, parameters, cv=5,
                           n_jobs=8, verbose=10)
# print progress to the console
print("Performing grid search...")
print("pipeline:", [name for name, _ in pipeline.steps])
print("parameters:")
pprint(parameters)
t0 = time()
# start searching
grid_search.fit(X_train, y_train)
print("done in %0.3fs" % (time() - t0))
print()
# print the best paramters found
print("Best score: %0.3f" % grid_search.best_score_)
print("Best parameters set:")
best_parameters = grid_search.best_estimator_.get_params()
for param_name in sorted(parameters.keys()):
    print("t%s: %r" % (param_name, best_parameters[param_name]))

 

To reduce the search time, some basic parameters are predefined when initializing the pipeline, such as ngram_range for CountVectorizer and use_idf for TfidfTransformer. ngram_range stands for the range of n-gram to be extracted. Here I choose to only consider 1-gram and 2-grams sequences.

Two parameters for two modules in the pipeline are explicitly set in the parameter grid, including max_df for CountVectorizer and alpha for Naive Bayes classifier. Specifically, max_df stands for maximal document-frequency, which let the CountVectorizer only to use the tokens that have the lower document-frequency to build the vocabulary. In other words, all the tokens that have a document-frequency higher than this threshold will be ignored so that the classifier can be more focused on other relatively more informative tokens. On the other hand, alpha is the smoothing coefficient that counts for unseen feature points to avoid zero probabilities in downstream calculations. 

The output of executing the code is shown below:

There are 10 different combinations of parameters to fit and a 5-fold cross-validation for each model; thus, 50 fits in total, as shown in the output. For an 8-core CPU, it took 96 seconds to find the best model with 5-fold CV. The corresponding parameters for the best model are picked, with alpha=0.1 and max_df=0.25. And the evaluation of the best model is shown in Figure 9: 

Figure 9. Evaluation for Naive Bayes Classifier after grid-search and cross-validation.
Figure 9. Evaluation for Naive Bayes Classifier after grid-search and cross-validation.

Compares to the default model, the best model after hyperparameter tuning has better performance: the precision for class 1 has increased from 0.852 to 0.92, the recall for class 0 has increased from 0.858 to 0.927, and the AUC has increased from 0.92 to 0.96, respectively.

Similarly, we can define a parameter grid for Linear SVC to find the best model, as shown in Figure 10: 

Figure 10. Evaluation for Linear SVC after grid-search and cross-validation.
Figure 10. Evaluation for Linear SVC after grid-search and cross-validation.

Conlusion

So far, the implementation of an automated Twitter user tagging model using NLP and machine learning models is done. Figure 11 illustrates a small demo of the application using the pre-trained Linear SVC model. I use the descriptions of two Twitter users as the input to generate their tags, i.e., to predict if he/she is a gamer or a programmer. One of them is Eefje Depoortere (@sjokz), who is my favorite E-sports host. And the other one is my friend Ben (@Benben_ben__), who is a programmer.  

demo1

Figure 11. A demo for an automated user-tagging application using pre-trained NLP model.

While the model is able to accurately predict the label for these two users, there is still space to improve. Such as multi-labeling and coupling with Computer Vision, etc. 

I had tons of fun during the process of developing this application. You can find all the code for this project here. As always, feel free to correct me if there is any mistake or concerns. Welcome to let me know if you have any questions.

Thanks for reading 🙂

,

Geographical Visualization and Network Analysis for Twitter Data

Introduction

In the previous post, I did some fundamental Exploratory Data Analysis (EDA) with the streamed Twitter data. While several essential pieces of information were extracted from those data, we were only looking at a tiny portion of the entire database. This post examines two other hugely impactful data sources: geographical data and social network data.  One can convert the geographical information to a figure by visualizing Twitter data on a geospatial map (like this demo shown here). Meanwhile, network analysis can be implemented by inspecting several user-based tables in our database.

Why put Twitter data on a map? First, it allows us to observe the concentration of the tweets collected spatially. Second, it enables us to work with the area of interest (if have one) more efficiently. Also, why network analysis? First, it helps to identify the user (or users) that has the most significant potential to impact a field. Second, it also helps clustering user base depend on some preliminary conditions. Of course, there are many other advantages of doing geographical and social-network analysis on Twitter data,  and some of them are right in this post.  

Visualize Twitter Data on a map

First, let us visualize the tweets collected geographically by denoting each one of them as a dot (circle) on a Mercator projected map. There are a few steps for data preparation:

  1. Only $1-3%$ of total tweets have geographical information embedded in their JSON file, hence we need to combine the base_tweets table with the place table, then filter out those tweets that do not have geographical information, to get those tweets that can be displayed on our map.
  2. When hovering the mouse on to a specific point shown on the map, it would be better to show some extra information related to that tweet, such as which game it is about, which user sent that tweet, how many followers that user currently has (thus how big is the impact of that tweet), etc. To do this, combine more tables into the base_tweets table, including tweet_user, hashtags, text, etc.
  3. Sometimes the keyword for the tweet is not contained in the hashtags or text field, and they might be mentioned in the quote tweet or retweet instead. To handle tweets like this, more tables need to combined into the base_tweets table, such as retweeted_tweet table. 

The following query does all the work I just mentioned. And the query results (10 records) are shown in Table 1.

Table1. Queried data for visualizing tweets geographically.
Table1. Queried data for visualizing tweets geographically.

From the query, seven tables in the relational database are combined through LEFT JOIN on different keys to provide the information needed. Now, export the query results as a CSV file and load them in Python.

Note that the geographical information for each tweet is stored as a bounding box on the map, and the coordinates column contains the four corners of the bounding box. To get the corresponding x and y coordinates on a Mercator-projected map, some mathematical calculations are necessary. For your reference, the functions I wrote for converting a bounding box into Mercator coordinates is shown below:

def get_lon_lat(df, geo_type):
    """
    Get longitude and latitude from the dataframe
    Append two new columns into the original dataframe
    Parameters
    ----------
        df: pandas dataframe
            input dataframe containing geographical information
        geo_type: string
            the type of the geographical dataframe
    Returns
    -------
        df: pandas dataframe
            with longitude and latitude extracted
    """
    # if the input dataframe is geo
    if geo_type == 'geo':
        # then the longitude is the second value in the list
        df['longitude'] = df['coordinates'].apply(lambda x: literal_eval(x)[1])
        # and the latitude is the first value
        df['latitude'] = df['coordinates'].apply(lambda x: literal_eval(x)[0])
    # if the input dataframe is coordinates
    elif geo_type == 'coordinates':
        # it is the other way around
        # NOTE: geo and coordinates contains the same information
        df['longitude'] = df['coordinates'].apply(lambda x: literal_eval(x)[0])
        df['latitude'] = df['coordinates'].apply(lambda x: literal_eval(x)[1])
    # if the input dataframe is place
    # then list contains 4 points of a bounding box
    # we need to calculate the centroid of the bounding box
    elif geo_type == 'place':
        # define a function to extract the centroid
        def get_centroid(row):
            # literalize the str to a list
            lst = literal_eval(row)
            # get the minimal and maximal latitude
            lat0 = lst[0][0][1]
            lat1 = lst[0][1][1]
            # calculate the center
            lat = (lat0 + lat1)/2
            # same for the longitude
            lon0 = lst[0][0][0]
            lon1 = lst[0][2][0]
            lon = (lon0 + lon1)/2
            # return the centroid
            return (lon, lat)
        # apply the defined function to the whole dataset
        df['longitude'] = df['coordinates'].apply(lambda x: get_centroid(x)[0])
        df['latitude'] = df['coordinates'].apply(lambda x: get_centroid(x)[1])
    # return the new dataset with longitude and latitude as new columns
    return df

def lonlat2merc(row):
    """
    Convert the longitude and latitude of a geographical point to
    x and y coordinates on a Mercator projected map
    Parameters
    ----------
        row: float, one row of a pandas dataframe
            Before apply this function to a dataframe
            make sure the dataframe contains two columns correspond
            to longitudinal and latitudinal numbers
    Returns
    -------
        (x, y): float tuple
            x and y coordinates of the points
    """
    # extract longitudinal and latitudinal info out of the dataframe row
    lon = row.iloc[0]
    lat = row.iloc[1]
    # do the conversion
    r_major = 6378137.000
    x = r_major * math.radians(lon)
    scale = x/lon
    y = 180.0/math.pi * math.log(math.tan(math.pi/4.0 + lat * (math.pi/180.0)/2.0)) * scale
    # return the X and Y coordinates
    return (x, y)
def add_xy_col(df):
    # simple function that calls the Merc conversion function to each row
    # and save x and y seperately to the dataframe
    df['merc_x'] = df[['longitude', 'latitude']].apply(lambda x: lonlat2merc(x)[0], axis=1)
    df['merc_y'] = df[['longitude', 'latitude']].apply(lambda x: lonlat2merc(x)[1], axis=1)
    return df

 

Once the x and y coordinates are available, one can put the tweets on to the map. Figure 1 shows the demo I have created:

figure1
Figure 1. Geographical map forTwitter data of 8 video games.

In this stand-alone bokeh plot, the circles for all the tweets are shaded in different colors for different games, and the size of the circle is determined by the number of followers of the user who sent that tweet.

By looking at Figure 1, one can quickly summarize several conclusions:

  1. Fortnite is super popular in North America and Europe, and Eastern South America compares to other regions.
  2. Dota 2 is more popular in South East Asia.
  3. Most North-Asian and African players do not use Twitter.
  4. Even though Apex Legends and Fortnite are comparably popular in Japan area as well, not many people like to attach their geo-location when they post on Twitter hence very few points on that side of the map.

On the other hand, if one zoom-in the map to the U.S. specifically, as shown in Figure 2:

figure2
Figure 2. Geographical map zoom in at the U.S.

The players for these games are mainly concentrated around Texas, California, and the Bay Area. You might already notice that there are only seven different colors for all the circles on the map while there are eight games in total, and the missing one is GTA5. As mentioned before, only a very small portion of all the tweets have geo-information embedded, unfortunately, none of the tweets about GTA5 contains geographical information. In fact, only 709 (out of more than 160k) tweets have their geo-location enabled. And the detailed numbers for all the point on the map are shown in Figure 3:

Bokeh Plot
Figure 3. Bar chart for the count of each game shown on the map.

In Figure 3, Fortnite has the most points on the map, which make it a little tricky if one wants to examine the geographical distribution of other games. Bokeh provides powerful tools that enable us to interact with the plot with an additional adjustment which the user can define themselves. These interactive plots are called Bokeh applications, which need to be hosted by connecting the code to a Bokeh server. When the user adjusts the parameters of the scheme, the server updates the data accordingly and replot the figure on the client end.

Figure 4 is a brief demonstration of the application. You can play with the app through this link, which is hosted on an AWS (Amazon Web Service) EC2 instance.

Figure 4. Interactive Bokeh application for video-game geographical visualization. The demo is recorded by Bandicam (free version), so you see its watermark on top of the image.
Figure 4. Interactive Bokeh application for video-game geographical visualization. The demo is recorded by Bandicam (free version), so you see its watermark on top of the image.

Now since there are more controls added to the plot, the geo-information from the Twitter data is much better interpreted:

  1. In the demo, there is a drop-down menu named Shading, which allows us to choose the way to shade the circles. It has two options, one called Game Name, which draws the colors by the name of the game, and you can refer to the legend on the bottom of the plot to distinguish between different games. The other option for the drop-down menu is Sentiment Score, which shades the circles depend on the sentiment scores of the tweets, the more negative the tweet is, the darker the circle would be. 
  2. A group of checkboxes below the Shading drop-down menu allows the user to pick which games they want to inspect individually on the map. This additional feature makes it easier to examine a specific game or a group of games.
  3. The last option of control is a slider called Hour, and it indicates the time window when the tweets were posted. Only those tweets posted during that one-hour window are shown on the map, by which one can examine the data on another domain: time domain. Set Hour equal to 50 to show all the tweets from all the hours. 

With all these tools available, here are another bunch of interesting conclusions about the data:

  1. CSGO is more prevalent in Europe than any other region in the world, while ApexLegends is more prevalent in North America.
  2. Dota2 and Hearthstone have higher sentiment score across the board.
  3. Most players’ tweets are neutral or slightly more favorable for these games. 

Network Analysis

There is a toolkit called NetworkX comes in handy when visualizing Network data. And in this section, I will explain how to extract the network information from the massive amount of Twitter data stored in our database and create highly interactive Bokeh plots to visualize the network.

There are three types of network existing in Twitter data, including reply-network, retweet-network, and quote-network. Each of these networks can convey different information about the community so we need to handle them individually. Firstly, extract the reply-network from the base_tweets table, specifically, the tweet_user_id and in_reply_to_user_id column. Then, use NetworkX to generate a network object from these two columns automatically.

For this experiment, I extracted the reply information only for Fortnite and removed the records where the Twitter user was replying to himherself, which results in a network with 1123 nodes and 860 edges. And the network is visualized in Figure 5 with two different network layouts. Note that you can check the id of the user by hovering your mouse on the nodes, and click-and-drag to select a group of nodes, and the edges and nodes connected to the selected nodes will be highlighted as well.

Bokeh Plot
Figure 5. Reply-network for Fortnite in Spring layout (left) and Kamada-Kawai layout (right).

The network visualization is highly informative in terms of highlighting the nodes that have a higher degree (the number of adjacent edges) in a network, which can be crucial for network analysis. In Figure 5, the number of replies for a specific user is characterized by the size of the node. Also, in the spring layout (plot on the left side), the denser a cluster of nodes is, the more users are connected to the node, and it is also expressed in the Kamada-Kawai layout (plot on the right side) by putting all the highly connected nodes to the center of the concentric circle. And Figure 6 is another example of the retweet-network for game CSGO.

Bokeh Plot
Figure 6. Retweet-network for CSGO in Spring layout (left) and Kamada-Kawai layout (right).

Alternatively, to reduce the number of irrelevant nodes and make the plot cleaner, plotting only the nodes that have a high degree-centrality value with their corresponding neighbors might be a better idea. And Figure 7 shows the sub-graph formed from the top-10 nodes (in terms of degree centrality) and their neighbors.

Bokeh Plot
Figure 7. Sub-retweet-network for CSGO in circular layout (left) and Kamada-Kawai layout (right).

Unlike the previous two figures, the plot on the left-hand side is plotted in a circular layout so we can spot out the nodes that have the highest degree centrality. Also, on the right hand side, the sub-communities and sub-reply-networks are better clustered, which makes it much easier to locate the high-impact Twitter user.

Conlusion

People often ignore or underestimate the power of visualizing data, whereas data interpretation is no less critical than data-processing or data-modeling by all means. Visualizing data in a more informative and reliable way is the key to find better solutions. In this post, I talked about how to create interactive applications to visualizing Twitter data on a geospatial map using Bokeh, as well as how to plot different types of networks using NetworkX. These demos shown here are just an illustration of the concept with minimal data, especially for the map application; thus, there is still a lot of space to be improved for each of them.

As always, you can find the code for generating all these demos here. Feel free to let me know if you have any questions.

Thanks for reading 🙂

,

Interactive Exploratory Data Analysis with Bokeh

Introduction

If one would recall, I talked about how to collect and store Twitter data in the last post (Figure 1). Now it is time to do some fun and exciting experiments with all the data available! First up is some fundamental exploratory data analysis that tells us the big story behind the data. Such as answering questions like, which game is the most popular one on Twitter? The game is more prevalent in which language? How often is the game being mentioned during a specific time window of the day? How does the community think about the game when people send tweets talking about it? Positively or negatively? Etc. All these questions will be answered by the end of this post. 

Figure 1. A relational database that stores information of more than 160,000 tweets.
Figure 1. A relational database that stores information of more than 160,000 tweets.

Game Ranking Analysis

The first question is, how the games are distributed on Twitter by counting the number of mentions for each game? Rigorously speaking, since one tweet could have multiple fields that contain text information, including text, extended_tweet, retweet_text, retweet_extended_text, quoted_tweet_text, and quoted_extended_tweet_text, to inspect the number of mentions for each game, one should check if the name of one specific game appears in any of these fields. To get a rough feeling about the total distribution, however, I approached through a shortcut by looking only at the hashtags field, which is already saved as an individual table in our database. The drawback is that not every tweet has a hashtag. Therefore, the number of tweets here is relatively smaller. A slightly smaller sample space should not matter too much, though, considering that almost every observation follows Normal Distribution in the real world.

Use the following query to extract game hashtags from the hashtags table in the database:

In the query above, first, create a view to avoid denormalizing any of our existing tables in the database and set a new column game in the view containing the video-game hashtags that are wanted. Also create new columns if_quoted and if_retweeted to indicate if the tweet is a retweet or quoted_tweet. Recall from the last post, when building the relational database, all tables are connected through multiple foreign keys in the base_tweets table. To extract the information, recombine tables by joining the hashtags table back to the base_tweets table, then left join the resulting table with retweeted_id table and quoted_id table. Then query all the columns and rows from the view to export and transport them into the Python environment to visualize the distribution. And the first ten records the query returns look like this: 

Figure 2. Extracted relevant video-game names from the hashtags.
Figure 2. Extracted relevant video-game names from the hashtags.

As shown in Figure 2, specific video-game names that we care about is extracted from all the hashtags, and the value is set to NULL if none of the names is mentioned in the hashtags, and true or false to check if it is from a retweet or a quoted tweet. Now, migrate this temporary table into a Pandas DataFrame to get some nice-looking visualization of the distribution as shown in Figure 3:

Bokeh Plot
Figure 3. Bar chart for all the games.

While there are some beautiful colors to differentiate between games, the amount of information conveyed in Figure 3 is limited. It is clear that Fortnite is the most popular game on Twitter since the majority of tweets are mentioning it. Surprise, GTA5 is the one that players mentioned for the least amount of times on Twitter, which is on the contrary to what is observed on Twitch. Well, if we think about it, it is not difficult to figure it out: GTA5 is a console game that has a limited size of the online community, but it is fun to watch on Twitch due to its game design.

Now we know that, on Twitter, the most popular game is Fortnite, and the least favorite game is GTA5. What else? Here is another more advanced bar chart in Figure 4:

Bokeh Plot
Figure 4. Another two bar charts for all the games.

The top chart in Figure 4. shows the number of quote tweets, retweets, and original tweets individually for each game, and the bottom chart is plotted in log scale for better comparison with smaller numbers. It turns out that, for most of the games, more than half of the total tweets are retweeted; even though Hearthstone has more mentions than GTA5, it has less original tweets than the latter; and Dota2 has the smallest original tweets percentage. 

Gaming Language Analysis

Besides the number of mentions for each game, similarly, we can query for the language information from the database, and analyze the language used in the gaming community globally. There are 42 languages used in the tweets streamed, including unknown languages marked as und, and some of them are plotted in the figure below:

Bokeh Plot
Bokeh Plot
Figure 5. Left: Pie chart for the top10 languages used. Right: Log-scaled bar chart for the top 20 languages.

In Figure 5, English dominates the language-distribution by almost half of the total amount, and the one following is Japanese with nearly 1/3 of the total number. After French and Spanish with comparable percentages, the rest of the languages are Undetermined and some other relatively non-common languages such as Portuguese and German, etc.  In the bar chart on the right-hand side, the numbers are scaled by taking the log value, and the detailed rank information for more languages are shown.

But how about the detailed language distribution inside of a specific gaming community? What if we want to know which language has been used the most often, say, in Fortnite? How much more or less is it compares to other gaming communities? Figure 6 answers these questions. Again, you can hover your mouse over the plot to check the value for each game-language combination.

Gaming Community Languages
Figure 6. Advanced pie plot mimicking Burtin’s Antibiotics plot.

By inspecting Figure 6, one can tell that even though English is the most often used language for most of the games, for Apex Legends, Japanese is the most frequently used language over English around 300 tweets. (If you hover the mouse over, you will see the number 7291 for Japanese and 6885 for English. ) Similarly, Hearthstone seems more prevalent in Spanish-speaking regions than French and Japanese since it is the second often used language for Hearthstone. 

Time series Analysis

One of the critical features of streaming tweets in real-time is time-series analysis. We can count all the mentions for each game during a time window, let’s say 1 hour, then shift this window throughout the 48 hours streaming duration, to visualize the time-varying mentioning frequencies for different games. I picked the top-4 most popular games on Twitter (Figure 3) and plotted their hourly mentioning frequency in Figure 7:

Bokeh Plot
Figure 7. Mentioning frequency averaged hourly for 48 hours.

The stream started at 1 am, May 9th, and ended at 1 am May 11th, as shown in Figure 7. We know that Fortnite is the most popular game (from Figure 4), so it is not surprising to see Fortnite has the highest frequency at all time. However, what is not expected is the peak for Fortnite around 8 am May 9th, while the frequencies for all three other games are going down. Based on what we have learned so far, it is not difficult to figure it out: from Figure 6 we know that the second most frequently used language for Fortnite is Japanese, which means there is another massive Fortnite online community on the other side of the earth existing, in Japan. While it was 8 am for the US; the night-time after working had just begun for Japanese Fortnite lovers. (Further investigation shows that there was an official Twitter event going on for Japanese Fortnite community.) Therefore there is a prominent peak shows up around that time stamp. It also makes a certain amount of sense why there is only one downhill at 8 am May 9th, because May 9th was a Friday, people still need to work on that day. And after that, the weekend was there so players can play video games all night! 

Another interesting experiment to do with tweets that have DateTime information is sentiment analysis, which can be crucial for commercialized marketing applications. I use the Natural Language Toolkit(NLTK) to extract the sentiment information from a tweet. After applying sentiment analysis to all the English tweets, the hourly averaged time-series sentiment score plot is shown in Figure 8:

Bokeh Plot
Figure 8. Sentiment scores averaged hourly for 48 hours.

Given the context that sentiment score above 0 means positive and below 0 means negative, interestingly, while Fortnite got mentioned in most of the tweets, it also got relatively more criticism around Friday night (Japanese time). It seems like that official event was not going on quite well. In contrast, Apex Legends was mentioned in more positive tweets later on around 9 am May 10th. And Overwatch and League of Legends were doing a good job maintaining a slightly positive sentiment score consistently. 

Conlusion

Twitter is an excellent resource not only for sharing our daily lives but also for data gathering and social-event analysis. In this post, I talked about how to use the pre-stored Twitter data to do some fundamental Exploratory Data Analysis. Specifically:

  1. Explicitly visualized ranking for eight hand-choosing video games. Fortnite is the most popular video game on Twitter. There are 90570 out of 165910 (54.59%) tweets mentioned Fortnite.
  2. language-distribution for these video games was extensively inspected. English (49.17%) and Japanese  (27.60%) dominate the Twitter community for these games.
  3. Mentioning frequency and sentiment scores are plotted in the time domain with an hourly-average window. Even though Fortnite has been mentioned more often, people were happier with Apex Legends and two other games with higher averaged sentiment scores.

Now we have a better understanding of the entire global video-game community, but only a tip of the iceberg is revealed. In the next post, I will explain how to utilize the geographical information contained in the tweets to build an interactive map application to reinforce our data analysis along with some interesting network analysis.

You can find the code here for plotting all the figures shown in this post. And welcome to reach out to me if you have any questions.

Thanks for reading 🙂