shihao

About shihao

Posts by :

What I Have Learned From a Senior Data Scientist

Intro

I joined the AI team at Dataminr last fall (2019) for an internship before I graduate from my Ph.D. program. I was lucky to be able to work with an extraordinary group of people at Dataminr and learned a lot in the AI team. Here I will briefly list the useful tips and experiences I have learned from my mentor –  a senior data scientist in the group during this internship. The order does not denote any priority.

Project Documentation

Documentation is a relatively important part of the project itself. Usually, this is done by cascading a Google slides file, including the initial ideas of the project, the methods we have looked up, then tie these methods back to the literature in another literature review file. Not only the working part of the project but also the sample points where the methodology does not work. Documentation, in a sense, is the tracking log of the project. In a commercial setting, every decision-making process needs to have its corresponding logical reason behind the scene, and the documents are the place to store all the foundations of these ideas and decisions.

More importantly, when communicating the ideas or different aspects of the project to other co-workers or stakeholders in the company, a well-documented file comes handy delivering convincing reasons while illustrating project progress.

For example, the project I was working on has its progress document, literature review, idea board, as well as weekly report deck. We are not only putting good stuff into the document, but also those ugly results from different methods. Later on, when we summarize and explain the project to our manager, we could refer back to the documentation to demonstrate the specific reasons or advantages of us picking one technique or pipeline over another. 

Daily Schedules

Here I mean something different from your Google Calendar but the “real” schedule which made by no one else but you and you want to stick yourself to. 

After the first week of the intern, my mentor sent me another Google doc with the following pattern:

schedule

The screenshot is just a replicate, but the idea is similar: there is a todo list for the whole week, which should be decided beforehand either in the previous meeting or at the beginning of the week — followed by some existing issues that need to be solved within this week. The items under each day are listed by me each morning around 8:30, or on the previous day before I leave work.

This is similar to the project documentation but working on ourselves. In a self-driven environment, I found these predefined schedules extremely helpful in terms of keep track of issues and deliver progress rigorously and on time. 

Effective Communication

It is widely known that communication and presentation skills are essential and crucial for a data scientist, only after working along with good data scientists in the industry I got the chance to know that it is always easier said than done. Here, “effective” is more towards the higher-level colleagues, such as supervisor or manager. This group of people is always under busy schedules, and how to utilize their limited time window to get your ideas and opinions out clear and understood is quite a challenge sometimes, at least to me.

At the end of my intern, I asked my mentor, what was the essential quality working as a data scientist in a business setting, and his answer was “effective communication” towards skate holders and management of the organization. The previous tips, including project documentation and persuasive presentations, all contribute to this point.

One personal advice, when communicating with your supervisor or manager, try not to overthink their words. At the beginning of the internship, I was trying to figure out what does “meet you again later this week” mean from my manager. I would be worried that he was not happy about my proposal, or he wanted me to come up with something different next time we meet. And my mentor told me, “do not spend your time and energy on stuff like this, just keep working on your own pace, and he was just tempted to talk to you again.”

Of course, I do not mean that being considerate is not good, but there should be a clear boundary between considerate and too sensitive. Communication is a critical soft skill, and I still need to keep working on it. 

Meeting Notes

Some people always bring their Mac Books into a meeting room and would never stop typing during the meeting, while some others would walk into the room with their bare hands and do nothing except listening. Is there a certain appropriate behavior during a meeting? I guess the answer is no, hence here I give out the way I found useful and favorite to me from my mentor.

Unless I were to present during the meeting, I would NOT bring either my laptop or my cell phone with me. The reason is that if someone is presenting to me, I should be focusing on the projection instead of my laptop or cell phone. The only things I will take with me is a notebook and a pen. Taking notes during the meeting is always a good habit, including the points you are confused about, or a good point that can light you up, or write down the core ideas of the talk. Knowing how to summarize and extract essential information from a lecture or discussion is crucial to a data scientist.

I noticed that I need to improve the way of joining a meeting after the first reporting meeting of my internship. We presented the initial version of the project to the whole group and asked for feedback. After the meeting, my mentor asked me, “how do you feel about the meeting, did you get something useful from the audience?”, and I replied, “almost nothing.” And my mentor said, “well, that is not true, we got at least three useful feedback.” And those points are those I heard during the meeting but did not realize that I should write them down, and by the time we had the discussion, I almost forgot about them. My mentor was the savior since he wrote them down clearly. 

Remember the daily schedule file where each day has its own section? These summarized meeting notes would go there.

Command Documentation

As a data scientist, mastering multiple languages is a must, including programming languages such as Python, R, C/C++, Java or JavaScript, scripting languages such as Bash Script, IPython Shell, even web editing languages such as HTML and CSS. And each language has its tricks and commands that are useful and worth noting. Someone can rarely memorize everything by a glimpse and never forget. Therefore it is essential and practical to have a document that keeps all the useful and specific commands or tricks we have used in the past.

The advantage of keeping such a document showed up obviously when I started to work with Docker. For different services or images, the commands for Docker to run the container can be quite different. For instance, some containers might need to be mounted with some volume on the host machine while some others might need to have some different ports. Unless I am working with Docker daily, I would perhaps forget how to deploy a specific service after a month or so. But if I have a document that I can easily refer back to, I can grab the necessary information immediately. Of course, while some of this information about Docker is usually documented on Git, some might not.

For another example, I wrote down the command of how to use ffmpeg to convert an mp4 file into a gif file with the best quality in the terminal into my most recent command document, so I can directly look into it next time if I need to convert something else. 

Yet Another Communication

During one of the in-person interviews with the research scientist in Dataminr, I asked the interviewer the same question that I asked my mentor: what is the essential skill for a scientist in Dataminr, or the industry in general. And he gave me the same answer: communication. However, the explanation he gave about the same word has been different from the last time I heard about it.

The communication here is more about a working attitude: positively involving in the team conversations and get to know the whole team with a leadership mentality.

Specifically, being “involved” in the team means to express yourself more and let the team heard your voice, no matter if it is closely related to the current work or if it is totally off the topic, the idea of “thinking out of the box” is essential and need not be suppressed.

On the other hand, “leadership mentality” does not mean that we should be too ambitious of being the manager of the group from the beginning. Instead, we should take part in every aspect of the team as if we were the leaders. Get to know what every group member is working on is vital in terms of collaboration.

Chatting with colleagues about their work seems not a high-efficiency event, but it is an essential part of the team-player mentality. 

The Ability to Learn

Two weeks ago, I had an interview with another senior data scientist in the industry. By the end of the interview, I asked the same question, which resulted in a different answer: he thinks there are so many skills that are essential and important to a data scientist, and if he had to pick one, he would rank the ability to learn as the top.

He somehow illustrated the history of machine learning and deep learning in the recent few years. He pointed out that we data scientists are working right in the golden era of this industry where vast improvements in technologies are updated every couple of weeks. If we are working as a research type of data scientist, we need to keep up with the pace of technology advancement. 

The space for understanding and better utilizing artificial intelligence is still huge. What we have seen and learned is just the tip of the iceberg. With higher and higher scores been broken through by numerous SOTA models, being able to adapt to the new and better algorithms and take advantage of better performance is crucial.

Live Twitter Data Analysis and Visualization using Python and Plotly Dash

Introduction

Twitter is a platform that embraces tons of information flow in every single second, which should be fully utilized if one wants to explore the real-time interaction between communities and real-life events. I think it would be cool to have an application that is capable of collecting, storing, analyzing, and finally, visualizing Twitter data in real-time so that we could know what is precisely happening by the time people run the application.

Having this exciting idea in mind, I implemented this app and deployed it to the cloud. The app itself can be accessed through this link (Figure 1). The app collects all the tweets related to the gaming community posted and visualizes some statistics. The first panel on the left-hand side is a line plot of the word-count trend, showing the changing pattern of the top-5 most frequently mentioned words. The top panel on the right-hand side is a bar chart showing the top-10 words for better comparison. The figure below is a time-shifting scatter plot of the averaged real-time sentiment score for all the tweets grouped by the top-5 mentioned words.

With the predefined tracking keywords, the user can get the corresponding real-time reaction from the Twitter community (for the gaming community in this example) about a specific topic. It can also be integrated into a recommendation system that requires short reacting time, or a Twitter surveillance system during an event, etc. In this post, I will talk about the framework and tools that I used to build this app.

Figure 1. A demo showing the interface of the live Twitter streaming and visualization app.
Figure 1. A demo showing the interface of the live Twitter streaming and visualization app.

Application Framework

I introduced how to stream tweets with Python using Tweepy package in one of the old posts. But previously, the process of data collection, data migration, storage, and processing are handled in individual steps, or even totally different timelines. For example, if the goal is to collect all the tweets that tweeted in the next two days, we will have to wait for 48 hours for the streaming to finish, then we can start to process the data. No matter how fast we finish the downstream work, there is still at least a two-day gap between the time when we collected the first record and the time we gain insight from the data.

For this live streaming app, however, the traditional workflow is divided into two independent pipelines working together (Figure 2). In detail, data processing starts right after the first line of record is received, followed by data analyzing and results-visualization, etc. While the visualization server (Plotly Dash) is handling data-processing, the streaming server (Tweepy), on the other hand, will bring in the next line of newly generated data. Each pipeline keeps looping at its own pace. Figure 2 shows the underlying framework of all the processes described: 

Figure 2. The underlying framework for the live app.
Figure 2. The underlying framework for the live app.

As shown in Figure 2, to make the application react promptly, we break the traditional data science pipeline into two modules that are taken care of by their corresponding local servers. Specifically, Tweepy is responsible for streaming data, i.e., control the flow of tweets, and we add extra functionalities to the trigger behavior, such as pushing data into a database or deleting old records from the database (optional). Note that the extra functionalities are triggered whenever there is a tweet falls under the predefined tracking conditions thus heard by the server. Therefore, for the Tweepy server,  the intervals between the last operation and the next are random, as shown in Figure 2.

Meanwhile, data processing and visualization are carried out by another platform, Dash by Plotly. Dash controls the rendering and refreshing of the visualization on the browser; hence, a constant time interval for this trigger is preferred. Depends on the data throughput, the time window can be ranging from 0.5 seconds to 5 seconds (2 seconds for this app). In a big picture, two running servers are executing their loops independently, and data is flowing in between them in the predefined time interval to update the application’s graphical interface. 

Data Ingestion

For the data collecting module, we use Tweepy to set up a local server to hear tweets from the Internet. First, create a file with name streaming.py; which sets up the authentication and trigger conditions, the code for streaming.py; is shown below:

# import packages
from tweepy import OAuthHandler
from tweepy import API
from tweepy import Stream
from sqlalchemy import create_engine
from sqlalchemy_utils import database_exists, create_database
from urllib3.exceptions import ProtocolError
from slistener import SListener
from key_secret import consumer_key, consumer_secret
from key_secret import access_token, access_token_secret

In the script above, we first import all the necessary packages, including Tweepy and Sqlalchemy. We also import ProtocalError from urllib3.exceptions so the app can automatically resume the streaming even if a connection error occurred later on. The following slistener and key_secrete object are just two other Python scripts in the same folder. Specifically, the key_secrete.py contains all the credential information such as keys and tokens. And slistener.py contains a child SListener class object inherited from Tweepy’s standard StreamListener class (more on this in a bit).

# consumer key authentication
auth = OAuthHandler(consumer_key, consumer_secret)
# access key authentication
auth.set_access_token(access_token, access_token_secret)
# set up the API with the authentication handler
api = API(auth)
# instantiate the SListener object
listen = SListener(api)
# instantiate the stream object
stream = Stream(auth, listen)
# set up words to hear
keywords_to_hear = ['#Fortnite',
                    '#LeagueOfLegends',
                    '#ApexLegends']

Following importing packages, the next five lines of code are the standard routine to set up a streaming API using the authentication and predefined SListener object. keywords_to_hear is a list of keywords we want to catch, in other words, only if the tweets have one or more of the keywords in the list, would our streaming server capture it.

# create a engine to the database
engine = create_engine("sqlite:///tweets.sqlite")
# if the database does not exist
if not database_exists(engine.url):
    # create a new database
    create_database(engine.url)
# begin collecting data
while True:
    # maintian connection unless interrupted
    try:
        stream.filter(track=keywords_to_hear)
    # reconnect automantically if error arise
    # due to unstable network connection
    except (ProtocolError, AttributeError):
        continue

Next, we use the create_engine function from Sqlalchemy to create a database to store our data if the database does not exist yet. We create the database here since we want a database ready to receive data before the stream is launched. Lastly, stream.filter(track=keywords_to_hear) launches the server and start to stream tweets that contain the keywords we specified in the list. We put this line of code into a while-loop since we do not want the streaming to be terminated if the network connection was randomly interrupted. The streaming will only be terminated if we stop the server manually.  

Once the streaming is launched, tweets will be flowing into our localhost machine. The way how the raw data should be handled is further specified by the SListener object. And the code below is a sample script for how to define the SListener object. Create a new file called slistener.py and put the following blocks of code into it. Begin with importing:

# import packages
from tweepy.streaming import StreamListener
import json
import pandas as pd
from sqlalchemy import create_engine

First of all, we import the base StreamListener class object from tweepy so we can add some extra functionalities to it later on. Next, to process the raw Twitter statuses object, JSON package is needed to parse the statuses object into a JSON object to select the useful fields. Pandas is also necessary since it provides a convenient bridge connecting our Python working environment to the database.

# inherit from StreamListener class
class SListener(StreamListener):
    # initialize the API and a counter for the number of tweets collected
    def __init__(self, api = None, fprefix = 'streamer'):
        self.api = api or API()
        # instantiate a counter
        self.cnt = 0
        # create a engine to the database
        self.engine = create_engine('sqlite:///tweets.sqlite')
    # for each tweet streamed
    def on_status(self, status): 
        # increment the counter
        self.cnt += 1
        # parse the status object into JSON
        status_json = json.dumps(status._json)
        # convert the JSON string into dictionary
        status_data = json.loads(status_json)

The rest of the script is for customizing our StreamListener class. First, in addition to the parent class provided by tweepy, we also initialize a database engine in the __init__ method, and the cnt variable is simply an indicator for the number of tweets streamed so far. The on_status method is the most essential part of the whole class. Whenever a new tweet is heard, this on_status method would be triggered. The first 3 lines of code parse the incoming raw data into a python-accessible data structure, JSON dictionary. Basically, the raw object is converted into a dictionary containing all the information of that specific tweet, such as the timestamp when it was created, the user, its text, etc. 

        # initialize a list of potential full-text
        full_text_list = [status_data['text']]
        # add full-text field from all sources into the list
        if 'extended_tweet' in status_data:
            full_text_list.append(status_data['extended_tweet']['full_text'])
        if 'retweeted_status' in status_data and 'extended_tweet' in status_data['retweeted_status']:
            full_text_list.append(status_data['retweeted_status']['extended_tweet']['full_text'])
        if 'quoted_status' in status_data and 'extended_tweet' in status_data['quoted_status']:
            full_text_list.append(status_data['quoted_status']['extended_tweet']['full_text'])
        # only retain the longest candidate
        full_text = max(full_text_list, key=len)

The rest of the script is for customizing our StreamListener class. First, in addition to the parent class provided by tweepy, we also initialize a database engine in the __init__ method, and the cnt variable is simply an indicator for the number of tweets streamed so far. The on_status method is the most essential part of the whole class. Whenever a new tweet is heard, this on_status method would be triggered. The first 3 lines of code parse the incoming raw data into a python-accessible data structure, JSON dictionary. Basically, the raw object is converted into a dictionary containing all the information of that specific tweet, such as the timestamp when it was created, the user, its text, etc. 

        # extract time and user info
        tweet = {
            'created_at': status_data['created_at'],
            'text':  full_text,
        }
        # uncomment the following to display tweets in the console
        print("Writing tweet # {} to the database".format(self.cnt))
        print("Tweet Created at: {}".format(tweet['created_at']))
        print("Tweets Content:{}".format(tweet['text']))
        #print("User Profile: {}".format(tweet['user']))
        print()
        # convert into dataframe
        df = pd.DataFrame(tweet, index=[0])
        # convert string of time into date time obejct
        df['created_at'] = pd.to_datetime(df.created_at)
        # push tweet into database
        df.to_sql('tweet', con=self.engine, if_exists='append')

The rest of the two fields (the timestamp and the user’s profile) can be accessed more straightforwardly as we generate the dictionary tweet. We use print functions to help verify if the streaming is working fine. Then, we convert the dictionary into a pandas DataFrame. Since there is just one observation (row) to be converted, we tell pandas that we only have one index for the row by passing in a list with 0 in it as a parameter. Then we turn the timestamp from a string object into a datetime object for time-series-analysis, and then push the data into a database using the engine we initialized in the __init__ method. 

        with self.engine.connect() as con:
            con.execute("""
                        DELETE FROM tweet
                        WHERE created_at IN(
                            SELECT created_at
                                FROM(
                                    SELECT created_at, strftime('%s','now') - strftime('%s',created_at) AS time_passed
                                    FROM tweet
                                    WHERE time_passed >= 60))""")

For our case, since the application is hosted on the cloud 24/7, I certainly do not want all the tweets to stay in the remote machine forever since there is a storage limit on the host machine. So I choose to connect to the database again and remove the records that are older than 60 seconds from my database using SQL query. If you are running this application on your local machine or you do not care about the storage limit, you can remove the query. 

Now, to launch the streaming, simply run python streaming.py in the console, and you should be able to get something like Figure 3 (make sure you have SQLite 3 installed):

Figure 3. Tweets flow in once the stream is launched.
Figure 3. Tweets flow in once the stream is launched.

Data Visualization

The framework we use to visualize the data is Dash by Plotly, which is a Python framework written on top of Flask, Plotly.js, and React.js, for building analytical web applications. It allows us to implement complicated web functionalities with much fewer code compares to Flask or Django. The two most essential libraries for creating an interactive visualization app with Dash is the dash_core_components and dash_html_components, they come in automatically once Dash is installed in the current working environment. dash_core_components provides components that can be added to the interactive webpage, such as a graph, a slider, or a dropdown menu. On the other hand, the general layout of the webpage is specified using dash_html_components, which contains functions for generating all HTML tags so that we can create the web content in pure Python environment with a similar code structure to an HTML file.

To kick things off, create a new file app.py. The code inside app.py can usually be divided into two parts, the first part specifies the layout, and the second part specifies the interactive behavior (i.e., callbacks). Start with importing:

# import basic packages
import os, datetime, re, nltk
import numpy as np
# import data ingestion packages
from collections import deque, Counter
from data_gathering.api import get_tweet_data
# import Dash packages
import dash
import dash_core_components as dcc
import dash_html_components as html
from dash.dependencies import Output, Input, State
# import Plotly Package
import plotly.graph_objs as go
# import NLTK packages
from nltk import word_tokenize
from nltk.corpus import stopwords
from nltk.sentiment.vader import SentimentIntensityAnalyzer

For this app specifically, we need to import some other packages to handle the extra functionalities, such as datetime for accessing current time, nltk for word counting and sentiment analysis, etc. We also need to import plotly.graph_objs, which contains all the graph objects, for instance, scatter plot, bar chart, and pie chart.

# download nltk dependencies
nltk.download('punkt')
nltk.download('stopwords')
nltk.download('vader_lexicon')
# stop words for the word-counts
stops = stopwords.words('english')
stops.append('https')
# global refresh interval for the application, ms
GRAPH_INTERVAL = os.environ.get("GRAPH_INTERVAL", 2000)
# initialize a sentiment analyzer
sid = SentimentIntensityAnalyzer()
# keywords for the tweets
keywords_to_hear = ['#Fortnite',
                    '#LeagueOfLegends',
                    '#ApexLegends']
# X_universal is the x-axis with time stamps
X_universal = deque(maxlen=30)
# initialize dictionaries to store Y axis values for plots
scatter_dict = {}
sentiment_dict = {}

Next up are some pre-configurations. First, we download some NLTK dependencies, which are essential components for tokenization. For example, punkt is a package that contains pre-trained tokenizers that detects sentence boundaries. stopwords is a package with lists of stopwords in different languages. Here we specify that we only need the stopwords in English, and we also append https to the stopwords since it is a high-frequency word that does not contain much information. Then we get the application fresh-rate GRAPH_INTERVAL from the environment variables and set it to 2000 ms (2 seconds) if it is not found. Next, the x- and y-axis of the scatter plot is initialized by allocating deque objects with a maximal length of 30 since we only want to show 30 timestamps on the plot.

# initialize the app and server
app = dash.Dash(__name__, meta_tags=[{"name": "viewport", "content": "width=device-width, initial-scale=1"}])
server = app.server
# add layout to the app
app.layout = html.Div(
    [
        # header
        html.Div(...
        ),
        # graph layout
        html.Div(
            [
                # left hand side, word count line plot
                html.Div(...
                ),
                # right hand side, bar plot and scatter chart
                html.Div(...
                ),
            ],
            className="app__content",
        ),
    ],
    className="app__container",
)

The code above shows how to initialize the application and its server with some default meta settings, followed by the layout object. Note that here I replace the detail layout setting with “…” to give you an overview of the webpage structure. The layout is straight-forward to specify and does not require a lot of coding skills. If you are curious about the details, you can refer to this page to see the raw code for the layout of this application. 

Next, we add some helper functions and callback functions to the application:

def hashtag_counter(series):...
def bag_of_words(series):...
def preprocess_nltk(row):...
# define callback function for number_of_tweets scatter plot
@app.callback(
    Output('number_of_words', 'figure'),
    [Input('query_update', 'n_intervals')])
def update_graph_scatter(n):...
# define callback function for word-counts
@app.callback(
    Output('word_counts', 'figure'),
    [Input('query_update', 'n_intervals')])
def update_graph_bar(interval, slider_value):...
# define callback function for sentiment_score
@app.callback(
    Output('sentiment_score', 'figure'),
    [Input('query_update', 'n_intervals')])
def update_graph_sentiment(interval):...
# define callback function indicating number of tweets
@app.callback(
    Output("num_tweets", "children"),
    [Input("query_update", "n_intervals")])
def show_num_tweets(n_intervals):...
# run the app
if __name__ == '__main__':
    app.run_server(debug=True, host='0.0.0.0')

Again, I hide the details of the functions to give you a general idea of the structure and make it less overwhelming to follow through. Helper functions can always differ for different applications, refer to the link above if you want to see the details. Basically, hashtag_counter takes a pandas series and return the number of rows fall into each keyword category and return a dictionary with all the counts. bag_of_words and preprocess_nltk are text-cleaning tools.

For the callback functions, we use Python decorators to add extra functionalities to the original app.callback function, including different input/output, and different refresh behaviors. Here I use the callback function for the line plot as an example: 

@app.callback(
    Output('number_of_tweets', 'figure'),
    [Input('query_update', 'n_intervals')])
def update_graph_scatter(n):
    # query tweets from the database
    df = get_tweet_data()
    # get the word count
    cnt = bag_of_words(df['text'])
    # get the current time for x-axis
    time = datetime.datetime.now().strftime('%D, %H:%M:%S')
    X_universal.append(time)
    # initialize a list of items to pop
    to_pop = []
    # loop through the dictionary to filter out outdated keywords
    for keyword, cnt_queue in scatter_dict.items():
        # if the count queue for current keyword is not empty
        if cnt_queue:
            # pop all the outdated count values
            while cnt_queue and (cnt_queue[0][1] < X_universal[0]):
                cnt_queue.popleft()
        # if the queue for the keyword is empty
        else:
            # append it to the pop list
            to_pop.append(keyword)
    # pop all the outdated keywords
    for keyword in to_pop:
        scatter_dict.pop(keyword)
    # update the top_N keywords
    top_N = cnt.most_common(num_tags_scatter)
    # loop through the new top_N to update the dictionary
    for keyword, cnt in top_N:
        # if the current keyword is newly appeared
        if keyword not in scatter_dict:
            # initialize a new deque for it and append its count and time
            scatter_dict[keyword] = deque(maxlen=30)
            scatter_dict[keyword].append([cnt, time])
        # if it is not a new one, just append the count and time
        else:
            scatter_dict[keyword].append([cnt, time])
    # update the colors for the new dictionary
    new_colors = chart_colors[:len(scatter_dict)]
    # plot the scatter plot
    data=[go.Scatter(
        x=[time for cnt, time in cnt_queue],
        y=[cnt for cnt, time in cnt_queue],
        name=keyword,
        mode='lines+markers',
        opacity=0.7,
        marker=dict(color=color, size=15)
    ) for color, (keyword, cnt_queue) in list(zip(new_colors, scatter_dict.items()))]
    # specify the layout
    layout = go.Layout(
            xaxis={
                'automargin': False,
                'range': [min(X_universal), max(X_universal)],
                'title': 'Current Time',
                'nticks': 8},
            yaxis={
                'type': 'log',
                'autorange': True,
                'title': 'Number of Tweets'},
            height=700,
            plot_bgcolor=app_color["graph_bg"],
            paper_bgcolor=app_color["graph_bg"],
            font={"color": app_color["graph_font"]},
            autosize=False,
            legend={
                'orientation': 'h',
                'xanchor': 'center',
                'yanchor': 'top',
                'x': 0.5,
                'y': 1.025},
            margin=go.layout.Margin(
                l=75, r=25, b=45, t=25, pad=4))
    return go.Figure(
        data=data,
        layout=layout)

While the majority of the code above explains itself by those comments, there are a few extra notes:

  1. The input/output of the callback process is denoted in the decorator, specifically, the Output object tells Dash that this callback is assigned to a figure called number_of_words, and the Input object tells Dash that the callback should be triggered by a module called query_udpate.
  2. get_tweet_data() is another helper function that sends a SQL query to the database to fetch all the records from the table we pushed all the tweets to (previously mentioned in SListener.on_status()). Since the database is deleting old records while receiving new records, it is okay for me to query all the current records. If that is not what you want, you can specify a time-window to query instead.
  3. X and Y coordinates of the scattering plot are updated by passing in the newly calculated x and y values into the go.Scatter() function which creates a scatter plot. The returned object is then embedded into a list called data, which specifies the returning data of this callback function. Similarly, the layout is set by the go.Layout() function. 
  4. Last but not least, the returned object of the callback function is a go.Figure object that contains the newly updated data and layout. 

The other callback functions can be written in a similar format. 

Conlusion

The last line of code in the app.py sets the debug mode to True so that, after the server is launched, changes will be automatically applied without restarting the server. If the application is ready to deploy, you might want to set debug mode to False. Set the host IP address to 0.0.0.0 to allow access from outside the localhost.

Now if you open a terminal and execute python app.py, the server should be up and running on your local workstation at 0.0.0.0:8050 (default port for Dash is 8050).

For dockerizing and deploying the software to the cloud, please refer to my previous post about Docker Containers and Amazon Web Services. You can also access all the code for this application here (including Docker files). And the application is available here.

Please feel free to reach out to me if you have any questions.

Thanks for reading 🙂

Deploying Data Visualization Application to the Cloud: Docker Compose and Amazon Web Service

Introduction

Back then, during 2017-2018, when data scientists are defined as mere practitioners that have limited functionalities that only involves the workflow related to data, such as data ingestion, migration, storage, manipulation, analysis, and modeling. From my point of view, however, with the fast pace of technology-growing and advancement, a good data scientist should evolve accordingly as well. While asking a data scientist to scale their machine learning models to the cloud is generally not practical several years ago, I think it is necessary for a modern data scientist to at least have some basic knowledge of the infrastructure behind the deployment process.

To gear myself up with the essential skills about cloud computing and software deployment, I integrated one of my data visualization applications with Flask, which is a web development framework, and dockerized both of them. Combined with AWS (Amazon Web Services), the deployed application is now on the cloud running 24/7. In this post, I will demonstrate the whole deployment process and break down the steps one by one, including building docker containers, scale services using docker compose, and deploy them on AWS EC2 instances. The application is live here: http://18.219.26.131/.

Web Development

In my previous post, I introduced how to make interactive data visualizations via Bokeh server. The extra options provided in the plot such as shading and radio check group, are supported by a Bokeh server running behind the scene. To display the content returned by the Bokeh server on a webpage, we need to build a framework to hold the information and present it through a web browser. There are various platforms out there that provide the functionality, including Flask, Django, etc. Compares to Django, Flask is more lightweight and has more freedom in terms of keeping the core of the web application extensible and straightforward. On the other hand, Django is a full-stack Python web framework which takes care of many built-in modules such as admin interface and ORM database support. Since what I need is a simple layout to mount my Bokeh visualization, I choose Flask to build the webpage. 

Essentially, Flask runs a server locally that hears all the inbound requests and sends the webpage content to the client backend. The following code sets up the Flask server and pulls the plot from our Bokeh server automatically once a request is heard: 

# import flask
from flask import Flask, render_template
from bokeh.embed import server_document
import sys, os
# instantiate the app
app = Flask(__name__)
# render the template
@app.route('/')
def index():
    # pull server session from the bokeh server
    vggm = server_document(url="http://localhost:5006/map")
    # return the rendered webpage
    return render_template("index.html", vggm=vggm)
# run the server
if __name__ == "__main__":
    app.run(host="0.0.0.0")

 

In the code, the returned content from the Flask server is an HTML file that can be displayed by the browser which is rendered by index.html. How to write HTML and CSS is beyond the scope of this post, but basically index.html specifies the layout of the webpage, and the corresponding CSS file specifies the style of the content.

Docker Containers

Docker helps to overcome individual differences between various platforms and operating systems when deploying software applications. A Docker image packs up the application core and all its dependencies into one package that can run anywhere on any computing environment as long as there is a docker engine installed. The environment for the application is internally installed and set up automatically by the docker image, such a micro-environment running based on the underlying operating system is called a docker container. Figure 1 illustrates the structure of Docker-containerized applications.

Figure 1. Docker bridges the gap between the operating system and applications.
Figure 1. Docker bridges the gap between the operating system and applications.

As we can see in Figure 1, there can be multiple applications that run in their Docker containers with different configurations, which can be a life savior in some cases where only one of the apps need to be updated or modified. In this case, two docker containers will be running within one operating system, one for Flask and one for Bokeh, and new applications can be added into the group later on as additional services. 

To dockerize a particular application, once the docker engine is installed on the local computer, one last component needs to be presented, which is called Dockerfile. A Dockerfile is a list of instructions that docker follows to build a Docker image. For example, the Dockerfile for Flask application is shown below: 

# from base image
FROM continuumio/miniconda3
# maintainer
LABEL maintainer="shihao1007@gmail.com"
# install packages
RUN conda install -y bokeh flask
# set the working directory to /app
WORKDIR /app
# copy the current directory contents into the working directory
COPY . /app
# make port 5006 avaiable to the world outside the container
EXPOSE 5006
# run bokeh serve whem container launches
CMD python /app/flaskapp/app.py

 

The FROM instruction initializes the building processing of the container with a base image for following instructions. Here we choose a base image Miniconda3. The RUN instruction installs all the packages needed for this container using conda install. The rest of the instructions are explained by the comments.

The Dockerfile for Bokeh application can be constructed similarly, just add pandas and nltk to the packages that need to be installed and change the exposed port number. Then the docker image can be built by running docker build.  To run a docker container from a docker image, use docker run.

Docker Compose

For a multi-functional service that is composed of multiple Docker containers, while one can run docker run multiple times, it can be tedious and hard to scale if there are too many containers to run at the same time. Fortunately, we have Docker Compose, which is a tool for defining and running multiple docker container applications. Referring to the official documentation, using Docker Compose includes three steps:

  1. Define individual Dockerfiles for each application.
  2. Define services in docker-compose.yaml to associate all docker containers.
  3. Run docker-compose up.

Step 1 is done previously, the next step is to build a docker-compose.yaml file using the following code: 

version: '2'
services:
  bokehapp:
    build:
      context: .
      dockerfile: bokehapp.Dockerfile
    ports:
      - '5006:5006'
    volumes:
      - './:/app'
  flaskapp:
    depends_on:
      - bokehapp
    build:
      context: .
      dockerfile: flaskapp.Dockerfile
    volumes:
      - './:/app'
    ports:
      - '80:5000'

 

As shown in the code:

  • version: '2' specifies that the file format version of this .yaml file is version 2. Different format versions have different syntax and upgrades compare to the older version.
  • services specifies all the containers to be launched.
  • Within each service (container), build specifies the docker image for that specific container to build from, context . dockerfile tells Docker Compose the container should be built from a Dockerfile.
  • ports binds the local port to the container port, in this case, for the Bokeh container, the local port 5006 is mapped to port 5006 on the container. Similarly, local port 80 is exposed to Flask container’s port 5000. Once the containers are running, we can access the servers running in each container through the local ports (e.g., 5006 for Bokeh server and 80 for Flask server).
  • volumes specifies the shared volumes for all containers and map the current working directory on localhost to the /app directory.

The .yaml works with the file structure illustrated in Figure 2, the current working directory will be vggm-webapp folder. 

Figure 2. File Structure for the application.
Figure 2. File Structure for the application.

As shown in Figure 2, by running docker-compose up command in the terminal, docker creates two containers, one for Bokeh using map.py as source file, and another one for Flask using app.py. Two containers are then associated with each other by the settings specified in docker-compose.yaml. Now the webpage with the Bokeh plot mounted by Flask is available at localhost:80.

To the cloud

There are numerous ways to deploy the application to the cloud, including Microsoft Azure, Google Cloud, and Amazon Web Services. Azure provides excellent services when deploying applications on Windows operating systems, or related to Microsoft Office, SQL server, etc. Google Cloud is a better fit for applications related to Big Data and Advanced Machine Learning. AWS provides a wide variety of Cloud Computing services and lots of free-tier eligible services. Hence I choose AWS for my small visualization application.

Specifically, I deploy the application using Amazon Elastic Compute Cloud (EC2) following these steps:

  1. Launch an EC2 t2.micro instance with Ubuntu Server 16.04 LTS (HVM), SSD Volume Type AMI (Amazon Machine Image).
  2. Configure inbound rules in the security group so that port 80 is enabled for hearing requests.
  3. Connect to the instance using ssh.
  4. Install Docker and git on the instance.
  5. Pull the repository.
  6. Run dock-compose up -d.
  7. Close the terminal to disconnect from the instance.

Now the services are up here in detach mode (by specifying -d flag in the last step). That simple.

Conlusion

Docker containers are handy when deploying software applications to different platforms. Combined with advanced cloud-computing infrastructure such as AWS, it is much easier to deploy and scale up the app. I learned a lot during deploying my little data visualization application to the cloud, including how to dockerize applications as well as how to launch AWS EC2 instances.

I hope this post can help you, as well.

Lastly, the code for this project can be found in this repository. 

Thank you for reading 🙂

,

Automated Twitter User Tagging Using Natural Language Processing

Introduction

Previously, some EDA and interactive data visualization have been implemented to Twitter data. In this post, I will talk about how to use unstructured text data from Twitter to build an automated user-tagging application. Since Twitter data is text-rich, the primary approach to handle Twitter data is Natural Language Processing (NLP). NLP teaches the computer about how to understand natural languages such as English. Combined with Machine Learning, it is possible to train a model that makes decisions based on the text input. In other words, based on NLP, we can train classifiers to do various tasks such as user clustering. For example, in this post, I will walk you through the process of building a binary classifier to predict if the Twitter user is a programmer or a gamer using that user’s profile as the input.

Data Ingestion

Following the steps in my previous post about streaming and storing tweets in a PostgreSQL database, I launched another round of Twitter streaming, using keywords in [python, programming, java, C++, fortnite]. The first 4 keywords capture users that are more likely to be programmers, and the last keyword, fortnite, on the other hand, captures users that are more likely to be gamers. The streaming last around 72 hours results in 299741 non-duplicate tweets tweeted from 195660 Twitter users.  To ensure the data purity, two additional conditions are applied to clean the raw samples:

  1. The user’s profile description is not empty and is written in English.
  2. The user’s tweets contain label information, in other words, the tweets under this user must include one of the five keywords specified, so the user can be correctly labeled.

After filtering, the total number of user reduce to 104303.  Figure 1 shows the first ten users and their profile description. 

Figure 1. First ten records of the user dataset.
Figure 1. First ten records of the user dataset.

Data Preprocessing

As mentioned in my last post, computational hardware can only handle numerical data, which is eventually converted into binary bytes. It is not possible that we simply feed the description column in Figure 1 to the machine and tell it, learn from this. The ultimate goal of text recognition is to convert the text data into numerical data and train our computer to be able to understand the underlying information. Natural Language Processing (NLP) does precisely this. NLP is a broad field of study that covers many cutting-edge challenges, such as sentiment analysis, topic recognition, and high-precision translation, etc. The model I talk about here belongs to topic-recognition, and there are several crucial steps to follow before building the machine learning model. 

Tokenization

Tokenization means to split the document into individual words. In our case, not only the whole description needs to be divided into a list of different words, but it also requires a fair amount of cleaning work, so all the symbols, emojis, and web links are removed. For example, define a cleaning function that takes the raw text as input and generates a cleaned list of words as output, as shown below: 

The steps followed to generate the processed data are the following:

  1. Lowercasing all the letters in the sentence.
  2. Tokenize the sentence (split the whole string into words).
  3. Keep only alphabetic words (remove other characters, e.g. emojis, Japanese symbols).
  4. Remove all stopwords (such as and, the).
  5. Lemmatize all tokens.

Lemmatization is the process of returning only the base of a word, which is called lemma of the word. For example, the lemma of words doggy, dogs is dog.

Data-Balancing

For any classification problem, if the dataset is skewed or imbalanced, the model would be biased towards the class that has more samples. The Twitter-user dataset is pre-labeled in the database by pattern matching their tweets. The distribution of the two classes is shown in Figure 2: 

Figure 2. The number of labels for all Twitter users.
Figure 2. The number of labels for all Twitter users.

Even though I used three other keywords to capture tweets from programmers, the number of programming users are still not even half of the number of gamers. There are possible approaches to balance the dataset: undersampling and oversampling. While either of them can eliminate the impact of the imbalanced dataset, they have different drawbacks and should be wisely chosen with different situations. In our case, since we do not have a massive amount of data, if we decide to undersample the gamer class, half of the samples would be dropped. For data analytics, reducing data is absolutely the least approach to go for. Hence here we choose to oversample the programmer class. The problem with this approach, though, is that the additional samples are just duplicates of random samples. 

To oversample the dataset, I use the imbalanced-learn library. Imblearn provides multiple strategies for re-sampling the dataset such as Naive Random Oversampling, Synthetic Minority Oversampling Technique (SMOTE), and Adaptive Synthetic sampling method for oversampling, Prototype Generation and Prototype Selection for undersampling. After applying Naive Random Oversampling to the original dataset, the amount of samples has been balanced to $67289$ for each class. The number is smaller since after preprocessing, some profiles are removed for being empty after removing all non-alphabetical words. 

Model Implementation

There are numerous machine learning models for binary classification, such as Logistic Regression, Random Forest, and Naive Bayes. To pick the best model for our application, we can write a python script to loop through the model pool and plot their performance to decide which is the best fit. With all default parameters for all the models and a small amount of training data, one can get a general idea about how the models would perform on a specific classification problem. From there, based on the initial performances, pick the best model to fine-tune its hyperparameters to achieve the best score.

To roughly train and evaluate all the models, two essential libraries are necessary: scikit-learn and yellowbrick, scikit-learn for building machine learning pipeline and yellowbrick for both model evaluation and report visualization. 

Implementing a machine learning pipeline is a great point to start. With the model as the last section in the pipeline, one can easily switch between different machine learning models with the same data configuration. The first section in the pipeline is CountVectorizer.

CountVectorizer

Since the preprocessing only tokenize all the descriptions, to enable the computer to understand the text content, the next step is to transform all the text information into numbers. First, similar to a hashtable, we give each word in the user’s description a unique index, and the number is decided by the appearances of that word among all the descriptions. For instance, as shown below, we first count the number of all the words in our dataset, then assign a number to each word. And the order of the indexing is the same with the descending order of the counts, which means 0 stands for the most frequently used word (http), and 1 stands for the second most used word (fortnite), and so on.  

It turns out that there are 89548 unique tokens in the entire dataset, which means the indices are in the range of [0, 89547]. Now we can use a vector that contains 89548 elements to represent a specific user description by putting 1s to the corresponding indices. For example, If fortnite is in the description and http is not, the first element in the vector would be 0, and the second element would be 1, and so on. However, it is not possible to store vectors with 89547 elements in memory as many as we want due to memory limit. In fact, the majority of the vector is composed of zeros, and only a small portion of it is ones, which is also called a sparse vector. Large-size sparse vectors are usually stored in sparse representations where only the indices and the corresponding non-zero values are stored. A simple example is shown below:

Here the dense vector [0, 0, 2, 1, 0] is transformed into a sparse matrix by scipy. In output 61, it says there are only 2 elements stored, which is the non-zero values of 2 and 1. And the printed information shows that it is stored with the indices (0, 2), (0, 3) (row 0, columns 2, 3) and the value 2, 1.

Fortunately, the CountVectorizer() module in scikit-learn does all the heavy lifting mentioned above, including counting, indexing, and saving the results as a sparse matrix. 

TfidfTransformer

While CountVectorizer handles all the work of converting text information into equivalent numerical values, which might work fine in most cases. However, it would be even better if a TfidfTransformer is attached next to the CountVectorizer. Tf stands for term-frequency, which is a normalization of the output from the CountVectorizer, and tf-idf means term-frequency times inverse document-frequency.

The formula for calculating tf-idf of a term t in a document set d is

tfidf(t, d) = tf(t, d) * idf(t)

where tf(t, d) calculates the term frequency of t in a document set, and

idf(t) = log[n/df(t)] + 1

computes the inverse ducoment frequency of term t, where df(t) is the document frequency of t. The document frequency is the number of documents in the set that contains the term t.

From the formula, when the document frequency of term t is approaching the number of entire document set n, the logarithmic would be approaching 0. And the weight for the term t would go towards 1. Whereas the term that has a lower document frequency would have a higher idf weight to magnify its term-frequency tf(t, d).

From the math behind tf-idf, one can tell that the goal of it is to scale down the impact of tokens that occur in the majority of the document set hence are empirically less informative than the tokens are occur in a smaller portion of the document set.

Similarly, TfidfTransformer module in scikit-learning receives the sparse matrix from the CountVectorizer and transforms it into an inverse-document-frequency weighted sparse matrix. 

Model Selection

So far, two sections of the pipeline are implemented, to finish up the pipeline, append a machine learning model after the TfidfTransformer. The next step is to pick a specific model that fits our classification problem the best. With a pre-defined model pool, a simple for loop would do the trick:

# pipeline
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
# classification models
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import MultinomialNB
from sklearn.svm import LinearSVC
# performance visualization tools
from yellowbrick.classifier import ClassPredictionError
from yellowbrick.classifier import ClassificationReport
from yellowbrick.classifier import ROCAUC
from yellowbrick.classifier import ConfusionMatrix
# train and visualize the results of each model individually
for model in [LogisticRegression, RandomForestClassifier, LinearSVC,
              KNeighborsClassifier, MultinomialNB]:
    # instantiate a pipeline
    pipeline = Pipeline([
    ('vect', CountVectorizer()),
    ('tfidf', TfidfTransformer()),
    ('model', model()),
    ])
    # print the current model in the pipeline
    print(model)
    # for four different metrics and visualization
    for viz in [ClassificationReport, ClassPredictionError,
                ROCAUC, ConfusionMatrix]:
        # print current model and visualization combination
        print(model, viz)
        # fit and evaluate the model
        viz1 = viz(pipeline)
        viz1.fit(X_train, y_train)
        viz1.score(X_test, y_test)
        plt.figure()
        viz1.poof()

 

It took around 5 mins to execute the for-loop for five models and four visualizations for each model with a single CPU core. And the results are shown in Figure 3- Figure 7.

Figure 3. Visualization of metrics for Logistic Regression Classifier.
Figure 3. Visualization of metrics for Logistic Regression Classifier.
Figure 4. Visualization of metrics for Random Forest Classifier.
Figure 4. Visualization of metrics for Random Forest Classifier.
Figure 5. Visualization of metrics for Linear SVC (Support Vector Classifier).
Figure 5. Visualization of metrics for Linear SVC (Support Vector Classifier).
Figure 6. Visualization of metrics for K-Neighbors (KNN) Classifier.
Figure 6. Visualization of metrics for K-Neighbors (KNN) Classifier.
Figure 7. Visualization of metrics for Naive Bayes Classifier.
Figure 7. Visualization of metrics for Naive Bayes Classifier.

As shown in the figures above, five models (from top to bottom: Logistic Regression, Random Forest, Linear SVC, K-Nearest Neighbor, and Multinomial Naive Bayes) are trained and evaluated individually (from left to right: prediction error, confusion matrix, precision-recall and f1-score, and roc-curve). In the prediction plots (first column), blue area denotes the number of predictions for class 0 (programmer), and the green area stands for predictions for class 1 (gamer).

It is clear that Random Forest, Linear SVC, and Naive Bayes classifiers can provide pretty good scores while K-Nearest Neighbors (KNN) classifier is not able to handle the task. Since the dimension of the dataset is too high (up to 80000 dimensions), the distance between two samples in the hyperspace can be tremendous thus impact the performance of the voting strategy of KNN algorithm.

Cross-Validation & Model Tuning

Figure 8 shows the benchmark profiling for all five models. Even though Random Forest Classifier has a pretty good performance, the training time is too long to be considered. Therefore, the model pool is shrunk into two models: Linear SVC and Naive Bayes. The next step is hyperparameter tuning and cross-validation.

Figure 8. Benchmark profiling for all the models.
Figure 8. Benchmark profiling for all the models.
GridSearchCV

GridSearchCV is a submodule provided by scikit-learn, which combines grid search and cross-validation together into one entity. It explicitly searches over a specified set of parameters. For each combination of parameters, it performs k-fold cross-validation automatically. Execute the following code to find the best hyperparameters for Naive Bayes classifier:

# instantiate the pipeline with some predefined parameters
pipeline = Pipeline([
    ('vect', CountVectorizer(ngram_range=(1, 2))),
    ('tfidf', TfidfTransformer(use_idf=True)),
    ('nb', MultinomialNB(fit_prior=False)),
])
# specify parameter grid to search over
parameters = {
    'vect__max_df': (0.25, 0.5),
    'nb__alpha': (0, 0.1, 0.2, 0.3, 0.4)
}
# initialize the GridSearchCV object
grid_search = GridSearchCV(pipeline, parameters, cv=5,
                           n_jobs=8, verbose=10)
# print progress to the console
print("Performing grid search...")
print("pipeline:", [name for name, _ in pipeline.steps])
print("parameters:")
pprint(parameters)
t0 = time()
# start searching
grid_search.fit(X_train, y_train)
print("done in %0.3fs" % (time() - t0))
print()
# print the best paramters found
print("Best score: %0.3f" % grid_search.best_score_)
print("Best parameters set:")
best_parameters = grid_search.best_estimator_.get_params()
for param_name in sorted(parameters.keys()):
    print("t%s: %r" % (param_name, best_parameters[param_name]))

 

To reduce the search time, some basic parameters are predefined when initializing the pipeline, such as ngram_range for CountVectorizer and use_idf for TfidfTransformer. ngram_range stands for the range of n-gram to be extracted. Here I choose to only consider 1-gram and 2-grams sequences.

Two parameters for two modules in the pipeline are explicitly set in the parameter grid, including max_df for CountVectorizer and alpha for Naive Bayes classifier. Specifically, max_df stands for maximal document-frequency, which let the CountVectorizer only to use the tokens that have the lower document-frequency to build the vocabulary. In other words, all the tokens that have a document-frequency higher than this threshold will be ignored so that the classifier can be more focused on other relatively more informative tokens. On the other hand, alpha is the smoothing coefficient that counts for unseen feature points to avoid zero probabilities in downstream calculations. 

The output of executing the code is shown below:

There are 10 different combinations of parameters to fit and a 5-fold cross-validation for each model; thus, 50 fits in total, as shown in the output. For an 8-core CPU, it took 96 seconds to find the best model with 5-fold CV. The corresponding parameters for the best model are picked, with alpha=0.1 and max_df=0.25. And the evaluation of the best model is shown in Figure 9: 

Figure 9. Evaluation for Naive Bayes Classifier after grid-search and cross-validation.
Figure 9. Evaluation for Naive Bayes Classifier after grid-search and cross-validation.

Compares to the default model, the best model after hyperparameter tuning has better performance: the precision for class 1 has increased from 0.852 to 0.92, the recall for class 0 has increased from 0.858 to 0.927, and the AUC has increased from 0.92 to 0.96, respectively.

Similarly, we can define a parameter grid for Linear SVC to find the best model, as shown in Figure 10: 

Figure 10. Evaluation for Linear SVC after grid-search and cross-validation.
Figure 10. Evaluation for Linear SVC after grid-search and cross-validation.

Conlusion

So far, the implementation of an automated Twitter user tagging model using NLP and machine learning models is done. Figure 11 illustrates a small demo of the application using the pre-trained Linear SVC model. I use the descriptions of two Twitter users as the input to generate their tags, i.e., to predict if he/she is a gamer or a programmer. One of them is Eefje Depoortere (@sjokz), who is my favorite E-sports host. And the other one is my friend Ben (@Benben_ben__), who is a programmer.  

demo1

Figure 11. A demo for an automated user-tagging application using pre-trained NLP model.

While the model is able to accurately predict the label for these two users, there is still space to improve. Such as multi-labeling and coupling with Computer Vision, etc. 

I had tons of fun during the process of developing this application. You can find all the code for this project here. As always, feel free to correct me if there is any mistake or concerns. Welcome to let me know if you have any questions.

Thanks for reading 🙂

,

Distributed Machine Learning Using PySpark

Introduction

Big Data is not a new thing nowadays, and numerous platforms and applications have been created to take advantage of Big Data. Apache Spark is one of the most popular platforms for handling Machine Learning with Big Data.  By implicitly distribute data into clusters, Spark enables developers to focus more on their analytical tasks without worrying about data parallelism under the hood. To learn how Apache Spark works and how to build a distributed ML pipeline, a local cluster on my desktop is implemented with PySpark. In this post, I will talk about how to make ML pipelines with PySpark and its automated parallel-computing functionality. 

Problem Definition & Data Ingestion

One of the most exciting ML applications would be predictive modeling, which utilizes the data collected in the past to train a model that is capable of predicting the future. A predictive model can be a regression model or a classification model. For the demonstration, I will use Apache Spark to build a predictive classification model. The dataset I use is collected from The U.S. Department of Transportation’s (DOT) Bureau of Transportation Statistics, which contains more than 5 million records of the flights’ information during 2015, including the carrier of the airline, the original airport, destination airport, the distance of the flight, the time spent, the departure delay time, and arrival delay time, etc. I will use this dataset to train a model to predict if a flight will be delayed or not. 

Data Preparation

In order to do some basic EDA, the dataset needs to be preprocessed. The following code sets up a Spark working environment (Spark Session) and loads the data into the Spark Session as a Spark DataFrame:

# instantiate a spark session
from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local[*]')
                            .appName('flights_delay')
                            .config('spark.driver.memory', '8g')
                            .getOrCreate()
# load raw data
df = spark.read.csv('/home/shihao/Desktop/flights/flights.csv', header=True,
                    inferSchema=True, nullValue=' ')
# clean data set
to_keep = ['MONTH', 'DAY', 'DAY_OF_WEEK', 'AIRLINE', 'FLIGHT_NUMBER',
           'ORIGIN_AIRPORT', 'DESTINATION_AIRPORT', 'TAXI_OUT',
           'SCHEDULED_DEPARTURE', 'DEPARTURE_DELAY', 'DISTANCE',
           'SCHEDULED_ARRIVAL', 'ARRIVAL_DELAY']
df = df.select(to_keep)
df = df.dropna()

 

When instantiate the Spark session in PySpark, passing 'local[*]' to .master() sets Spark to use all the available devices as executor (8-core CPU hence 8 workers). The original dataset has 31 columns, here I only keep 13 of them, since some columns cannot be acquired beforehand for the prediction, such as the wheels-off time and tail number.

After selecting all the useful columns, drop all the null values in the Spark DataFrame. Since only roughly 20% of the total flights are delayed (flights are delayed if the actual arrival time is more than 15 minutes later than the scheduled arrival time), use the following code to create the label column. The following code also balances the dataset, specifically, makes the number of delayed samples equal to the number of on-time samples.

# create label
df = df.withColumn('label', (df.ARRIVAL_DELAY > 15).cast('integer'))
# balance data set
# get number of delayed flights
num_0 = df.filter(df.label==0).count()
# calculate the sub-sampling ratio for the on-time flights
ratio = (df.count() - num_0)/num_0
# under sample the redundant class
df = df.sampleBy('label', {0: ratio, 1:1})

Explanatory Data Analysis

Now the dataset is clean and balanced, we can do some EDA to examine the dataset. First, take a look at how each airline is performing in terms of delivering flights on time. As shown in Figure 1 (left), Frontier Airlines (F9) has the highest average delay time (68.58 minutes) among 14 airline companies and Hawaiian Airlines (HA) has the lowest average delay time (40.98 minutes). 

Figure 1. The sorted average arrival delay and maximal delay of each airline.
Figure 1. The sorted average arrival delay and maximal delay of each airline.

On the other hand, American Airlines (AA) has the most extended delay with 1546 mins \approx 25.77 hrs, and Virgin America (VX) has the shorted maximal delay with 440 mins \approx 7.33 hrs as shown in Figure 1 (right).

Another interesting fact is illustrated in Figure 2, seems like all the airlines are doing pretty poorly around February and June, and doing better in April, September, and October. Therefore, Month should be a useful indicator for our predictive model. 

Figure 2. Monthly average arrival difference of each airline.
Figure 2. Monthly average arrival difference of each airline.

How about other features? Such as the distance of the flight. Does it have some relationship with the arrival timing? Figure 3 shows the average arrival difference for each airline, along with different flight distances. Arrival difference is defined as the difference between the scheduled arrival time and the actual arrival time. 

Figure 3. Average arrival difference plotted along distance for each airline.
Figure 3. Average arrival difference plotted along distance for each airline.

While there are a lot of outliers in Figure 3, we can still see a small trend that longer distance tends to have longer arrival difference. Figure 4. shows the relationship between arrival difference and taxi-out time, where we can see a more apparent linear relationship. 

Figure 4. Average arrival difference plotted along taxi-out time.
Figure 4. Average arrival difference plotted along taxi-out time.

Finally, as shown in Figure 5, departure-delay is the best predictor for our model.

Figure 5. Average arrival difference plotted along departure-delay.
Figure 5. Average arrival difference plotted along departure-delay.

Data Segregation

Now that the data-preprocessing and EDA are done, we can go ahead and build the model. I created three simple ML models with the dataset and compare their performances. Since the problem is to predict whether a flight will be delayed or not, which is a binary classification problem, I choose the Random Forest Classification model, Logistic Regression model, and Neural Network model, to implement the predictive model. 

First of all, split the dataset into training and test set before building ML pipelines to avoid leakage. Leakage happens when any of the .fit() methods are applied to the test set, which may induce unneglectable bias to our model.  Therefore, extra attention should be paid to only apply .transform() method to the test set. The dataset can be split in PySpark using DataFrame.randomSplit([training_ratio, test_ratio], random_state) method as shown here:

# create training and test set
flights_train, flights_test = df.randomSplit([.8, .2], 5)

Model training

When the data is ready, we can begin to build our machine learning pipeline and train the model on the training set. PySpark provides us powerful sub-modules to create fully functional ML pipeline object with the minimal code. Before putting up a complete pipeline, we need to build each individual part in the pipeline.

First, convert all the categorical data in the dataset into numerical data since the underlying machine is more familiar with numbers than texts. Similar to pandas.get_dummies(), use StringIndexer() combined with OneHotEncoderEstimator() sub-module in PySpark to convert all the categorical data into numerical data. For the flights dataset, there are three categorical columns, including AIRLINE, ORIGINAL_AIRPORT, and DESTINATION_AIRPORT. Therefore, construct three StringIndexer() objects, like so: 

# create parts
from pyspark.ml.feature import StringIndexer, OneHotEncoderEstimator
arr_indexer = StringIndexer(inputCol='AIRLINE', outputCol='arr_idx').setHandleInvalid("keep")
org_indexer = StringIndexer(inputCol='ORIGIN_AIRPORT', outputCol='org_idx').setHandleInvalid("keep")
des_indexer = StringIndexer(inputCol='DESTINATION_AIRPORT', outputCol='des_idx').setHandleInvalid("keep")
onehot = OneHotEncoderEstimator(inputCols=[arr_indexer.getOutputCol(), org_indexer.getOutputCol(),
                                           des_indexer.getOutputCol()],
                                outputCols=['arr_dummy', 'org_dummy', 'des_dummy'])

 

The OneHotEncoderEstimator() object will convert all the categorical columns into their corresponding numerical columns with sparse vector representations. Basically, a sparse vector representation is a more efficient way to represent a vector if the majority of the vector is zero values. 

Next, merge all the sparse vectors and other numerical values in the dataset into a single sparse vector by concatenating them one by one and call this new vector “features”, which can be done using the PySpark VectorAssembler() sub-module:

from pyspark.ml.feature import VectorAssembler
assembler = VectorAssembler(inputCols=['MONTH', 'DAY', 'DAY_OF_WEEK',
                                       'arr_dummy', 'org_dummy', 'des_dummy',
                                       'SCHEDULED_DEPARTURE','TAXI_OUT',
                                       'DEPARTURE_DELAY', 'DISTANCE',
                                       'SCHEDULED_ARRIVAL'],
                            outputCol='features')

 

Everything so far should be identical for all the three ML models since they all share the same dataset. After this point, however, the workflow will be slightly different for different models in terms of hyperparameter-tuning. For Random Forest Classifier and Logistic Regression Classifier, instantiate the ML model and attach it to the end of our pipeline:

from pyspark.ml.classification import RandomForestClassifier
from pyspark.ml import Pipeline
rf = RandomForestClassifier(labelCol='label', featuresCol='features', 
                            numTrees=30, maxDepth=30)
pipeline = Pipeline(stages=[arr_indexer, org_indexer, des_indexer,
                            onehot, assembler, rf])

 

For Logistic Regression Model, switch to the following lines of code to instantiate the pipeline:

from pyspark.ml.classification import LogisticRegression
from pyspark.ml import Pipeline
lr = LogisticRegression(maxIter=30)
pipeline = Pipeline(stages=[arr_indexer, org_indexer, des_indexer, onehot, assembler, lr])

 

During training, it is common practice to implement K-fold Cross-validation and Grid Search. Fortunately, PySpark provides sub-modules for both of them as well. To build a parameter-searching grid, use the ParamGridBuilder() sub-module and pass the parameter map into the CrossValidator(). The code for the Random Forest Classifier is shown here:

from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
param = ParamGridBuilder().build()
cv = CrossValidator(estimator=pipeline,
                    estimatorParamMaps=params,
                    evaluator=evaluator,
                    numFolds=5)

 

Do not pass any parameters when building the parameter map for Random Forest (RF) Classifier since its parameters are more straight forward and easy to interpret. The hyperparameters for RF are initialized to the best case when instantiating the RF object previously. Specifically, numTrees stands for the number of single decision trees make up the RF classifier and maxDepth stands for the maximal depth for each tree in the forest. RF does not require a lot of tuning since the higher the values of these two parameters, the better the performance of the model in terms of test-accuracy. On the other hand, of course, with more, deeper trees, the training process takes linearly more time.

Whereas for Logistic Regression (LR) Classifier, build the parameter grid by doing the following:

from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
params = ParamGridBuilder()
params = params.addGrid(lr.regParam, [.01, .1, 1, 10]) 
               .addGrid(lr.elasticNetParam, [0, .5, 1])
params = params.build()
cv = CrossValidator(estimator=pipeline,
                    estimatorParamMaps=params,
                    evaluator=evaluator,
                    numFolds=5)

 

As shown in the code, there are two parameters to be tuned for LR classifier. lr.regParam is the regularization coefficient which specifies that, in the range of 0.01 to 10, how much the model should be regularized. The bigger the number, the more the model is regularized. Meanwhile, lr.elasticNetParam specifies the way the model is regularized, 0 means it’s an L2 penalty and 1 means it’s an L1 penalty. And 0.5 means the penalty is a combination of both L1 and L2.

For Neural Network Classifier, the story is a little bit different. Since Spark Mllib does not have predefined models for ANNs, instead, use keras (with tensorflow as backend) to build the ANN model for comparison. Also, keras Dense layer does not take sparse vectors as input, therefore just convert the dataset in a conventional way. In other words, creating dummy variables for categorical data:

import pandas as pd
from sklearn.model_selection import train_test_split
from keras.utils import to_categorical
# convert Spark DataFrame into Pandas DataFrame
df_pd = df.toPandas()
# create feature and label set
X = df_pd.drop(['label', 'ARRIVAL_DELAY'], axis=1)
y = df_pd.label
# convert categorical data into numerical data
X = pd.get_dummies(X)
# split into training and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
                                                    random_state=5)
# reshape the dataset to fit ANNs
X_train = X_train.values[...,None]
X_test = X_test.values[...,None]
# convert the label into categorical data
y_train = to_categorical(y_train.values)
y_test = to_categorical(y_test.values)

 

Use the following code the initialize the ANN model:

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Flatten
from keras.layers import BatchNormalization
ann = Sequential()
ann.add(Dense(10, activation='relu', input_shape=(X_train.columns.size, 1,)))
ann.add(BatchNormalization())
ann.add(Flatten())
ann.add(Dense(2, activation='softmax'))
ann.compile(optimizer='adam', metrics=['accuracy'], loss='categorical_crossentropy')

 

A simple ANN structure that only has one hidden layer with ten hidden nodes is modeled for comparison. After the hidden layer, a BatchNormalization() layer normalizes the output of the hidden layer. Since the numerical scales of the features are different, the normalization layer makes it easier for the model to converge.

The last step is to train our models. For ANN, call the following line of code: 

from keras.callbacks import ModelCheckpoint
model_path = 'ann_keras_model'
model_check = ModelCheckpoint(model_path, save_best_only=True)
history = ann.fit(X_train, y_train,
                  batch_size = 200, epochs = 25,
                  validation_split = 0.2,
                  callbacks=[model_check])

 

Add a callback function for the training so that only the best model during training will be saved. For the Random Forest and Logistic Regression models, simply call the following line for training:

cv = cv.fit(flights_train)

Model Evaluation

For a binary classification problem, it is common practice to evaluate the performance of the model by plotting the ROC (Receiver Operating Characteristic) curve or calculating precision, recall, and F1-scores. For Random Forest classifier and Logistic Regression model, use the following code to calculate the AUC (area under ROC curve), precision, and recall, etc:

from pyspark.ml.evaluate import BinaryClassificationEvaluator
# get the prediction
prediction = cv.transform(flights_test)
# initialize the evaluator
evaluator = BinaryClassificationEvaluator()
# calculate AUC
auc = evaluator.evaluate(prediction, {evaluator.metricName: 'areaUnderROC'})
print('AUC: %0.3f' % auc)
# compute TN, TP, FN, and FP
prediction.groupBy('label', 'prediction').count().show()
# Calculate the elements of the confusion matrix
TN = prediction.filter('prediction = 0 AND label = prediction').count()
TP = prediction.filter('prediction = 1 AND label = prediction').count()
FN = prediction.filter('prediction = 0 AND label <> prediction').count()
FP = prediction.filter('prediction = 1 AND label <> prediction').count()
# calculate accuracy, precision, recall, and F1-score
accuracy = (TN + TP) / (TN + TP + FN + FP)
precision = TP / (TP + FP)
recall = TP / (TP + FN)
F =  2 * (precision*recall) / (precision + recall)
print('n precision: %0.3f' % precision)
print('n recall: %0.3f' % recall)
print('n accuracy: %0.3f' % accuracy)
print('n F1 score: %0.3f' % F)

 

Furthermore, take a look at the parameters of the best Logistic Regression model selected by Grid Search:

For the Logistic Regression model, the Grid Search chooses to have a combination of L1 and L2 penalty to get the best performance. And it also chooses to use the least regularization (0.01). For the Neural Network model, Figure 6 (left) shows the training history about training loss, validation loss, training accuracy, and validation accuracy. The training takes 15 seconds for a single epoch on a GPU implemented device. The right-hand figure shows the ROC curves for the three models.

Figure 6. Training history (left) for ANN and ROC curves (right).
Figure 6. Training history (left) for ANN and ROC curves (right).

Call .transform() method to the held-out test set to further evaluate these models :

from pyspark.ml.evaluation import BinaryClassificationEvaluator
# get predictions
prediction = cv.transform(flights_test)
# Calculate the elements of the confusion matrix
TN = prediction.filter('prediction = 0 AND label = prediction').count()
TP = prediction.filter('prediction = 1 AND label = prediction').count()
FN = prediction.filter('prediction = 0 AND label <> prediction').count()
FP = prediction.filter('prediction = 1 AND label <> prediction').count()
# show confusion matrix
prediction.groupBy('label', 'prediction').count().show()
# calculate metrics by the confusion matrix
accuracy = (TN + TP) / (TN + TP + FN + FP)
precision = TP / (TP + FP)
recall = TP / (TP + FN)
F =  2 * (precision*recall) / (precision + recall)
# calculate auc
evaluator = BinaryClassificationEvaluator()
auc = evaluator.evaluate(prediction, {evaluator.metricName: 'areaUnderROC'})
print('n precision: %0.3f' % precision)
print('n recall: %0.3f' % recall)
print('n accuracy: %0.3f' % accuracy)
print('n F1 score: %0.3f' % F)
print('AUC: %0.3f' % auc)

 

The code above prints the confusion matrix of the prediction and also calculates all metrics. And the results are summarized as a table shown below:

table1

 

From the table, all three models can predict the label pretty accurately. While Neural Network gets higher accuracy, its precision and recall are relatively lower. After all, the overall difference between these three models is relatively small. 

Conlusion

With all the models providing similar performances, it is fair to say that, for machine learning, sometimes it is the data that dominates the solution, instead of the model. In other words, with enough dataset, all models can achieve a pretty good score. With that being said, the focus shifted from data to approach in Big Data era. What’s important is no longer the result, but the way to approach the result.

For our experiment, the flights’ dataset contains more than 2 million records, which can take a long time for stand-alone devices to implement the machine learning application. Apache Spark provides an automatic distributed clustering-computing framework, under which a simple Logistic Regression can achieve almost identical performance as a GPU implemented Neural Network model with shorter training time. 

In this post, I talked about how to use PySpark to build machine learning pipelines which are suitable for Big Data analysis. With a smaller size of data, though, using standard machine learning library should be sufficient and more efficient. Nevertheless, Apache Spark is definitely a reliable tool for solving challenging machine learning problems using Big Data.

The author is still learning Apache Spark, and there might be error or mistakes in this post. Welcome to correct me if I’m wrong and let me know if you have any questions!

Thanks for reading 🙂 

,

Geographical Visualization and Network Analysis for Twitter Data

Introduction

In the previous post, I did some fundamental Exploratory Data Analysis (EDA) with the streamed Twitter data. While several essential pieces of information were extracted from those data, we were only looking at a tiny portion of the entire database. This post examines two other hugely impactful data sources: geographical data and social network data.  One can convert the geographical information to a figure by visualizing Twitter data on a geospatial map (like this demo shown here). Meanwhile, network analysis can be implemented by inspecting several user-based tables in our database.

Why put Twitter data on a map? First, it allows us to observe the concentration of the tweets collected spatially. Second, it enables us to work with the area of interest (if have one) more efficiently. Also, why network analysis? First, it helps to identify the user (or users) that has the most significant potential to impact a field. Second, it also helps clustering user base depend on some preliminary conditions. Of course, there are many other advantages of doing geographical and social-network analysis on Twitter data,  and some of them are right in this post.  

Visualize Twitter Data on a map

First, let us visualize the tweets collected geographically by denoting each one of them as a dot (circle) on a Mercator projected map. There are a few steps for data preparation:

  1. Only $1-3%$ of total tweets have geographical information embedded in their JSON file, hence we need to combine the base_tweets table with the place table, then filter out those tweets that do not have geographical information, to get those tweets that can be displayed on our map.
  2. When hovering the mouse on to a specific point shown on the map, it would be better to show some extra information related to that tweet, such as which game it is about, which user sent that tweet, how many followers that user currently has (thus how big is the impact of that tweet), etc. To do this, combine more tables into the base_tweets table, including tweet_user, hashtags, text, etc.
  3. Sometimes the keyword for the tweet is not contained in the hashtags or text field, and they might be mentioned in the quote tweet or retweet instead. To handle tweets like this, more tables need to combined into the base_tweets table, such as retweeted_tweet table. 

The following query does all the work I just mentioned. And the query results (10 records) are shown in Table 1.

Table1. Queried data for visualizing tweets geographically.
Table1. Queried data for visualizing tweets geographically.

From the query, seven tables in the relational database are combined through LEFT JOIN on different keys to provide the information needed. Now, export the query results as a CSV file and load them in Python.

Note that the geographical information for each tweet is stored as a bounding box on the map, and the coordinates column contains the four corners of the bounding box. To get the corresponding x and y coordinates on a Mercator-projected map, some mathematical calculations are necessary. For your reference, the functions I wrote for converting a bounding box into Mercator coordinates is shown below:

def get_lon_lat(df, geo_type):
    """
    Get longitude and latitude from the dataframe
    Append two new columns into the original dataframe
    Parameters
    ----------
        df: pandas dataframe
            input dataframe containing geographical information
        geo_type: string
            the type of the geographical dataframe
    Returns
    -------
        df: pandas dataframe
            with longitude and latitude extracted
    """
    # if the input dataframe is geo
    if geo_type == 'geo':
        # then the longitude is the second value in the list
        df['longitude'] = df['coordinates'].apply(lambda x: literal_eval(x)[1])
        # and the latitude is the first value
        df['latitude'] = df['coordinates'].apply(lambda x: literal_eval(x)[0])
    # if the input dataframe is coordinates
    elif geo_type == 'coordinates':
        # it is the other way around
        # NOTE: geo and coordinates contains the same information
        df['longitude'] = df['coordinates'].apply(lambda x: literal_eval(x)[0])
        df['latitude'] = df['coordinates'].apply(lambda x: literal_eval(x)[1])
    # if the input dataframe is place
    # then list contains 4 points of a bounding box
    # we need to calculate the centroid of the bounding box
    elif geo_type == 'place':
        # define a function to extract the centroid
        def get_centroid(row):
            # literalize the str to a list
            lst = literal_eval(row)
            # get the minimal and maximal latitude
            lat0 = lst[0][0][1]
            lat1 = lst[0][1][1]
            # calculate the center
            lat = (lat0 + lat1)/2
            # same for the longitude
            lon0 = lst[0][0][0]
            lon1 = lst[0][2][0]
            lon = (lon0 + lon1)/2
            # return the centroid
            return (lon, lat)
        # apply the defined function to the whole dataset
        df['longitude'] = df['coordinates'].apply(lambda x: get_centroid(x)[0])
        df['latitude'] = df['coordinates'].apply(lambda x: get_centroid(x)[1])
    # return the new dataset with longitude and latitude as new columns
    return df

def lonlat2merc(row):
    """
    Convert the longitude and latitude of a geographical point to
    x and y coordinates on a Mercator projected map
    Parameters
    ----------
        row: float, one row of a pandas dataframe
            Before apply this function to a dataframe
            make sure the dataframe contains two columns correspond
            to longitudinal and latitudinal numbers
    Returns
    -------
        (x, y): float tuple
            x and y coordinates of the points
    """
    # extract longitudinal and latitudinal info out of the dataframe row
    lon = row.iloc[0]
    lat = row.iloc[1]
    # do the conversion
    r_major = 6378137.000
    x = r_major * math.radians(lon)
    scale = x/lon
    y = 180.0/math.pi * math.log(math.tan(math.pi/4.0 + lat * (math.pi/180.0)/2.0)) * scale
    # return the X and Y coordinates
    return (x, y)
def add_xy_col(df):
    # simple function that calls the Merc conversion function to each row
    # and save x and y seperately to the dataframe
    df['merc_x'] = df[['longitude', 'latitude']].apply(lambda x: lonlat2merc(x)[0], axis=1)
    df['merc_y'] = df[['longitude', 'latitude']].apply(lambda x: lonlat2merc(x)[1], axis=1)
    return df

 

Once the x and y coordinates are available, one can put the tweets on to the map. Figure 1 shows the demo I have created:

figure1
Figure 1. Geographical map forTwitter data of 8 video games.

In this stand-alone bokeh plot, the circles for all the tweets are shaded in different colors for different games, and the size of the circle is determined by the number of followers of the user who sent that tweet.

By looking at Figure 1, one can quickly summarize several conclusions:

  1. Fortnite is super popular in North America and Europe, and Eastern South America compares to other regions.
  2. Dota 2 is more popular in South East Asia.
  3. Most North-Asian and African players do not use Twitter.
  4. Even though Apex Legends and Fortnite are comparably popular in Japan area as well, not many people like to attach their geo-location when they post on Twitter hence very few points on that side of the map.

On the other hand, if one zoom-in the map to the U.S. specifically, as shown in Figure 2:

figure2
Figure 2. Geographical map zoom in at the U.S.

The players for these games are mainly concentrated around Texas, California, and the Bay Area. You might already notice that there are only seven different colors for all the circles on the map while there are eight games in total, and the missing one is GTA5. As mentioned before, only a very small portion of all the tweets have geo-information embedded, unfortunately, none of the tweets about GTA5 contains geographical information. In fact, only 709 (out of more than 160k) tweets have their geo-location enabled. And the detailed numbers for all the point on the map are shown in Figure 3:

Bokeh Plot
Figure 3. Bar chart for the count of each game shown on the map.

In Figure 3, Fortnite has the most points on the map, which make it a little tricky if one wants to examine the geographical distribution of other games. Bokeh provides powerful tools that enable us to interact with the plot with an additional adjustment which the user can define themselves. These interactive plots are called Bokeh applications, which need to be hosted by connecting the code to a Bokeh server. When the user adjusts the parameters of the scheme, the server updates the data accordingly and replot the figure on the client end.

Figure 4 is a brief demonstration of the application. You can play with the app through this link, which is hosted on an AWS (Amazon Web Service) EC2 instance.

Figure 4. Interactive Bokeh application for video-game geographical visualization. The demo is recorded by Bandicam (free version), so you see its watermark on top of the image.
Figure 4. Interactive Bokeh application for video-game geographical visualization. The demo is recorded by Bandicam (free version), so you see its watermark on top of the image.

Now since there are more controls added to the plot, the geo-information from the Twitter data is much better interpreted:

  1. In the demo, there is a drop-down menu named Shading, which allows us to choose the way to shade the circles. It has two options, one called Game Name, which draws the colors by the name of the game, and you can refer to the legend on the bottom of the plot to distinguish between different games. The other option for the drop-down menu is Sentiment Score, which shades the circles depend on the sentiment scores of the tweets, the more negative the tweet is, the darker the circle would be. 
  2. A group of checkboxes below the Shading drop-down menu allows the user to pick which games they want to inspect individually on the map. This additional feature makes it easier to examine a specific game or a group of games.
  3. The last option of control is a slider called Hour, and it indicates the time window when the tweets were posted. Only those tweets posted during that one-hour window are shown on the map, by which one can examine the data on another domain: time domain. Set Hour equal to 50 to show all the tweets from all the hours. 

With all these tools available, here are another bunch of interesting conclusions about the data:

  1. CSGO is more prevalent in Europe than any other region in the world, while ApexLegends is more prevalent in North America.
  2. Dota2 and Hearthstone have higher sentiment score across the board.
  3. Most players’ tweets are neutral or slightly more favorable for these games. 

Network Analysis

There is a toolkit called NetworkX comes in handy when visualizing Network data. And in this section, I will explain how to extract the network information from the massive amount of Twitter data stored in our database and create highly interactive Bokeh plots to visualize the network.

There are three types of network existing in Twitter data, including reply-network, retweet-network, and quote-network. Each of these networks can convey different information about the community so we need to handle them individually. Firstly, extract the reply-network from the base_tweets table, specifically, the tweet_user_id and in_reply_to_user_id column. Then, use NetworkX to generate a network object from these two columns automatically.

For this experiment, I extracted the reply information only for Fortnite and removed the records where the Twitter user was replying to himherself, which results in a network with 1123 nodes and 860 edges. And the network is visualized in Figure 5 with two different network layouts. Note that you can check the id of the user by hovering your mouse on the nodes, and click-and-drag to select a group of nodes, and the edges and nodes connected to the selected nodes will be highlighted as well.

Bokeh Plot
Figure 5. Reply-network for Fortnite in Spring layout (left) and Kamada-Kawai layout (right).

The network visualization is highly informative in terms of highlighting the nodes that have a higher degree (the number of adjacent edges) in a network, which can be crucial for network analysis. In Figure 5, the number of replies for a specific user is characterized by the size of the node. Also, in the spring layout (plot on the left side), the denser a cluster of nodes is, the more users are connected to the node, and it is also expressed in the Kamada-Kawai layout (plot on the right side) by putting all the highly connected nodes to the center of the concentric circle. And Figure 6 is another example of the retweet-network for game CSGO.

Bokeh Plot
Figure 6. Retweet-network for CSGO in Spring layout (left) and Kamada-Kawai layout (right).

Alternatively, to reduce the number of irrelevant nodes and make the plot cleaner, plotting only the nodes that have a high degree-centrality value with their corresponding neighbors might be a better idea. And Figure 7 shows the sub-graph formed from the top-10 nodes (in terms of degree centrality) and their neighbors.

Bokeh Plot
Figure 7. Sub-retweet-network for CSGO in circular layout (left) and Kamada-Kawai layout (right).

Unlike the previous two figures, the plot on the left-hand side is plotted in a circular layout so we can spot out the nodes that have the highest degree centrality. Also, on the right hand side, the sub-communities and sub-reply-networks are better clustered, which makes it much easier to locate the high-impact Twitter user.

Conlusion

People often ignore or underestimate the power of visualizing data, whereas data interpretation is no less critical than data-processing or data-modeling by all means. Visualizing data in a more informative and reliable way is the key to find better solutions. In this post, I talked about how to create interactive applications to visualizing Twitter data on a geospatial map using Bokeh, as well as how to plot different types of networks using NetworkX. These demos shown here are just an illustration of the concept with minimal data, especially for the map application; thus, there is still a lot of space to be improved for each of them.

As always, you can find the code for generating all these demos here. Feel free to let me know if you have any questions.

Thanks for reading 🙂

,

Interactive Exploratory Data Analysis with Bokeh

Introduction

If one would recall, I talked about how to collect and store Twitter data in the last post (Figure 1). Now it is time to do some fun and exciting experiments with all the data available! First up is some fundamental exploratory data analysis that tells us the big story behind the data. Such as answering questions like, which game is the most popular one on Twitter? The game is more prevalent in which language? How often is the game being mentioned during a specific time window of the day? How does the community think about the game when people send tweets talking about it? Positively or negatively? Etc. All these questions will be answered by the end of this post. 

Figure 1. A relational database that stores information of more than 160,000 tweets.
Figure 1. A relational database that stores information of more than 160,000 tweets.

Game Ranking Analysis

The first question is, how the games are distributed on Twitter by counting the number of mentions for each game? Rigorously speaking, since one tweet could have multiple fields that contain text information, including text, extended_tweet, retweet_text, retweet_extended_text, quoted_tweet_text, and quoted_extended_tweet_text, to inspect the number of mentions for each game, one should check if the name of one specific game appears in any of these fields. To get a rough feeling about the total distribution, however, I approached through a shortcut by looking only at the hashtags field, which is already saved as an individual table in our database. The drawback is that not every tweet has a hashtag. Therefore, the number of tweets here is relatively smaller. A slightly smaller sample space should not matter too much, though, considering that almost every observation follows Normal Distribution in the real world.

Use the following query to extract game hashtags from the hashtags table in the database:

In the query above, first, create a view to avoid denormalizing any of our existing tables in the database and set a new column game in the view containing the video-game hashtags that are wanted. Also create new columns if_quoted and if_retweeted to indicate if the tweet is a retweet or quoted_tweet. Recall from the last post, when building the relational database, all tables are connected through multiple foreign keys in the base_tweets table. To extract the information, recombine tables by joining the hashtags table back to the base_tweets table, then left join the resulting table with retweeted_id table and quoted_id table. Then query all the columns and rows from the view to export and transport them into the Python environment to visualize the distribution. And the first ten records the query returns look like this: 

Figure 2. Extracted relevant video-game names from the hashtags.
Figure 2. Extracted relevant video-game names from the hashtags.

As shown in Figure 2, specific video-game names that we care about is extracted from all the hashtags, and the value is set to NULL if none of the names is mentioned in the hashtags, and true or false to check if it is from a retweet or a quoted tweet. Now, migrate this temporary table into a Pandas DataFrame to get some nice-looking visualization of the distribution as shown in Figure 3:

Bokeh Plot
Figure 3. Bar chart for all the games.

While there are some beautiful colors to differentiate between games, the amount of information conveyed in Figure 3 is limited. It is clear that Fortnite is the most popular game on Twitter since the majority of tweets are mentioning it. Surprise, GTA5 is the one that players mentioned for the least amount of times on Twitter, which is on the contrary to what is observed on Twitch. Well, if we think about it, it is not difficult to figure it out: GTA5 is a console game that has a limited size of the online community, but it is fun to watch on Twitch due to its game design.

Now we know that, on Twitter, the most popular game is Fortnite, and the least favorite game is GTA5. What else? Here is another more advanced bar chart in Figure 4:

Bokeh Plot
Figure 4. Another two bar charts for all the games.

The top chart in Figure 4. shows the number of quote tweets, retweets, and original tweets individually for each game, and the bottom chart is plotted in log scale for better comparison with smaller numbers. It turns out that, for most of the games, more than half of the total tweets are retweeted; even though Hearthstone has more mentions than GTA5, it has less original tweets than the latter; and Dota2 has the smallest original tweets percentage. 

Gaming Language Analysis

Besides the number of mentions for each game, similarly, we can query for the language information from the database, and analyze the language used in the gaming community globally. There are 42 languages used in the tweets streamed, including unknown languages marked as und, and some of them are plotted in the figure below:

Bokeh Plot
Bokeh Plot
Figure 5. Left: Pie chart for the top10 languages used. Right: Log-scaled bar chart for the top 20 languages.

In Figure 5, English dominates the language-distribution by almost half of the total amount, and the one following is Japanese with nearly 1/3 of the total number. After French and Spanish with comparable percentages, the rest of the languages are Undetermined and some other relatively non-common languages such as Portuguese and German, etc.  In the bar chart on the right-hand side, the numbers are scaled by taking the log value, and the detailed rank information for more languages are shown.

But how about the detailed language distribution inside of a specific gaming community? What if we want to know which language has been used the most often, say, in Fortnite? How much more or less is it compares to other gaming communities? Figure 6 answers these questions. Again, you can hover your mouse over the plot to check the value for each game-language combination.

Gaming Community Languages
Figure 6. Advanced pie plot mimicking Burtin’s Antibiotics plot.

By inspecting Figure 6, one can tell that even though English is the most often used language for most of the games, for Apex Legends, Japanese is the most frequently used language over English around 300 tweets. (If you hover the mouse over, you will see the number 7291 for Japanese and 6885 for English. ) Similarly, Hearthstone seems more prevalent in Spanish-speaking regions than French and Japanese since it is the second often used language for Hearthstone. 

Time series Analysis

One of the critical features of streaming tweets in real-time is time-series analysis. We can count all the mentions for each game during a time window, let’s say 1 hour, then shift this window throughout the 48 hours streaming duration, to visualize the time-varying mentioning frequencies for different games. I picked the top-4 most popular games on Twitter (Figure 3) and plotted their hourly mentioning frequency in Figure 7:

Bokeh Plot
Figure 7. Mentioning frequency averaged hourly for 48 hours.

The stream started at 1 am, May 9th, and ended at 1 am May 11th, as shown in Figure 7. We know that Fortnite is the most popular game (from Figure 4), so it is not surprising to see Fortnite has the highest frequency at all time. However, what is not expected is the peak for Fortnite around 8 am May 9th, while the frequencies for all three other games are going down. Based on what we have learned so far, it is not difficult to figure it out: from Figure 6 we know that the second most frequently used language for Fortnite is Japanese, which means there is another massive Fortnite online community on the other side of the earth existing, in Japan. While it was 8 am for the US; the night-time after working had just begun for Japanese Fortnite lovers. (Further investigation shows that there was an official Twitter event going on for Japanese Fortnite community.) Therefore there is a prominent peak shows up around that time stamp. It also makes a certain amount of sense why there is only one downhill at 8 am May 9th, because May 9th was a Friday, people still need to work on that day. And after that, the weekend was there so players can play video games all night! 

Another interesting experiment to do with tweets that have DateTime information is sentiment analysis, which can be crucial for commercialized marketing applications. I use the Natural Language Toolkit(NLTK) to extract the sentiment information from a tweet. After applying sentiment analysis to all the English tweets, the hourly averaged time-series sentiment score plot is shown in Figure 8:

Bokeh Plot
Figure 8. Sentiment scores averaged hourly for 48 hours.

Given the context that sentiment score above 0 means positive and below 0 means negative, interestingly, while Fortnite got mentioned in most of the tweets, it also got relatively more criticism around Friday night (Japanese time). It seems like that official event was not going on quite well. In contrast, Apex Legends was mentioned in more positive tweets later on around 9 am May 10th. And Overwatch and League of Legends were doing a good job maintaining a slightly positive sentiment score consistently. 

Conlusion

Twitter is an excellent resource not only for sharing our daily lives but also for data gathering and social-event analysis. In this post, I talked about how to use the pre-stored Twitter data to do some fundamental Exploratory Data Analysis. Specifically:

  1. Explicitly visualized ranking for eight hand-choosing video games. Fortnite is the most popular video game on Twitter. There are 90570 out of 165910 (54.59%) tweets mentioned Fortnite.
  2. language-distribution for these video games was extensively inspected. English (49.17%) and Japanese  (27.60%) dominate the Twitter community for these games.
  3. Mentioning frequency and sentiment scores are plotted in the time domain with an hourly-average window. Even though Fortnite has been mentioned more often, people were happier with Apex Legends and two other games with higher averaged sentiment scores.

Now we have a better understanding of the entire global video-game community, but only a tip of the iceberg is revealed. In the next post, I will explain how to utilize the geographical information contained in the tweets to build an interactive map application to reinforce our data analysis along with some interesting network analysis.

You can find the code here for plotting all the figures shown in this post. And welcome to reach out to me if you have any questions.

Thanks for reading 🙂

, ,

Building a Relational Database from Twitter Scraping

Introduction

Twitter is one of the largest social media nowadays. The amount of information hiding in tons of tweets can be enormous, and a tremendous amount of useful information can be extracted if approached correctly. Knowing how to search and manage data efficiently is especially crucial for a data scientist. Before doing any analysis, the data to be analyzed should be stored in a database that is easy to work with. In this post, I will talk about how to scrape data from millions of tweets posted in 48 hours using Twitter API and a Python package Tweepy. Followed by how to save the data into a PostgreSQL or SQLite database automatically for better management. Lastly, I will talk about how to add constraints to the relational-database to make it well-structured. Once the data is acquired and stored in good shape, the down-stream work can be handled more efficiently.

As a video-game enthusiast, I will take advantage of these data science tools to investigate the current trend of the most popular video games through scraping twitter data. Such as figuring out the distribution of the population of each video-game community globally, creating a geospatial map that displays which region has the highest live activity for each game, plot the sentiment score of tweets towards each video game varying along with different timestamps, etc. All these information behind the massive twitter data can be used for multiple potential applications to predict the future of various video games.

Scrape Twitter Data

Unlike scraping other websites where hard-coding is necessary to extract information from specific sections or divs on a webpage, by summarizing all information related to a particular tweet into a JSON file, Twitter APIs make data extraction a lot easier. Before exploring Twitter data, let’s first take a look at how Twitter API works.

Twitter APIs includes Searching API, Ads API, and Streaming API. Searching API allows us to search for specific tweets during the past week. Ads API provides information about the advertisements on Twitter. And the Streaming API provides functionalities that will enable us to collect tweets in real-time with:

  1. Specific keywords, user ID, or locations, called filtering.
  2. 1% of the total tweets randomly sampled, called sampling. 

And I only use Streaming API which can be set up by Python package Tweepy easily. The code showing below sets up the stream connection and can be executed through a command-line tool:

# -*- coding: utf-8 -*-
"""
This script sets up tweepy and hears twitter streams
@author: Shihao Ran
         STIM Laboratory
"""
from tweepy import OAuthHandler
from tweepy import API
from tweepy import Stream
from slistener import SListener
from urllib3.exceptions import ProtocolError
# consumer key authentication
consumer_key = 'YOUR CONSUMER KEY'
consumer_secret = 'YOUR CONSUMER SECRET'
auth = OAuthHandler(consumer_key, consumer_secret)
# access key authentication
access_token = 'YOUR ACCESS TOKEN'
access_token_secret = 'YOUR TOKEN SECRET'
auth.set_access_token(access_token, access_token_secret)
# set up the API with the authentication handler
api = API(auth)
# set up words to hear
keywords_to_hear = ['#GrandTheftAutoV',
                    '#LeagueOfLegends',
                    '#Fortnite',
                    '#dota2',
                    '#CSGO',
                    '#ApexLegends',
                    '#Hearthstone',
                    '#overwatch']
# instantiate the SListener object
listen = SListener(api)
# instantiate the stream object
stream = Stream(auth, listen)
# begin collecting data
while True:
    # maintian connection unless interrupted
    try:
        stream.filter(track=keywords_to_hear)
    # reconnect automantically if error arise
    # due to unstable network connection
    except (ProtocolError, AttributeError):
        continue

 

As shown in the code, import the tweepy package first, then authenticate Twitter API with the consumer key-secrete pair as well as the access token-secrete pair of the account that will be using to stream the tweets. Next, specify the keywords that you want to capture in the tweets. In this case, the hashtags for eight video games. Finally, keep the stream.filter() command in a while True loop to keep the streaming connected unless interrupted manually. The except sentence handles poor network connections from time to time, which may cause disconnections between our workstation and Twitter API, and reconnects the streaming automatically if an error pops up. Before launching the streaming, specify a SListener object, which informs tweepy how to handle the incoming tweets. And the code for SListener is shown below:

# -*- coding: utf-8 -*-
"""
this script defines the streaming behavior passed to tweepy
@author: Shihao Ran
         STIM Laboratory
"""
# import packages
from tweepy.streaming import StreamListener
import json
import time
import sys
# inherit from StreamListener class
class SListener(StreamListener):
    def __init__(self, api = None, fprefix = 'streamer'):
        # define the filename with time as prefix
        self.api = api or API()
        self.counter = 0
        self.fprefix = fprefix
        self.output  = open('%s_%s.json' % (self.fprefix, time.strftime('%Y%m%d-%H%M%S')), 'w')
    def on_data(self, data):
        if  'in_reply_to_status' in data:
            self.on_status(data)
        elif 'delete' in data:
            delete = json.loads(data)['delete']['status']
            if self.on_delete(delete['id'], delete['user_id']) is False:
                return False
        elif 'limit' in data:
            if self.on_limit(json.loads(data)['limit']['track']) is False:
                return False
        elif 'warning' in data:
            warning = json.loads(data)['warnings']
            print("WARNING: %s" % warning['message'])
            return
    def on_status(self, status):
        # if the number of tweets reaches 20000
        # create a new file
        self.output.write(status)
        self.counter += 1
        if self.counter >= 20000:
            self.output.close()
            self.output  = open('%s_%s.json' % (self.fprefix, time.strftime('%Y%m%d-%H%M%S')), 'w')
            self.counter = 0
        return
    def on_delete(self, status_id, user_id):
        print("Delete notice")
        return

Twitter Json File

Now let’s take a look at how Twitter API passes information about a specific tweet into our streamed files. Once the streaming pipeline is established, the code provided above will automatically capture every single tweet that contains any of the keywords specified. Take one of my tweets for example:

Since this tweet has a user mention of @LeagueOfLegends, and I did run the program before I posted this tweet, it was successfully captured by our program and is saved as a JSON file into my computer.

If we open the output file by a notepad, the tweet above looks like something shown here:

{'created_at': 'Sun May 12 16:03:53 +0000 2019',
 'id': 1127605322581987329,
 'id_str': '1127605322581987329',
 'text': 'I LOVE LEAGUE OF LEGENDS! @LeagueOfLegends',
 'source': 'Twitter Web Client',
 'truncated': False,
 'in_reply_to_status_id': None,
 'in_reply_to_status_id_str': None,
 'in_reply_to_user_id': None,
 'in_reply_to_user_id_str': None,
 'in_reply_to_screen_name': None,
 'user': {'id': 4790635572,
          'id_str': '4790635572',
          'name': 'Shihao Ran',
          'screen_name': 'shihao1007',
          'location': 'Houston, TX',
          'url': 'https://shihaojran.com',
          'description': 'PhD :D',
          'translator_type': 'none',
          'protected': False,
          'verified': False,
          'followers_count': 4,
          'friends_count': 43,
          'listed_count': 0,
          'favourites_count': 126,
          'statuses_count': 3,
          'created_at': 'Wed Jan 20 22:54:19 +0000 2016',
          'utc_offset': None,
          'time_zone': None,
          'geo_enabled': False,
          'lang': 'en',
          'contributors_enabled': False,
          'is_translator': False,
          'profile_background_color': 'F5F8FA',
          'profile_background_image_url': '',
          'profile_background_image_url_https': '',
          'profile_background_tile': False,
          'profile_link_color': '1DA1F2',
          'profile_sidebar_border_color': 'C0DEED',
          'profile_sidebar_fill_color': 'DDEEF6',
          'profile_text_color': '333333',
          'profile_use_background_image': True,
          'profile_image_url': 'http://pbs.twimg.com/profile_images/1115448619878973440/Chg2Uabn_normal.jpg',
          'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1115448619878973440/Chg2Uabn_normal.jpg',
          'profile_banner_url': 'https://pbs.twimg.com/profile_banners/4790635572/1554778651',
          'default_profile': True,
          'default_profile_image': False,
          'following': None,
          'follow_request_sent': None,
          'notifications': None},
 'geo': None,
 'coordinates': None,
 'place': None,
 'contributors': None,
 'is_quote_status': False,
 'quote_count': 0,
 'reply_count': 0,
 'retweet_count': 0,
 'favorite_count': 0,
 'entities': {'hashtags': [],
              'urls': [],
              'user_mentions': [{'screen_name': 'LeagueOfLegends',
                                 'name': 'League of Legends',
                                 'id': 577401044,
                                 'id_str': '577401044',
                                 'indices': [26, 42]}],
              'symbols': []},
 'favorited': False,
 'retweeted': False,
 'filter_level': 'low',
 'lang': 'en',
 'timestamp_ms': '1557677033454'}

 

The JSON file is essentially a list of key: value pairs that contain information about every field of that single tweet, such as the time when the tweet was created, the text of that tweet, the user who sent that tweet, etc. In Python, these key:value pairs can be conveniently stored in dictionaries. From there, we can use Pandas to convert a dictionary into a Pandas DataFrame without manually specify the name of each column. Then, we can use Sqlalchemy to create our database and convert all the records in the DataFrame into a table inside our database. The complete workflow is shown in Figure 1.

Figure 1. Workflow for creating a relational database from scraping twitter data.
Figure 1. Workflow for creating a relational database from scraping twitter data.

If you noticed, the JSON file could be considered as a dictionary of dictionaries since some values in the top-level dictionary is also a dictionary, 'user' for example. In fact, if one tweet contains geographical information such as the place where the tweet as created, or if the tweet were just a retweet or quoted tweet, the JSON file would contain more complex fields that contain the corresponding information. While Pandas DataFrame can handle a dictionary as a row element, databases can not. Therefore, one needs to add some tweaks when converting the raw JSON file into Pandas DataFrames, that is, to save these children dictionaries as individual tables into the database, and connect them to the parent table through some constraints such as the common field tweet_id (More on later).

Examining and extracting useful information from a large JSON file can be tedious and varies from application to application. To create a new database and store all these DataFrames as individual tables, one can use SqlAlchemy in a Python environment. SqlAlchemy also allows creating tables without specifying column names and column types manually for each table if it is created from a Pandas DataFrame, which can be a life-savior if there are lots of tables and lots of columns to be inserted into a database.

Once the database is installed in your workstation, you can create a new database using SqlAlchemy. Here I will use Postgresql as an example. As shown in the following code, the first thing is to create an engine that is used to connect to the database by specifying the dialect, username, password, host, and port of the database.

# create a new database
engine = create_engine('postgresql://username:password@host:port')
# connect to the database
con = engine.connect()
# end open transaction
con.execute('commit')
# create new databse my_databse
con.execute('create database my_databse')
# close the connection
con.close()

Then, make a connection to the default database. Since the connection is under a transaction with the default database right now (Sqlalchemy always runs queries during a transaction), end the open transaction by calling con.execute('commit'). Then create the new database followed by closing the connection. Once the database is created, the last step would be creating tables from the DataFrames, which can be easily handled by Pandas Pandas.DataFrame.to_sql() function.

You can refer to my code of saving JSON files of raw tweets into tables in a Postgresql database here.

Building a relational database

For my video-game geospatial map application, I have streamed more than 160,000 tweets within 48 hours with 8 keywords correspond to the top 8 most popular video games (rank information provided by Twitch.com). By running the code linked above, the tweets are loaded into a database.

Cool, now all streamed tweets stored in a database that has multiple tables corresponding to different fields of the tweets, which look like this:

Figure 2. Existing tables in our database
Figure 2. Existing tables in our database

In Figure 2, all the tables are presented as one individual entity, they do not have any explicit relationship between them, nor do they have their primary keys or any foreign keys specified. While this database can be used without any defect, there is A LOT of redundant information hiding in most of the tables.

Take tweet_user table as an example: what would happen if we run the following query through pgadmin 4?

This SQL query asks for the number of distinct tweet_id in base_tweets table, as well as the number of distinct id in tweet_user table. It essentially counts the number of non-duplicate tweets and different Twitter users in our database. Note that there is a column called tweet_id in every table, so that the base_tweets table and tweet_user table have the same amount of records. However, a Twitter user might tweet more than one tweets during the streaming process.  The number of distinct users should be less than the total number of tweets. And the query results is shown in Figure 3.

Figure 3. The query results
Figure 3. The query results

Well, that is quite interesting. The number of total tweets streamed is almost twice the number of users, from which we can draw a bulk conclusion that in the gaming community, by average, people usually tweet two tweets per day. While finding out this side information is fun, it also informs us that nearly half of the records in the tweet_user table is redundant. But before deleting all the duplicate rows, save the mapping information (between a specific tweet to its user) to the base_tweets table for referring between these two tables later. To add a new column to a table with values from another table, execute the following query:

Now there would be a new column called twitter_user_id in base_tweets table, which can be used later as a foreign key to link itself to twitter_user table. To delete the duplicates in twitter_user table, we can use the following query:

In the query above, by specifying a.tweet_id < b.tweet_id, the statement only keeps the record with the highest tweet_id and all the records with lower tweet_id are removed.

Now all the duplicated id in the tweet_user table is removed thus it only contains unique values. We can now use it as the primary key of this table and be referenced in base_tweets table as a foreign key to establish the connection. To set it as the primary key of tweet_user table and foreign key of base_tweets table, execute the following query:

Now the “relationship” between these two tables are established, that is, extra information about the user who sent the tweet in the base_tweets table can now be provided by tweet_user table by referring the tweet_user_id column. Likewise, repeat this process for other tables reduces data redundancy. For those tables that do not have their distinct primary key, just use the tweet_id column.

After removing duplicate records, cutting off unnecessary columns, and adding constraints to all the tables, our database should look like this:

Figure 4. Relational database diagram showing all columns.
Figure 4. Relational database diagram showing all columns.

And it can also be presented with cleaner form as shown here:

Figure 5. Relational database diagram showing only primary keys.
Figure 5. Relational database diagram showing only primary keys.
Figure 6. Relational database diagram showing only relationships.
Figure 6. Relational database diagram showing only relationships.

Conlusion

Twitter is an excellent resource for us data scientists in terms of all kinds of data mining needs. Meanwhile, it can also be overwhelming to deal with all the complex JSON fields returned by Twitter API. How to collect and manage the data from tons of tweets efficiently can be crucial and indispensable.

As a summary, in this post, I talked about:

  1. Use tweepy to set up a streaming pipeline with Twitter Stream API to collect tweets with specific keywords, user ID, or location.
  2. Use pandas and sqlalchemy to load JSON files of raw tweets and save them into a Postgres database.
  3. Use SQL queries to build a relational database and minimize the data-redundancy.

So far, all these works are just data preprocessing, in the next post, I will talk about some exciting work that utilizes all the information gathered and do some fundamental exploratory data analysis based on these data.

Again, the code used for this post can be found here.

Thanks for reading 🙂 

, , , ,

Deep Learning with Phase Retrieval

Introduction

I spent the second year and half of the third year during my Ph.D. program to develop Chemical Holographical Imaging System (CHIS). It is designed to acquire not only the intensity of the light (electromagnetic field) but also the phase information, which is usually lost when using a standard camera or detector to collect the photons. I will write another blog to explain the mechanism behind the scene. Here, I will talk about the downstream work, which happens after the phase of the field is reconstructed.

To characterize and prove the concept, we implemented a forward model and an inverse model for CHIS, simulating the process of acquiring a phase-sensitive measurement and reconstructing the sample, respectively. Here I will only talk about the inverse model, where machine learning algorithms are applied.

We use the Mie scattering model to simulate a micro-sphere whose radius is close or longer than the wavelength of the incident light. Mie scattering is the light scattered when electromagnetic wave interacts with uniform isotropic particles (spheres) in a uniform isotropic dielectric infinite medium. Mie theory is a rigorous solution to Maxwell equations that describes the propagation of electromagnetic radiation by spherical scatterers. It has been widely applied in the characterization of atmospheric absorption and cellular structure detection in spectroscopic imaging of biological tissues.

According to the superposition principle, any form of propagating electromagnetic field is equivalent to a combination of multiple plane waves. To simplify the model, the incident light going through a condenser with a low numerical aperture is considered as a plane wave. While the forward model of CHIS can be described by Mie scattering, the inverse model is equivalent to solving the inverse Mie problem.

Here we first examine the forward Mie scattering. The scattered field for a single plane wave produced by a sphere placed at the origin with radius a can be written as:

\textbf{E}_s(\textbf{p}) = \sum _{l = 0} ^ \infty B_l(\lambda, n, a) h_l^{(1)} (kr)P_l(cos\theta)

where h_l^{(1)}(\cdot) is the order-l spherical Hankel function. The coefficients B that couple the internal and external fields are given by:
B(\lambda, n, a) = (2l+1)i^l\frac{j_l(ka)j_l^{‘}(kna)n-j_l(kna)j_l^{‘}(k)}{j_l(kna)h_l^{(1)}(ka)-h_l^{(1)’}(ka)-h_l^{(1)}(ka)j_l^{‘}(kna)n}

where j_l^{‘}(\cdot) is the first derivative of the spherical Bessel function of the first kind. And h_l^{(1)’}(\cdot) is the first derivative of the spherical Hankel function.

If we take a closer look at these two equations, it is not difficult to notice that the material properties of the sphere (radius a and refractive index n) are only embedded in the B coefficient. In other words, to build an inverse model of the system, our first step is to extract the B coefficient from the field \textbf{E}_s.

Since the Hankel function and its derivatives can be derived from the Bessel functions of the first kind, the Bessel functions of the second kind, and their derivatives, the B coefficient behaves similarly to Bessel functions. In Figure 1, B coefficients of different scattered fields scattered by different spheres are plotted (only real part is shown).

Figure 1. B Coefficients of the scattered field with different sphere radius
Figure 1. B Coefficients of the scattered field with different sphere radius

The animation shows how the B coefficient changes with a \in [1, 10] \mu m. You can run this Jupyter Notebook demo to examine the property of B coefficient with Bokeh interactive plot implementation. 

The point is, the B coefficient trends to converge to zero after going up and down along increasing order. In other words, it is highly sparse and fluctuating, which makes a lot of sense: because the incident light is scattered by a sphere, the scattered field around the surface of the sphere would have the highest intensity. Then the intensity goes up and down due to scattering and interference. When the scattered field travels further away from the sphere, its intensity converges to zero eventually (scattered into broader space). In fact, on top of the sparsity of the B coefficient, the rest of the equation is highly sparse and fluctuating as well. If we consider the scattered field as a system of linear equations where the B matrix is the input and the rest of the equation is the transferring matrix and the field is the output, we can get the following linear system: Ax = b where A is the scatter matrix, A = h_l^{(1)} (kr)P_l(cos \theta) and  x is the B coefficient and b is the field E_s. If we calculate the condition number of the linear system, it can go up to 10^{21}.

In numerical analysis, the condition number with respect to an argument of a function stands for the amount of change to the output with respect to a small shift in the input argument. A low condition number means the system is well-conditioned thus more robust to the disturbance in the input. In contrast, a high condition number means that the system is sensitive to disturbance. A small change in the input could cause a dramatic change to the output hence an ill-conditioned system. A linear system with condition number \kappa = 10^k means besides the arithmetic error during the calculation, another k digits of loss of accuracy would be induced.

In our case, if we apply traditional analytical methods such as Pseudo-inverse or Gaussian elimination to solve the inversed linear system to get B coefficient from E_s (which is our measurement), the ill-conditioned property makes the recovery with noise impossible since a tiny little bit of noise in the measurement would result in massive change in the calculation of B. Another way to think about it is the error is the noise multiplied by the condition number.

Deep Learning Approach

One pathway to the solution is broken does not mean the answer is unreachable. Before we jump into machine learning, let us take a look at some of the advantages and disadvantages of different strategies:

  1. Analytical solutions are mathematical grounded. We can break down the solution into pieces and prove each of part of it rigorously. We know what we are doing and what should we expect.
  2. Analytical solutions are universal and scalable. Top level applications are grounded with similar fundamental theories, the analytical solution (if we have one) of our system can be applied to any other similar imaging systems.
  3. Analytical solutions are one-time work. They can be applied directly once the strategy is finalized whereas machine learning methods are not. A different training set has to be provided if we want to use the model for another system even if there is only a little difference, for accuracy.
  4. Machine learning is a black box, which is well-known in the community. We do not have the detailed information of what our models are doing when they give the prediction, either do we know what to expect when we are trying to tweak some fine gears (not excluding the ability to judge if the model has a high bias or high variance, of course).

At this point you might want to ask, seems like there must be better choices than using machine learning algorithms to solve the ill-conditioned inverse problem, why don’t you try something else? The answer is Yes, we did try something else. And that something is called Compressive Sensing, which is specialized for sparse reconstruction and convex optimization. The short story is 1. the linear system we have is not a convex problem, 2. accuracy of sparse reconstruction algorithms seems to suffer from the synthetic noise, 3. they all require training somehow.

Look over all the pros and cons of all methods available, we stick with the machine learning (ANN specifically, more on later) algorithms despite all the disadvantages they might have: for proof of concept, we believe a better accuracy should be weighted higher than scalability and the ability to be generalized; the training process can be optimized in various ways such as feature selection and smart sampling.

Convolutional Neural Network

Even though Convolutional Neural Networks (CNNs) were invented during the 1980s, they were not so popular until the GPU implementation was done and generalized at the early 2010s. They are mighty in terms of image processing with the function of “convolving” the images to extract higher dimensional features. One would normally take CNN as the first choice to solve the problem (at least we did). The structure of the CNN we applied to our system is shown in Figure 2:

Figure 2. CNN Structure
Figure 2. CNN Structure

As shown in Figure 2, a convolution and max-pooling based set of layers are adopted as the basic structure. Three pairs of convolutional layer and max-pooling layer are cascaded to enable the extraction of higher level abstract features from the scattering images. The number of filters for each convolutional layer is 128, 64, and 32, respectively. The window sizes are set to 3\times 3 and 2\times 2 for convolutional layers and the sequential max-pooling layers. A fully connected layer with 128 units is attached to the last max pooling layer followed by the output layer. Given a complex field as the input for the trained CNN model, the final output is represented by the three units in the output layer corresponding to the radius a, the refractive index \eta, and the attenuation coefficient \kappa, respectively.

Since we have the forward model implemented for the system, one of the advantages is that we do not need to worry about the amount of data we could use for training. However, the trade-off between model structural complexity and training time cost is not neglectable. The three-layer structure is a result of leveraging accuracy and time cost. 

CNN performance

To ensure the CNN is trained with low bias and low variance, 8000 phase-reconstructed images with 640 \times 640 spatial resolution are generated by the forward model with different material properties for the simulated spheres. Each of the feature vectors has 20 entries uniformly distributed within their predefined range. Specifically, the radius a of the sphere is ranging in [1, 2]\mu m, the refractive index \eta is varying in the range [1.1, 2.0], and the attenuation coefficient \kappa \in [0.01, 0.05]. 

Before training the model, stratified sampling is applied to split the training and testing data set since the overall data set is too large to fit in memory. The sphere radius clusters the 8000 images into 20 subsets. Within each subset, the training set and test set are randomly sampled and shuffled with a 20\% ratio, and all images are cropped to 128 \times 128 resolution during this stage. As a result, there are 6400 and 1600 images in the training and test set, respectively. And the order of the features is randomized for \eta and \kappa but ascending for a.

During the training, the training set is further divided into training and validation set with the same ratio (20% for validation), which results in 5120 images for training and 1280 images for validation. The training process contains 25 epochs (i.e., the training and cross-validation are repeated for 25 times) and the batch size is set to 50. Meanwhile, label scaling is applied to the attenuation coefficient set due to scale differences.

Figure 3. Relative RMSE of (left) complex CNN (with phase) and (right) intensity CNN (without phase)
Figure 3. Relative RMSE of (left) complex CNN (with phase) and (right) intensity CNN (without phase)

Figure 3 shows the performance of the CNN trained on 6400 complex images and tested on the rest of the 1600 images. The plots only show 50 samples uniformly down-sampled from the testing and prediction set by a factor of 32. The relative root-mean-square error (RMSE) \epsilon_R is calculated by:

\epsilon_R = \frac{\sum_{m=1}^M\frac{\sqrt{(y_{pred}^{(m)} – y_{test}^{(m)})^2}}{y_{test}^{(m)}}}{M}

where y_{pred} is the prediction given by the CNN and y_{test} is the groud truth (label) of the corresponding sample. And the total number of testing samples is denoted by M = 1600.

To demonstrate the merits of phase-sensitive measurements achieved by chemical holography, another CNN model is trained with traditional intensity images with an identical structure and the same number of training and testing samples. The intensity images are acquired by calculating the square of the absolute value of the complex images. And the performance of the intensity CNN is shown on the right (Figure 3).

Everything seems beautiful if we stop here: not only the CNN is working with a pretty lovely regression accuracy; also the one trained with complex images (phase reconstructed) is outperforming the one trained with standard intensity images. To generalize our model, or, to demonstrate the robustness of CHIS, we added more experiments to back it up.

Table 1. Relative Root Mean Square Error (RMSE) for CNNs (%)
Table 1. Relative Root Mean Square Error (RMSE) for CNNs (%)

Table 1 shows the performance of complex CNNs and intensity CNNs summarized from 30 repetitive trails. It is clear that the complex CNNs perform better than intensity CNNs, which can be partially explained by the fact that the additional phase information existing in the complex training set is no longer available for the intensity training set (during the transaction from complex to intensity).

Artificial Neural Network

It seems a little bit weird seeing Artificial Neural Network (ANN) being mentioned at this point since, for most of us, CNN is far more advanced and powerful than ANN. We already implemented the model with CNN, why bother even talking about ANN? Well, one of the most useful lessons I learn from my advisor Dr. David Mayerich is that, as a scientist, or an engineer, we often need to calm down and take a step back. Essentially, when we are doing research or trying hard to get things done, it is quite natural for us to be tunnel-visioned. Once we stuck, it is relatively easy to get too extreme and anxious as well. Only if we take a step back and make sure everything basic has been done correctly, the path to move on would show up.

Back to the topic, why we choose ANN? The answer can be as simple as, we don’t need CNN because we can achieve the same performance with ANN, even faster, easier, and better. And I will explain the reasons explicitly and rigorously.

First of all, CNNs are widely applied to image processing to take care of complicated high dimensional feature extraction. The goal is to train the network so it can distinguish features or patterns from a 2-D image, or even a 3-D volume and CNN is extremely good at this. Now let’s consider the following statement:

We have a dataset and a model trained with this dataset, if the model is CNN and it is the best model we tried, then this dataset must contains two (or higher) dimensional images.

This statement is true (at least in most of the cases, welcome to correct me if I am wrong). But how about the converse of the statement:

If we have a dataset that contains two (or higher) dimensional images, then CNN will be the best model for this dataset.

Is this converse statement also true? There are numerous examples where it does not hold, including our case. Consider an image with all the pixel values in the x axis being the same number and only varies along the y axis, then we can “compress” this 2-D image into a 1-D vector with one of the vertical lines without losing any information. Furthermore, if all the pixel values are the same, we can even compress the whole image into a single pixel, and this is where the symmetricity comes into play. In highly symmetrical situations, such as center symmetry, radial symmetry, or even reflectional symmetry, the more symmetrical the original 2-D image is, the more redundant information it has thus, the less spatial diversity encoded in the local pixels. If there is not much or no higher dimensional features to be extracted, a simpler model such as ANN can take care of all the work adequately. 

In our case, assume there is no systematic noise or the noise is low, the whole image will be radially symmetrical if we place the sphere at the center of the image. Hence we can reduce the dimension of our dataset from 2-D images to 1-D vectors to train the model without losing any spatial information.

Once we switched from CNN to ANN using 1-D vectors reduced from the 2-D images, several obvious advantages appear naturally:

  1. Dramatically reduced data size, computational cost, and time cost: if the original 2-D simulation has a N\times N resolution, there are N^2 points to be calculated. Once been reduced to 1-D, the number of calculations decreased into N/2 (one vertical or horizontal line from half of the image). Therefore, the dataset for training and testing for the 1-D ANN model is 2N times smaller than the 2-D case. Previously, our training image resolution after padding is 640\times 640 with a total size of more than 100 GB of data, and now it will be shrunk into 1/1280 of the original size which is no more than 100MB, while without losing any information. This 2N relationship also applies to the amount of computational cost and time cost reduced.
  2. Simpler network model and more sophisticated training: as one of the more fundamental machine learning models, ANNs are more mathematically grounded, especially for single layer ANNs. The weights between layers can be interpreted more concretely. That is, for each pixel in the input 1-D vector, there is a combination of weights (the hidden layer) being optimized to minimize the cost function, including the bias term. The more units in the hidden layer, the more “fit” the model will be (also more chance of overfitting). This point-to-point mapping is less abstract than CNN’s higher dimensional feature extraction mechanism thus, easier to be controlled. With that being said, ANN models can be trained more thoroughly with much less work required compares to CNN models.

Figure 5 demonstrates the radial symmetry of the simulation and the effectiveness of the dimension reduction. As we mentioned before, for the simulation to be radially symmetrical, the sphere must be placed at the center of the field of view. One thing to keep in mind is that this condition is hard to meet in real data. In other words, in the experimental dataset, the error induced by non-symmetry would impact the performance of the proposed pipeline. However, this drawback also applies to other models such as CNNs. Adding more synthetic data for training can be a potential solution.

Figure 5. (a) 2-D Mie scattering simulation in far-field (top) and near-field (bottom). (b) 1-D far-field Mie scattering simulation corresponds to the red dashed line in (a). (c) 1-D near-field Mie-scattering simulation corresponds to the green dashed-line in (a).
Figure 5. (a) 2-D Mie scattering simulation in far-field (top) and near-field (bottom). (b) 1-D far-field Mie scattering simulation corresponds to the red dashed line in (a). (c) 1-D near-field Mie-scattering simulation corresponds to the green dashed-line in (a).

With one-dimensional data, the structure of the model can be simplified dramatically, as shown in Figure 6. As mentioned before, the 1-D dataset only occupies 1/1280 of the memory space compares to the 2-D dataset; hence the bottom neck of memory limit no longer exists. The dimension of each feature is increased to 30 results in 30^3=27000 total number of samples. Each sample contains a 320\times 1 vector as the input for the neural network. 

Figure 6. ANN structure
Figure 6. ANN structure

In terms of fine-tuning ANN model, there are several things to pay attention to:

  1. Unlike CNN where processes like convolution and max-pooling generalize the local differences in the data, ANN takes all the local information and arrange combinations of weights to each of them. As a result, ANNs with more complicated structures are more likely to be struggling with overfitting, which becomes more noticeable when applying deep ANNs to real-life data where lots of noise are presented. In a word, complicated ANNs with higher accuracy are more sensitive to noise than simple ANNs assuming that they are trained without noise. How to leverage on the trade-off between noise sensitivity and accuracy is yet to be determined by the problem to be solved.
  2. Random initialization. To ensure that the number of stochastic processes is minimized while developing a model, each operation involving randomness, such as weights initialization, should be assigned with a seed or random state. 
  3. Activation function. While the most popular activation function being ReLu (Rectified Linear units), to optimize the training process, other activation functions should be considered, too. One thing to keep in mind is the dying ReLu problem. Since ReLu sets everything negative to zero, the gradient can converge towards zero. Once a neuron (unit) hits this state, the corresponding weights will stop getting updated thus so-called dead neuron. In our case, there is a fair chance that the weighted sum can hit negative numbers. Therefore a tanh (hyperbolic tangent) activation function would be a better choice. In fact, the training process does converge significantly faster with tanh function than sigmoid or ReLu when the network has a small number of hidden units. When the network grows deeper or has more hidden units, ReLu would be a better choice. If a lot of units stuck at the edge of sigmoid or tanh functions, vanishing gradients appears which can make the training unstable and more difficult to converge. On the other hand, Dying ReLu would not be a problem here since the number of units is big enough.
ANN Performance

Starting with a simple model as shown in Figure 6, there is only one hidden layer and five hidden units. And the number of parameters of the intensity ANN is shown below.

For complex cases, since the 1-D vector contains both real and complex components, it is split into two channels for the training. The model has the same structure with 1 hidden layer and 5 hidden units. And the number of parameters in the complex model is shown below.

As shown in the tables, the number of parameters only increased by 5 for the complex ANN, which is equal to the number of extra channels added to the input layer. The model is trained with Keras package (GPU implemented) and 400 epochs, which takes around 2 minutes for 19440 training samples (19440 =  27000 \times 10% \times 20% , 10% for test set and 20% for validation set). And the results averaged from 30 repetitive trials are shown in table 2.

Table 2. Relative RMSE for simple ANNs (%)
Table 2. Relative RMSE for simple ANNs (%)

Before we judge the performances of ANNs and CNNs, let’s compare the number of parameters in both models first. Recall that the number of parameters for the complex ANN is 4818, which is not a small number. However, it is just the tip of an iceberg compares to complex CNN, of which the number of parameters is shown below:

Well, there are more than 1.7 \times 10^6 parameters in the CNN model even with max-pooling layers. In contrast, 5\times 10^3 is a small number of parameters. While using no more than 1/1000 of the parameters, ANNs are able to get only several percentages of accuracy loss, which is the point we want to point out here: 

Choosing the correct model is the first priority in machine learning.

If you are not convinced yet, totally understandable. One of the questions could be: the accuracy we are getting with the current ANNs is still worse than the CNN version, which is unwanted if we do not care about resources cost to get the model trained. In other words, what if we simply want to achieve the best accuracy, should we switch back to CNN?

The answer is, not yet. The ANN we currently have is just the simplest version. Instead of turning into CNN again, what we can do is to make our ANN model more complicated, such as adding more hidden layers, or more units at each layer. Such as a deeper ANN shown in Figure 7:

Figure 7. A more complicated ANN with 3 hidden layers and more hidden units in each layer.
Figure 7. A more complicated ANN with 3 hidden layers and more hidden units in each layer.

And the number of parameters of this new ANN is shown below:

Without the max-pooling layer and dropout layer, the number of parameters in the new ANN is significantly increased due to the last fully connected layer. But still, the training process costs less time compares to the training time of a deep CNN (no convolution needed for ANN). In this case, the network was trained with 100 epochs in 13 minutes with an Nvidia GeForce GTX 970 GPU. And its performance is shown in Table 3.

Table 2. Relative RMSE for deep ANNs (%)
Table 2. Relative RMSE for deep ANNs (%)

In this single experiment, ANN trained with phase-reconstructed 1-D data outperforms CNN trained with 2-D images, despite the comparable numbers for the intensity ANN accuracy, which in return demonstrates the merits of phase retrieval.

Conclusion

Machine learning is a profound and sophisticated discipline that should be carefully considered and examined before implementing it. ANN seems weaker than CNN at first glance, or at least the author was having some trouble giving up CNN until the ANN results were carried out.

In other words, model selection should be placed at the top with priority before taking any action. Only after taking every factor related to a specific problem into account (even considering changing the strategy of data generation), should the model be determined.

There might be error or confusion exist in this blog, feel free to correct me if I am wrong or let me know if you have any questions.

At last, the code for 

  • Generating the Mie scattering data can be found here.
  • Training and evaluating the CNN model can be found here. 
  • Training and evaluating the ANN model can be found here. 

Thanks for reading 🙂