What: Using bag of words to categorize text
Why: Build your own chatbot or classify documents
How: Using scikit-learn and pandas

Introduction

Bag of words is a simple classification approach which looks at the occurrence of (key) words in different classes of documents (the bags). The document which should be classified is assigned to the class, where the best mach is found between the document words and the words within the matching bag.

scikit-learn is a python machine learning library with a very nice concept for handling data from preprocessing to model building: Pipelines.

pandas is a python library which helps storing data in table like objects. It makes the handling of data within python much easier.

The following is inspired by the scikit-learn documentation.

Code

For bag of words, a text has to be tokenized, the words have to be stemmed and a classification has to be build. nltk is used for text processing. The used SnowballStemmer is also able to handle german as long as the german module is downloaded. If you don’t mind the space, you can download all nltk data with:

sudo python3 -m nltk.downloader all

import nltk
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import SGDClassifier
from sklearn.pipeline import Pipeline
import pandas
import logging
import warnings
# Supress some sklearn warnings
warnings.filterwarnings("ignore", category=FutureWarning)

class ModelBuilder():
    __stemmer=None
    
    def __init__(self, language):
        self.__stemmer=nltk.stem.SnowballStemmer(language)
        logging.basicConfig(format='%(asctime)-15s %(message)s')
        self.__logger=logging.getLogger('ModelBuilder')
    
    # Taken from: https://stackoverflow.com/q/26126442
    def __stem_tokens(self, tokens, stemmer):
        stemmed = []
        for item in tokens:
            stemmed.append(stemmer.stem(item))
        return stemmed

    # Taken from: https://stackoverflow.com/q/26126442
    def __tokenize(self, text):
        tokens = nltk.tokenize.WordPunctTokenizer().tokenize(text)
        stems = stem_tokens(tokens, stemmer)
        return stems
    
    def buildModel(self, data):
        # Taken from: http://scikit-learn.org/stable/auto_examples/model_selection/
        # grid_search_text_feature_extraction.html#
        # sphx-glr-auto-examples-model-selection-grid-search-text-feature-extraction-py
        pipeline = Pipeline([
            ('vect', CountVectorizer()),
            ('tfidf', TfidfTransformer()),
            ('clf', SGDClassifier()),
        ])

        parameters = {
            'vect__max_df': (0.5, 0.75, 1.0),
            'vect__ngram_range': ((1, 1), (1, 2)),  # unigrams or bigrams
            'clf__alpha': (0.00001, 0.000001),
            'clf__penalty': ('l2', 'elasticnet'),
            'clf__max_iter': (1000,),
            'clf__tol': (1e-3,),
            # Modified huber allows getting probabilities for the classes out. 
            # See predict_proba for details
            'clf__loss':('modified_huber',)
        }

        # find the best parameters for both the feature extraction and the
        # classifier
        grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=False)

        grid_search.fit(data.Text, data.Class)
        self.__logger.info("Best score: %0.3f" % grid_search.best_score_)
        self.__logger.info("Best parameters set:")
        best_parameters = grid_search.best_estimator_.get_params()
        for param_name in sorted(parameters.keys()):
            self.__logger.info("\t%s: %r" % (param_name, best_parameters[param_name]))

        return grid_search.best_estimator_

The code can be tested via the following snippet, which can be embedded as self test in the same script, where the ModelBuilder class is defined.

import unittest
 
class TestModelBuilder(unittest.TestCase):
    
    def setUp(self):
        self.__out=ModelBuilder('english')
        
        self.__testdata=pandas.DataFrame(columns=['Text', 'Class'])
        self.__testdata.loc[self.__testdata.shape[0]]=["Hello", "greeting"]
        self.__testdata.loc[self.__testdata.shape[0]]=["Hi", "greeting"]
        self.__testdata.loc[self.__testdata.shape[0]]=["How are you", "greeting"]
        self.__testdata.loc[self.__testdata.shape[0]]=["Bye", "farewell"]
        self.__testdata.loc[self.__testdata.shape[0]]=["Goodbye", "farewell"]
        self.__testdata.loc[self.__testdata.shape[0]]=["See you", "farewell"]
 
    def test_buildModel(self):
        classifier=self.__out.buildModel(data)
        classifier.predict(["See you"])
        self.assertEqual('farewell', classifier.predict(["See you"]))
        self.assertEqual('greeting', classifier.predict(["Hello"]))
 
suite = unittest.TestLoader().loadTestsFromTestCase(TestModelBuilder)
unittest.TextTestRunner().run(suite)

Instead of english, you can also use ‚german‘ as language but you need different test data. Please note, this is a simple example. For a real world use case you need more categories and examples.

The classifier can output instead of the class probabilities for classes, which may help with determining the quality of the classification in case of data which was not included in the model train data.

Usage

Create your test data or read it from file into a pandas data frame and build the model:

classifier=ModelBuilder(<language>).buildModel(<data>)

Once this is done, use it to classify unknown documents:

documentclass=classifier.predict([<text>])
documentclassProbabilities=classifier.predict_proba([<text>])

What: Using Ansible to setup a development system with Couchdb and Docker
Why: Having a phoenix like dev setup
How: Using Ansible and some simple roles to provision the system

Requirements

You need a system, where Ansible is installed on. In case you don’t have it at hand, you can use the following Vagrantfile to set it up:

Vagrant.configure("2") do |config|
  config.vm.box = "debian/jessie64"
  config.vm.synced_folder ".", "/vagrant", type: "virtualbox"

  config.vm.provision "shell", inline: <<-SHELL
    echo "deb http://ppa.launchpad.net/ansible/ansible/ubuntu trusty main" >> /etc/apt/sources.list
    sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 93C4A3FD7BB9C367
    sudo apt-get update
    sudo apt-get install -y ansible
  SHELL
end

Preparing the playbook

Lets set up a simple playbook. Because something is installed, become is needed to install as root. Create a file called playbook.yml with the following content (there are more roles in the repository, but these should be enough for the beginning):

1
2
3
4
5
6
7
8
9
10
11

- name: playbook
  hosts: all
  become: true
  
  roles:
    - install-dockerce
    - install-docker-py
    - install-couch
    - install-java8
    - install-maven
    - install-vim

The roles

Clone the following git repository and change to the directory usefulansibleroles. Copy the roles-folder next to your playbook file.

Note: The install-couch role will install couchdb via docker (see https://hub.docker.com/r/klaemo/couchdb/) in version 2.0. Docker will be setup to restart couchdb at every boot.

Run playbook

Run the playbook. You can use a hosts file at /etc/ansible/hosts or run it locally:

ansible-playbook -i "localhost," -c local playbook.yml

Test

Connect to the provisioned machine. The following commands should give you correct results:

java -version

mvn -version

vim

sudo docker run hello-world

curl http://localhost:5984

What: Extracting weather forecast data from DWD grib2 files with Java
Why: Using weather forecasts in your (Java) applications
How: Download the data from DWD and using NetCDF for parsing

Getting the data

The data (in the following, the 3-day forecast is used) is freely available via ftp from here. You have to register with a valid EMail.

There are several different data sets on the server. The interesting one is the ICON model. It contains forecasts with 0.125 and 0.25 degree resolution for wind, temperature, precipitation and more. You find the data under the path /ICON/grib/europe.

There is a list of available content for the ftp server here.

The data is published in 6hour intervalls at 5, 11, 17 and 23 o’clock. Each file is a zipped grib2 file.

For this tutorial, download and unzip the file /ICON/grib/europe/ICON_GDS_europe_reg_0.250×0.250_T_2M_[yyyyMMddhh].grib2.bz2 (replace the date) which contains the temperature 2 meters above ground and unzip it.

Parsing the data

You can find the full example here. Clone the repository, change to weatherforecastdata directory and adapt it to your needs (hard coded file name, …). After you have finished your changes run:

1

mvn clean package && java -jar target/weatherforecastdata-1.0-SNAPSHOT-jar-with-dependencies.jar

If you want to build it from scratch (it is simple) create a maven project and add the following repository and dependency to your pom.xml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

<repositories>
    <repository>
        <id>artifacts.unidata.ucar.edu</id>
        <url>https://artifacts.unidata.ucar.edu/content/repositories/unidata-releases/</url>
    </repository>
</repositories>
 
<dependencies>
    <dependency>
        <groupId>edu.ucar</groupId>
        <artifactId>netcdfAll</artifactId>
        <version>4.6.8</version>
    </dependency>
</dependencies>

NetCDF is able to read in the grib files. You can create a new instance of the data file via:

1

NetcdfFile dataFile = NetcdfFile.open("filename");

Each file contains dimensions, attributes and variables. The variables depend on the dimensions. For example: The temperature value depends on the latitude and longitude as well as on time and height above ground. The data can be retrieved in arrays, with the shape depending on the corresponding dimensions. Latitude, longitude, time and height above ground are 1-dimensional arrays while the temperature depends on all of them and thus is 4-dimensional.

You can get dimensions, attributes and variables via:

1
2
3

List<Dimension> dimensions = dataFile.getDimensions();
List<Attribute> globalAttributes = dataFile.getGlobalAttributes();
List<Variable> variables = dataFile.getVariables();

Lets concentrate on the variables. For each variable you can get name, units and dimensions it depends on:

1
2
3
4
5

for (Variable variable : variables) {
    System.out.println(variable.getFullName());
    System.out.println(variable.getUnitsString());
    System.out.println(variable.getDimensions());
}

For the example above this will give for the temperature:

Temperature_height_above_ground
K
[time = 79;, height_above_ground = 1;, lat = 301;, lon = 601;]

There is a variable called Temperature_height_above_ground, which depends on time (there are 79 different values for the time dimension), height_above_ground (with just one different value because we look at the temperature measured at 2m above ground), latitude and longitude with (301/601 different values for the respective dimension).

This is enough information to retreive the data from the file:

1
2
3
4
5

ArrayDouble.D1 time = (ArrayDouble.D1) dataFile.findVariable("time").read();
ArrayFloat.D1 height = (ArrayFloat.D1) dataFile.findVariable("height_above_ground").read();
ArrayFloat.D1 lat = (ArrayFloat.D1) dataFile.findVariable("lat").read();
ArrayFloat.D1 lon = (ArrayFloat.D1) dataFile.findVariable("lon").read();
ArrayFloat.D4 temp = (ArrayFloat.D4) dataFile.findVariable("Temperature_height_above_ground").read();

Iterating over the temperature values can now be done by iterating over each of the dimensions:

1
2
3
4
5
6
7
8
9
10
11

for (int timeIndex = 0; timeIndex < time.getShape()[0]; timeIndex++) {
    for (int heightIndex = 0; heightIndex < height.getShape()[0]; heightIndex++) {
        for (int latIndex = 0; latIndex < lat.getShape()[0]; latIndex++) {
            for (int lonIndex = 0; lonIndex < lon.getShape()[0]; lonIndex++) {
                System.out.println(String.format("%f %f %f %f %f", time.get(timeIndex),
                    height.get(heightIndex), lat.get(latIndex), lon.get(lonIndex),
                    temp.get(timeIndex, heightIndex, latIndex, lonIndex)));
            }
	}
    }
}

Have fun with the amazing DWD source of weather forecast data.

What: Installing Vim one every machine you work with
Why: Avoid complicated setup of the favourite IDE each time on a new (virtual) machine
How: Using Ansible to install and pimp Vim

Vim is a great IDE, but with some plugins it is even better. I regularly use and want to have the following plugins available:

• air-line for status information
• syntastic for syntax checking
• Zenburn as color scheme
• vim markdown for markdown
• nerd tree for directory browsing
• ctrl p for search and opening files in the workspace
• Jedi vim for auto completion

Install vim on multiple machines is quite simple with Ansible. See my other blog entry for installing ansible and a first toy example of ansible in action.

Install and pimp Vim

Install Ansible on your system and clone my blog repository. It contains the playbook (vim.yml) and a helper file (installVim.sh).

Note: If you use the playbook to install and pimp Vim on the same machine like ansible is running: Clone to a different location than /tmp/blog. Use /tmp/bloginstall for example.

1

git clone https://github.com/frankdressel/blog.git /tmp/bloginstall

In the blog/ansible directory you will find the playbook file vim.yml. By default, it is configured to install Vim on a test remote machine (if you are using Vagrant, you can find the Vagrantfile here). If you don’t want this, change the following line and replace ansibletest with your remote machines name.

  hosts: ansibletest

  hosts: your_machine_name

Afterwards, run the playbook with the following command (password based authentication is used here with the option –ask-pass, see here):

1
2

cd /tmp/bloginstall/ansible
ansible-playbook vim.yml -u  --ask-pass --ask-become-pass

If you want to run it on your local machine use the following instead:

ansible-playbook -i "localhost," -c local vim.yml

Ansible will produce some output which should end with something like:

1
2

PLAY RECAP *********************************************************************
10.0.2.2                   : ok=17   changed=0    unreachable=0    failed=0

If you are a linux user: Done! If you are using Windows and putty, there is one last step to be done to have a nice user experience with power line: Change the font used on your vim terminal. See this blog for the setup. It is done in less than 5 minutes. By default, the Vim instance presented here is using the DejaVu Sans Mono for Powerline font as described in the tutorial.