Archiv der Kategorie: General programming

Tomcat & Meecrowave on a server: Slow startup

What: Using Meecrowave on a server without large delays on startup
Why: Faster startup during development
How: Use a faster entropy source

Background

Meecrowave uses Tomcat, which uses a SecureRandom instance for session ids. On a server, the underlying entropy source can run short resulting in large delays until the application endpoints are scanned and enabled.

Details

See the tomcat how to.

Solution

Add the following line to the JAVA_OPTS in meecrowave.sh:

1
-Djava.security.egd=file:/dev/./urandom

Flashing christmas poems

What: Using a raspberry pi and some LEDs to send christmas poems in morse code
Why: Controlling LEDs with raspberry
How: Using RPi.GPIO library, some simple LEDs and a python script

Morse code

A basic morse code library for python can be found at github. Essentially, it maps lower case characters (no special ones) to a sequence of morse symbols.
It is a python dictionary with the characters as keys:

You can use it in any python program:

Translate text to LED signals

Using two LEDs, one can encode the dit (short symbol) and the other one the dah (long symbol) in morse code. You need additional breaks between characters and words. These can be encoded by not flashing any LED.

Putting it together with the morse alphabet and simple logic for processing arbitrary text (note: the script below does not check, if all characters used are present in the alphabet; just use lower case ones and comma and dot):

You can start it with (assume you saved the above code in a file called morseflash.py and assigned execution rights):

See it in action:

Simple bag of words

What: Using bag of words to categorize text
Why: Build your own chatbot or classify documents
How: Using scikit-learn and pandas

Introduction

Bag of words is a simple classification approach which looks at the occurrence of (key) words in different classes of documents (the bags). The document which should be classified is assigned to the class, where the best mach is found between the document words and the words within the matching bag.

scikit-learn is a python machine learning library with a very nice concept for handling data from preprocessing to model building: Pipelines.

pandas is a python library which helps storing data in table like objects. It makes the handling of data within python much easier.

The following is inspired by the scikit-learn documentation.

Code

For bag of words, a text has to be tokenized, the words have to be stemmed and a classification has to be build. nltk is used for text processing. The used SnowballStemmer is also able to handle german as long as the german module is downloaded. If you don’t mind the space, you can download all nltk data with:

The code can be tested via the following snippet, which can be embedded as self test in the same script, where the ModelBuilder class is defined.

Instead of english, you can also use ‚german‘ as language but you need different test data. Please note, this is a simple example. For a real world use case you need more categories and examples.

The classifier can output instead of the class probabilities for classes, which may help with determining the quality of the classification in case of data which was not included in the model train data.

Usage

Create your test data or read it from file into a pandas data frame and build the model:

Once this is done, use it to classify unknown documents:

Docker on Raspbian

What: Getting Docker running without hassle on raspberry3
Why: Using Docker images on raspberry
How: Using arm version of docker and standard apt-functionality

This is an extract from the docker documentation, which worked for me on a raspberry3 with raspbian jessie.

Install requirements

Prepare installation

Install

Test

Update Eclipse (from Neon to Oxygen)

What: Updating Eclipse without new installation
Why: Beeing up to date
How: Using update site and Oomph-Settings

Setting up Oomph

The first step is to tell Oomph, which version of Eclipse should be used. Select from the menu: Navigate Open SetupInstallation.

A new tab should open with the installation object. Select it and open properties view. Change the product version of Eclipse in the drop down menu to Oxygen.

Adding Update site for oxygen

The second step involves adding the Oomph update site. Select from the menu: WindowPreferences and open Install/UpdateAvailable Software Sites. Add a new site with the oxygen repository (http://download.eclipse.org/releases/oxygen/).

Click Apply and Close.

Update

Update via the standard Eclipse update mechanism. Select from the menu: HelpCheck for Updates.

Perform the update as normal and restart. The Eclipse version starting should now be Oxygen.

Mosquittos on the couch

What: Put mosquitto messages to the couch
Why: Using mosquitto broker as relay between your IoT devices and a database backend
How: Use mosquitto, curl and some linux magic

Requirements

You need couchdb (assumming it runs locally on port 5984 for this example) and mosquitto (also assuming it runs locally for this example). If you dont have it on your system, have a look at my other blog entry. Additionally, you need curl and the bash.

Set up a simple publisher

Create a simple script test.sh, which will publish messages periodically to the mosquitto broker under the topic test:

Change the permission for this script in such a way that you can execut it.

Create a test database

Connect mosquitto and couch via curl

Mosquitto and couchdb can be connected via a simple shell pipe:

Note: You could think about piping mosquitto directly to couch, if your message is already a json string. Something like this:

This will not work, because curl starts reading the input after it is complete (after the stream from mosquitto is closed). You need the while read line construction like shown above.

Run the test publisher script and verify results

Run the script:

Wait some seconds. Now query the database and you should have some documents there:

The result should look like:

Useful ansible roles

What: Using Ansible to setup a development system with Couchdb and Docker
Why
: Having a phoenix like dev setup
How: Using Ansible and some simple roles to provision the system

Requirements

You need a system, where Ansible is installed on. In case you don’t have it at hand, you can use the following Vagrantfile to set it up:

Preparing the playbook

Lets set up a simple playbook. Because something is installed, become is needed to install as root. Create a file called playbook.yml with the following content (there are more roles in the repository, but these should be enough for the beginning):

The roles

Clone the following git repository and change to the directory usefulansibleroles. Copy the roles-folder next to your playbook file.

Note: The install-couch role will install couchdb via docker (see https://hub.docker.com/r/klaemo/couchdb/) in version 2.0. Docker will be setup to restart couchdb at every boot.

Run playbook

Run the playbook. You can use a hosts file at /etc/ansible/hosts or run it locally:

Test

Connect to the provisioned machine. The following commands should give you correct results:

Using weather forecast data of DWD for Europe

What: Extracting weather forecast data from DWD grib2 files with Java
Why: Using weather forecasts in your (Java) applications
How: Download the data from DWD and using NetCDF for parsing

Getting the data

The data (in the following, the 3-day forecast is used) is freely available via ftp from here. You have to register with a valid EMail.

There are several different data sets on the server. The interesting one is the ICON model. It contains forecasts with 0.125 and 0.25 degree resolution for wind, temperature, precipitation and more. You find the data under the path /ICON/grib/europe.

There is a list of available content for the ftp server here.

The data is published in 6hour intervalls at 5, 11, 17 and 23 o’clock. Each file is a zipped grib2 file.

For this tutorial, download and unzip the file /ICON/grib/europe/ICON_GDS_europe_reg_0.250×0.250_T_2M_[yyyyMMddhh].grib2.bz2 (replace the date) which contains the temperature 2 meters above ground and unzip it.

Parsing the data

You can find the full example here. Clone the repository, change to weatherforecastdata directory and adapt it to your needs (hard coded file name, …). After you have finished your changes run:

If you want to build it from scratch (it is simple) create a maven project and add the following repository and dependency to your pom.xml:

NetCDF is able to read in the grib files. You can create a new instance of the data file via:

Each file contains dimensions, attributes and variables. The variables depend on the dimensions. For example: The temperature value depends on the latitude and longitude as well as on time and height above ground. The data can be retrieved in arrays, with the shape depending on the corresponding dimensions. Latitude, longitude, time and height above ground are 1-dimensional arrays while the temperature depends on all of them and thus is 4-dimensional.

You can get dimensions, attributes and variables via:

Lets concentrate on the variables. For each variable you can get name, units and dimensions it depends on:

For the example above this will give for the temperature:

There is a variable called Temperature_height_above_ground, which depends on time (there are 79 different values for the time dimension), height_above_ground (with just one different value because we look at the temperature measured at 2m above ground), latitude and longitude with (301/601 different values for the respective dimension).

This is enough information to retreive the data from the file:

Iterating over the temperature values can now be done by iterating over each of the dimensions:

Have fun with the amazing DWD source of weather forecast data.

Setup Raspberry Pi with WLAN and ssh from start

What: Setup Wlan on a fresh raspberry without ethernet cable
Why: Fast and headless setup while still sitting on the couch
How: Use the latest raspbian image, provide wpa_supplicant.conf and ssh file

Download raspbian image

Download the zipped image file from here. I took the raspbian jessie lite.

Unpack it.

Put the image on a SD card

Use a suitable SD card (mine is 8GB) and format it. You can use SDFormatter on Windows for that.

Afterwards, copy the image file to the card. You can use Win32DiskImager for that.

Setup Wlan and ssh

Go to the SD card drive. There should be a file called cmdline.txt.

Create a new file called wpa_supplicant.conf in the same directory like the cmdline.txt and put the following in (Update: The lines changed between Raspbian Jessie and Stretch, see here):

This step is taken from here.

Since december 2016, ssh is by default disabled in raspbian. To enable it, create a new and empty file called ssh in the same directory like the cmdline.txt. See documentation.

Start the raspberry

Put the SD card into the raspberry and start it. The raspberry should now be visible in your network and you should be able to establish a ssh connection via Wlan. For the first raspberry start it may take some time (3 min. in my case) but for further restarts the raspberry was available via ssh within seconds.

Vim forever

What: Installing Vim one every machine you work with
Why: Avoid complicated setup of the favourite IDE each time on a new (virtual) machine
How: Using Ansible to install and pimp Vim

Vim is a great IDE, but with some plugins it is even better. I regularly use and want to have the following plugins available:

Install vim on multiple machines is quite simple with Ansible. See my other blog entry for installing ansible and a first toy example of ansible in action.

Install and pimp Vim

Install Ansible on your system and clone my blog repository. It contains the playbook (vim.yml) and a helper file (installVim.sh).

Note: If you use the playbook to install and pimp Vim on the same machine like ansible is running: Clone to a different location than /tmp/blog. Use /tmp/bloginstall for example.

In the blog/ansible directory you will find the playbook file vim.yml. By default, it is configured to install Vim on a test remote machine (if you are using Vagrant, you can find the Vagrantfile here). If you don’t want this, change the following line and replace ansibletest with your remote machines name.

Afterwards, run the playbook with the following command (password based authentication is used here with the option –ask-pass, see here):

If you want to run it on your local machine use the following instead:

Ansible will produce some output which should end with something like:

If you are a linux user: Done! If you are using Windows and putty, there is one last step to be done to have a nice user experience with power line: Change the font used on your vim terminal. See this blog for the setup. It is done in less than 5 minutes. By default, the Vim instance presented here is using the DejaVu Sans Mono for Powerline font as described in the tutorial.