Creating vim split window with arbitrary content

What: Create a split window in Vim with arbitrary content
Why: Displaying information easily along text your are editing
How: Use Vimscript

A simple Vimscript script

Create a file test.vim with the following content. It defines a function which creates a new buffer, adds text to it and show it in a split window on the right.

function! test#test()
  let buffernum = bufadd("")
  call bufload(buffernum)
  call setbufline(buffernum, 1, "Hey ho")
  execute "vertical rightbelow sb " . buffernum
endfunc

Use the defined function

There are multiple ways, the function can be used. You can create a plugin (see https://learnvimscriptthehardway.stevelosh.com/chapters/41.html) or call the function from within Vim:

1
2
:source test.vim
:call test#test()

Tomcat & Meecrowave on a server: Slow startup

What: Using Meecrowave on a server without large delays on startup
Why: Faster startup during development
How: Use a faster entropy source

Background

Meecrowave uses Tomcat, which uses a SecureRandom instance for session ids. On a server, the underlying entropy source can run short resulting in large delays until the application endpoints are scanned and enabled.

Details

See the tomcat how to.

Solution

Add the following line to the JAVA_OPTS in meecrowave.sh:

1
-Djava.security.egd=file:/dev/./urandom

Measure loudness with a USB micro on a raspberry pi

What: Measuring loudness with a simple USB microphone on a raspberry pi
Why: Create devices which are activated by a certain noise level
How: Use python and some python libraries to analyse the sound stream

Requirements

The following hardware was used for this setup:

The following os was used on the raspberry pi:

The following python version was used (preinstalled with the corresponding raspbian):

  • Python 2.7 (PyAudio was not working for me with Python3 easily)

Setup

Besides plugging the USB microphone in the corresponding USB port (such a surprise), the following things needs to be done.

Install PortAudio

Install the dependencies:

sudo apt-get install libasound-dev

Download the corresponding version (at the time of writing this is 190600_20161030) and uncompress it:

wget http://www.portaudio.com/archives/pa_stable_v190600_20161030.tgz
tar -xvf pa_stable_v190600_20161030.tgz

Build PortAudio from source:

cd portaudio
./configure && make
sudo make install
sudo ldconfig

Install NumPy, PyAudio and SoundAnalyse

pip install numpy
pip install PyAudio
pip install SoundAnalyse

Code

Below is the code I used for measuring. There are some caveats with measuring the loudness:

  • There are by default some warnings when opening the stream
  • When reading the bytes from the stream, there may be an overflow exception which will kill the script. This can be avoided by the parameter exception_on_overflow = False
  • The parameters for pyaud.open may be different depending on the used microphone. The parameters can be determined by iterating over the devices and use: pyaud.get_device_info_by_index(i)
1
2
3
4
5
6
7
8
9
10
11
12
13
import analyse
import numpy
import pyaudio
 
pyaud = pyaudio.PyAudio()
stream = pyaud.open(format = pyaudio.paInt16,channels = 1,rate=44100,input_device_index=2,input=True)
 
while True:
    raws=stream.read(1024, exception_on_overflow = False)
    samples= numpy.fromstring(raws, dtype=numpy.int16)
    loudness = analyse.loudness(samples)
    if loudness > -15:
        print "Really loud"

Feature engineering in python

What: Generating features for regression models in python based on formulas (like in R)
Why: Model notation in formulas and not in code => Better maintainability
How: Use patsy

Test data

Lets take as example a simple: fitting an exponential to some test data. First, some test data needs to be generated:

import numpy
import pandas

length = 40
linear = numpy.array(range(length))*20
quadratic = numpy.array([e**2 for e in range(length)])
intersect = 10
random = numpy.random.random(length)*40

df = pandas.DataFrame(
    {
        "y":  linear + quadratic + intersect + random,
        "x1": range(length)
    })

The test data and its components look like this:

Feature engineering

Lets assume, we want to fit the data with a linear and quadratic term (surprise, surprise). When using a linear model, you would provide a matrix where each column is one of the features. In our case this would mean writing a small script to fill a length x 3 matrix. Although it is no big deal it involves some lines of code and is not very readable.

With patsy we can write instead something like the following and let the matrices be generated by patsy:

1
2
3
import patsy
formula = "y ~ x1 + numpy.power(x1, 2)"
y, x = patsy.dmatrices(formula, df)

This will result for x in something like:

[[1.000e+00 0.000e+00 0.000e+00]
 [1.000e+00 1.000e+00 1.000e+00]
 [1.000e+00 2.000e+00 4.000e+00]
 [1.000e+00 3.000e+00 9.000e+00]
 [1.000e+00 4.000e+00 1.600e+01]
 ...

Fitting

With sklearn it is easy to do the fit:

1
2
3
4
import sklearn.linear_model
model = sklearn.linear_model.LinearRegression()
model.fit(x, y)
predicted = model.predict(x)

In the end, it looks like:

Convert raw images to jpeg

What: Convert raw camera images (like in CR2 format) to jpeg
Why: Smaller image size and easier handling in other programms
How: Use dcraw and pnmtojpeg in Debian

Installation of tools

1
2
sudo apt-get install dcraw
sudo apt-get install pnmtopng

Small script to iterate over the raw image directory

1
2
3
4
for fn in /home/frank/images/*.CR2; do
    target="${fn/.CR2/.jpeg}"
    dcraw -c "$fn" | pnmtojpeg > "$target"
done

Fly me to the stars II: Explorative data analysis in Jupyter with python

What: Use Dygraphs within Jupyter notebooks
Why: Interactive explorative data analysis
How: Use jupyter_dygraphs for dygraphs plots within Jupyter

Installation

Jupyter_dygraphs can be easily imported directly from github:

1
pip install git+https://github.com/frankdressel/jupyter_dygraphs.git#egg=jupyter_dygraphs

Usage

Import the jupyter_dygraphs module into your notebook:

1
from jupyter_dygraphs import jupyter_dygraphs

Call the _dygraphplot_ function with dictionaries containing the pandas data frame and plot options:

1
2
3
4
5
6
7
8
9
jupyter_dygraphs.dygraphplot({
    'df': pandas.read_csv('https://moduliertersingvogel.de/wp-content/uploads/2018/05/temperatures.csv'),
    'options': {
        'title': 'Title for test plot',
        'xlabel': 'Date',
        'ylabel':
        'Temperature'
    }
})

Example plot

Further information

See github.

Bulk delete in CouchDB

What: Deleting all documents from Couchdb with a single command from the command line without deleting the database/design documents
Why: Truncate the database
How: Python3 and requests

Retrieve all documents

Couchdb has a rest api, which allows the retrieval of all documents from a database. To delete documents, the corresponding id and revision of each document is needed. Further attributes of the document can be ignored.

To retrieve all documents, a simple get request is enough, which will return a json document with an attribute rows which contains a list of ids and revisions of all documents:

import json
import requests

r=requests.get("http://localhost:5984/databasename/_all_docs")
rows=json.loads(r.text)['rows']

Set delete flag

Documents can be deleted from Couchdb by setting the attribute _deleted to true (for some subtleties see: this blog). Lets create the minimal information for deletion for each document:

todelete=[]
for doc in rows:
    todelete.append({"_deleted": True, "_id": doc["id"], "_rev": doc["value"]["rev"]})

Push changes

While all documents can be retrieved from the data base at once, it is also possible to submit multiple documents in one request:

r=requests.post("http://localhost:5984/databasename/_bulk_docs", json={"docs": todelete})

Make it user friendly

To make it a little bit more user friendly, the name of the database can be set as argument and the python script should be called from the command line. In the end it looks like:

#!/usr/bin/env python3
# coding: utf-8
import json
import requests
import sys

database=sys.argv[1]
if len(database)==0:
    sys.exit(1)

r=requests.get("http://localhost:5984/{}/_all_docs".format(database))
rows=json.loads(r.text)['rows']
print(len(rows))

todelete=[]
for doc in rows:
    todelete.append({"_deleted": True, "_id": doc["id"], "_rev": doc["value"]["rev"]})

r=requests.post("http://localhost:5984/{}/_bulk_docs".format(database), json={"docs": todelete})
print(r.status_code)

Have fun in extending the script and use it for maintaining your Couchdb!

Cooking with meecrowave

What: Building microservices in Java(9) painlessly
Why: Small and maintainable services
How: Using meecrowave

Introduction

Creating microservices in Java can be quite complicated. You either do it by yourself using the Java internal HTTP-Server, using one of the many application servers or using one of the integrated frameworks like wildfly-swarm. While the first option doesn’t work well if you want to use something like dependency injection and you have to include all useful libraries like jax-rs by yourself, the second option already contains the most parts of it. You can use for example Glassfish, Wildfly, Websphere or Tomcat (and TomEE). Nevertheless, you rely on heavy application servers and you have to start an instance of these and deploy for each testing (although there exists some integration solutions into IDEs).

The integrated frameworks are sometimes huge, need extended configuration or doesn’t play well with Java9. Partly testing is not as easy as it should be (dependency injection is one of the issues).

Meecrowave on the other hand is a small framework which works well with CDI, JAX-RS and Jackson out of the box which is super easy to set up and performing integration tests is as easy as starting a JUnit test. The following tutorial shows an easy example.

The source code for this tutorial is available here (folder micro).

Note: Although the example runs with Java9 it is not modularized. Some of the dependencies are not yet available as Java9 modules and thus creating this example as a module is outof scope for this tutorial).

Setup

In the following, maven and jdk9 (both for compiling and running) is used.

Add the following dependencies to your pom.xml to include the needed libraries for this example.

1
2
3
	org.apache.meecrowave
	meecrowave-core
	1.2.0

Server

Starting meecrowave is simple: Just start the meecrowave server. All the rest like scanning classes for endpoints, … is done automatically. Create a class with a main method and add the following code:

1
2
3
4
5
6
public static void main(String[] args) {
	try (final Meecrowave meecrowave = new Meecrowave();final Scanner scanner=new Scanner(System.in);) {
		meecrowave.bake();
	    scanner.nextLine();
	}
}

If you start the class, you should see some printout and meecrowave is up and running. To start it from Java9 you have to add –add-modules java.xml.bind as argument to the virtual machine.

Note: The main class is not needed at all for running it outside of ides(at least not on Linux machines) since meecrowave can create a whole distribution package (see below).

You should see output like:

[09:56:57.591][INFO ][           main][.webbeans.config.BeansDeployer] All injection points were validated successfully.
[09:56:57.904][INFO ][           main][apache.cxf.endpoint.ServerImpl] Setting the server's publish address to be /
[09:56:57.959][INFO ][           main][ifecycle.WebContainerLifecycle] OpenWebBeans Container has started, it took [694] ms.
[09:56:58.119][WARN ][           main][na.util.SessionIdGeneratorBase] Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [120] milliseconds.
[09:56:58.164][INFO ][           main][meecrowave.cxf.CxfCdiAutoSetup] REST Application: / -> org.apache.cxf.cdi.DefaultApplication
[09:56:58.164][INFO ][           main][meecrowave.cxf.CxfCdiAutoSetup]      Service URI: /test  -> de.moduliertersingvogel.micro.SimpleEndpoint
[09:56:58.169][INFO ][           main][meecrowave.cxf.CxfCdiAutoSetup]               GET /test/ ->      Response test()

JAX-RS endpoints

You can create arbitrary endpoints based on the jax-rs annotations. Each endpoint needs to be annotated with a Path and scope annotation. The following example defines a simple endpoint returning the string „Hello World“:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
package de.moduliertersingvogel.micro;
 
import javax.enterprise.context.RequestScoped;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.core.Response;
 
@RequestScoped
@Path("test")
public class SimpleEndpoint {
	@GET
	public Response test() {
		return Response.ok().entity("Hello World").build();
	}
}

You can point your browser to http://localhost:8080/test and see the result.

Dependency injection

Dependency injection works like expected. You need a class, which is annotated with the scope and inject it somewhere. Lets test it with a simple object:

1
2
3
4
5
6
7
8
9
10
package de.moduliertersingvogel.micro;
 
import javax.enterprise.context.ApplicationScoped;
 
@ApplicationScoped
public class SimpleObject {
	public boolean callMe() {
		return true;
	}
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
import javax.enterprise.context.RequestScoped;
import javax.inject.Inject;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.Response.Status;
 
@RequestScoped
@Path("test")
public class SimpleEndpoint {
	@Inject
	SimpleObject obj;
 
	@GET
	public Response test() {
		if(obj.callMe()) {
			return Response.ok().entity("Hello World").build();
		}
		return Response.status(Status.BAD_REQUEST).entity("Something went wrong").build();
	}
}

Run your main class and check in yout browser (see above) to see that everything works fine.

Testing

Testing in meecrowave is as simple as writing (annotated) unit tests. In order to get it working, you have to add the following dependencies to your pom.xml (okhttp is used for getting the result from the running microservice):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
	org.apache.meecrowave
	meecrowave-junit
	1.2.0
	test
 
 
	org.junit.jupiter
	junit-jupiter-api
	5.0.2
	test
 
 
	org.junit.jupiter
	junit-jupiter-engine
	5.0.2
	test
 
 
	com.squareup.okhttp3
	okhttp
	3.9.1

Additionally, you need some tweaking to get the tests working with Java9. Add the following line to the properties section in the pom.xml:

1
--add-modules java.xml.bind

Add the following test class:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
package de.moduliertersingvogel.micro;
 
import static org.junit.jupiter.api.Assertions.assertEquals;
 
import org.apache.meecrowave.Meecrowave;
import org.apache.meecrowave.junit5.MeecrowaveConfig;
import org.apache.meecrowave.testing.ConfigurationInject;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
 
import com.squareup.okhttp.OkHttpClient;
import com.squareup.okhttp.Request;
import com.squareup.okhttp.Response;
 
@MeecrowaveConfig /*(some config)*/
public class SimpleEndpointTest {
    @ConfigurationInject
    private Meecrowave.Builder config;
	private static OkHttpClient client;
 
	@BeforeAll
	public static void setup() {
		client = new OkHttpClient();
	}
 
	@Test
	public void test() throws Exception {
		final String base = "http://localhost:" + config.getHttpPort();
 
		Request request = new Request.Builder()
	      .url(base+"/test")
	      .build();
		Response response = client.newCall(request).execute();
		assertEquals("Hello World",  response.body().string());
	}
}

And run it either from your IDE or from maven:

1
mvn clean test

During test execution the server should be started and the tests should be executed successfully against your running meecrowave application should be performed:

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.8 sec - in de.moduliertersingvogel.micro.SimpleEndpointTest

Distribution

Creating a distribution package (for Linux) is as simple as adding a new goal to the maven call:

1
mvn clean package meecrowave:bundle

Note: The meecrowave:bundle goal creates the distribution and includes what is already in compiled as jar in target directory. A call to meecrowave:bundle without package would result in an empty meecrowave server without your application.

After that, your target directory should contain a file called micro-meecrowave-distribution.zip. The zip archive contains a bin folder in which the executable (….sh) is located. For running this in Java9, the java.xml.bind module needs too be added (remember: We are not using modules here, therefore no module-info and no automatic way for Java to figure this out). Search the line strting with JAVA_OPTS in the start script and add:

--add-modules java.xml.bind

Now you can start the service by:

1
2
cd bin
./meecrowave.sh start

The default port used is 8080 and thus you can test your application easily with curl or the browser. Enjoy!

PS: Cors

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
/**
 * See: http://stackoverflow.com/a/28067653
 *
 */
@ApplicationScoped
@Provider
public class CorsFilter implements ContainerResponseFilter {
    @Override
    public void filter(ContainerRequestContext request, ContainerResponseContext response) throws IOException {
        response.getHeaders().add("Access-Control-Allow-Origin", "*");
        response.getHeaders().add("Access-Control-Allow-Headers", "origin, content-type, accept, authorization");
        response.getHeaders().add("Access-Control-Allow-Credentials", "true");
        response.getHeaders().add("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE, OPTIONS, HEAD");
    }
}

PPS: GSON

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.OutputStreamWriter;
import java.lang.reflect.Type;
 
import javax.enterprise.context.ApplicationScoped;
import javax.ws.rs.Consumes;
import javax.ws.rs.Produces;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.MultivaluedMap;
import javax.ws.rs.ext.MessageBodyReader;
import javax.ws.rs.ext.MessageBodyWriter;
import javax.ws.rs.ext.Provider;
 
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
 
@ApplicationScoped
@Provider
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
public class GsonMessageBodyHandler implements MessageBodyWriter<object width="300" height="150">, MessageBodyReader<object> {	private static final String UTF_8 = "UTF-8";	private Gson gson = new GsonBuilder().create();	@Override	public boolean isReadable(Class<!--?--> type, Type genericType, java.lang.annotation.Annotation[] annotations,			MediaType mediaType) {		return true;	}	@Override	public Object readFrom(Class<object> clazz, Type type, java.lang.annotation.Annotation[] annotations,			MediaType mediatype, MultivaluedMap&lt;string, string=""&gt; headers, InputStream instream)			throws IOException, WebApplicationException {		try (InputStreamReader streamReader = new InputStreamReader(instream, UTF_8)) {			return gson.fromJson(streamReader, type);		}	}	@Override	public boolean isWriteable(Class<!--?--> arg0, Type arg1, java.lang.annotation.Annotation[] arg2, MediaType arg3) {		return true;	}	@Override	public void writeTo(Object obj, Class<!--?--> clazz, Type type, java.lang.annotation.Annotation[] annotations,			MediaType mediatype, MultivaluedMap&lt;string, object=""&gt; headers, OutputStream outstream)			throws IOException, WebApplicationException {		try (OutputStreamWriter writer = new OutputStreamWriter(outstream, UTF_8)) {			final String content = gson.toJson(obj);			writer.write(content);		}	}}

Update 20200331:Logging

There seems to be a problem in the generated meecrowave.bat file for starting in Windows. It does not include the log4j2 configuration file. The meecrowave.sh for Linux is working. As a workaround the Windows bat file can be edited manually.

Building Java 9 projects with Eclipse and Gradle

What: Using gradle to build and setup a Java 9 project in Eclipse Oxygen (4.7.1)
Why: Automate settings in Eclipse like module path, … for Java 9 projects
How: Using gradle ‚eclipse’/’java‘ plugin

Background

The following is taken from this buildship issue.

If you create a Java 9 project, a module-info.java file is needed describing your module. The dependencies defined in the build.gradle file are not automatically added to the project settings and your module-info.java file will have compile errors in Eclipse. If added manually, the settings are gone if the project is refreshed via gradle or each dependency has to be added manually if no refresh is used. This can be avoided with the gradle ‚eclipse‘ plugin.

At compile time, the dependencies of your project needs to be available as (automatic) modules and at runtime the dependencies should be available somewhere in the module path.

Requirements

Buildship 2.2

You can install the buildship plugin from Eclipse Marketplace or from the update site. You need version 2.2 at least. Currently, only the update site has this version.

Eclipse project nature

Your project needs to be a gradle project. If this is not the case, you can convert it to a gradle project by right-click on the project -> Configure -> Add Gradle Nature.

Modifications of build.gradle

Add the following to your build.gradle file:

apply plugin 'eclipse'
eclipse {
    classpath {
        file {
            whenMerged {
                entries.findAll { isModule(it) }.each { it.entryAttributes['module'] = 'true' }
            }
        }
    }
}
 
boolean isModule(entry) {
    // filter java 9 modules
	entry.kind == 'lib'  // Only libraries can be modules
}

Update Eclipse project

Rightclick on your project -> Gradle -> Refresh Gradle Project.

Now, the project is set up in such a way, that all your dependencies are available for use in module-info.java.

Add dependencies as modules for compilation

The dependencies defined by the build.gradle are by default not added to the module path if you are using Java 9 for compilation. They can be added by:

compileJava {
    inputs.property("moduleName", moduleName)
    doFirst {
        options.compilerArgs = [
            '--module-path', classpath.asPath,
        ]
        classpath = files()  
    }
}

Add dependencies in a folder which can be used as module path

A simple derived copy task can be used to put all the dependencies in a subfolder of the build diretory:

task makePackage(type: Copy) {
    into "$buildDir/lib"
    from configurations.runtime
}

This folder can be used as module path while running your project:

java --module-path <other entries>:<buildDir/lib> <modulename>/<Main class>

Flashing christmas poems

What: Using a raspberry pi and some LEDs to send christmas poems in morse code
Why: Controlling LEDs with raspberry
How: Using RPi.GPIO library, some simple LEDs and a python script

Morse code

A basic morse code library for python can be found at github. Essentially, it maps lower case characters (no special ones) to a sequence of morse symbols.
It is a python dictionary with the characters as keys:

s='short'
l='long'
p='pause'

alphabet={}
alphabet['a']=[s, l]
alphabet['b']=[l, s, s, s]

You can use it in any python program:

from morsecode import alphabet

for character in text:
    code=alphabet[character]
    for dur in code:
        if dur=='short':
            # Do something here
            pass
        if dur=='long':
            # Do something here
            pass
        if dur=='short' or dur=='long':
            # Do something here
            pass
        if dur=='pause':
            # Do something here
            pass
    characterbreak()
textbreak()

Translate text to LED signals

Using two LEDs, one can encode the dit (short symbol) and the other one the dah (long symbol) in morse code. You need additional breaks between characters and words. These can be encoded by not flashing any LED.

Putting it together with the morse alphabet and simple logic for processing arbitrary text (note: the script below does not check, if all characters used are present in the alphabet; just use lower case ones and comma and dot):

#!/usr/bin/python
import RPi.GPIO as GPIO
import time
from morsecode import alphabet
import sys

GPIO.setmode(GPIO.BCM)
GPIO.setup(4, GPIO.OUT)
GPIO.setup(10, GPIO.OUT)

if len(sys.argv)==1:
    print('you have to provide the text as argument')
    sys.exit(1)
text=sys.argv[1].lower()

dittime=0.25

def dit():
    GPIO.output((4, 10), (GPIO.HIGH, GPIO.LOW))
    time.sleep(dittime)

def dah():
    GPIO.output((4, 10), (GPIO.LOW, GPIO.HIGH))
    time.sleep(3*dittime)

def symbolbreak():
    GPIO.output((4, 10), (GPIO.LOW, GPIO.LOW))
    time.sleep(dittime)

def pause():
    GPIO.output((4, 10), (GPIO.LOW, GPIO.LOW))
    time.sleep(4*dittime)

def characterbreak():
    GPIO.output((4, 10), (GPIO.LOW, GPIO.LOW))
    time.sleep(3*dittime)

def textbreak():
    GPIO.output((4, 10), (GPIO.LOW, GPIO.LOW))
    time.sleep(10*dittime)

while True:
    for character in text:
        code=alphabet[character]
        for dur in code:
            if dur=='short':
                dit()
            if dur=='long':
                dah()
            if dur=='short' or dur=='long':
                symbolbreak()
            if dur=='pause':
                pause()
        characterbreak()
    textbreak()

GPIO.cleanup()

You can start it with (assume you saved the above code in a file called morseflash.py and assigned execution rights):

./morseflash.py "I syng of a mayden that is makeles, \
kyng of alle kynges to here sone che ches. ..."

See it in action:

Chess in space, wondrous stuff and IT