Creating vim split window with arbitrary content

What: Create a split window in Vim with arbitrary content
Why: Displaying information easily along text your are editing
How: Use Vimscript

A simple Vimscript script

Create a file test.vim with the following content. It defines a function which creates a new buffer, adds text to it and show it in a split window on the right.

Use the defined function

There are multiple ways, the function can be used. You can create a plugin (see https://learnvimscriptthehardway.stevelosh.com/chapters/41.html) or call the function from within Vim:

Tomcat & Meecrowave on a server: Slow startup

What: Using Meecrowave on a server without large delays on startup
Why: Faster startup during development
How: Use a faster entropy source

Background

Meecrowave uses Tomcat, which uses a SecureRandom instance for session ids. On a server, the underlying entropy source can run short resulting in large delays until the application endpoints are scanned and enabled.

Details

See the tomcat how to.

Solution

Add the following line to the JAVA_OPTS in meecrowave.sh:

1
-Djava.security.egd=file:/dev/./urandom

Measure loudness with a USB micro on a raspberry pi

What: Measuring loudness with a simple USB microphone on a raspberry pi
Why: Create devices which are activated by a certain noise level
How: Use python and some python libraries to analyse the sound stream

Requirements

The following hardware was used for this setup:

The following os was used on the raspberry pi:

The following python version was used (preinstalled with the corresponding raspbian):

  • Python 2.7 (PyAudio was not working for me with Python3 easily)

Setup

Besides plugging the USB microphone in the corresponding USB port (such a surprise), the following things needs to be done.

Install PortAudio

Install the dependencies:

Download the corresponding version (at the time of writing this is 190600_20161030) and uncompress it:

Build PortAudio from source:

Install NumPy, PyAudio and SoundAnalyse

Code

Below is the code I used for measuring. There are some caveats with measuring the loudness:

  • There are by default some warnings when opening the stream
  • When reading the bytes from the stream, there may be an overflow exception which will kill the script. This can be avoided by the parameter exception_on_overflow = False
  • The parameters for pyaud.open may be different depending on the used microphone. The parameters can be determined by iterating over the devices and use: pyaud.get_device_info_by_index(i)

Feature engineering in python

What: Generating features for regression models in python based on formulas (like in R)
Why: Model notation in formulas and not in code => Better maintainability
How: Use patsy

Test data

Lets take as example a simple: fitting an exponential to some test data. First, some test data needs to be generated:

The test data and its components look like this:

Feature engineering

Lets assume, we want to fit the data with a linear and quadratic term (surprise, surprise). When using a linear model, you would provide a matrix where each column is one of the features. In our case this would mean writing a small script to fill a length x 3 matrix. Although it is no big deal it involves some lines of code and is not very readable.

With patsy we can write instead something like the following and let the matrices be generated by patsy:

This will result for x in something like:

Fitting

With sklearn it is easy to do the fit:

In the end, it looks like:

Convert raw images to jpeg

What: Convert raw camera images (like in CR2 format) to jpeg
Why: Smaller image size and easier handling in other programms
How: Use dcraw and pnmtojpeg in Debian

Installation of tools

Small script to iterate over the raw image directory

Fly me to the stars II: Explorative data analysis in Jupyter with python

What: Use Dygraphs within Jupyter notebooks
Why: Interactive explorative data analysis
How: Use jupyter_dygraphs for dygraphs plots within Jupyter

Installation

Jupyter_dygraphs can be easily imported directly from github:

Usage

Import the jupyter_dygraphs module into your notebook:

Call the _dygraphplot_ function with dictionaries containing the pandas data frame and plot options:

Example plot

Further information

See github.

Bulk delete in CouchDB

What: Deleting all documents from Couchdb with a single command from the command line without deleting the database/design documents
Why: Truncate the database
How: Python3 and requests

Retrieve all documents

Couchdb has a rest api, which allows the retrieval of all documents from a database. To delete documents, the corresponding id and revision of each document is needed. Further attributes of the document can be ignored.

To retrieve all documents, a simple get request is enough, which will return a json document with an attribute rows which contains a list of ids and revisions of all documents:

Set delete flag

Documents can be deleted from Couchdb by setting the attribute _deleted to true (for some subtleties see: this blog). Lets create the minimal information for deletion for each document:

Push changes

While all documents can be retrieved from the data base at once, it is also possible to submit multiple documents in one request:

Make it user friendly

To make it a little bit more user friendly, the name of the database can be set as argument and the python script should be called from the command line. In the end it looks like:

Have fun in extending the script and use it for maintaining your Couchdb!

Cooking with meecrowave

What: Building microservices in Java(9) painlessly
Why: Small and maintainable services
How: Using meecrowave

Introduction

Creating microservices in Java can be quite complicated. You either do it by yourself using the Java internal HTTP-Server, using one of the many application servers or using one of the integrated frameworks like wildfly-swarm. While the first option doesn’t work well if you want to use something like dependency injection and you have to include all useful libraries like jax-rs by yourself, the second option already contains the most parts of it. You can use for example Glassfish, Wildfly, Websphere or Tomcat (and TomEE). Nevertheless, you rely on heavy application servers and you have to start an instance of these and deploy for each testing (although there exists some integration solutions into IDEs).

The integrated frameworks are sometimes huge, need extended configuration or doesn’t play well with Java9. Partly testing is not as easy as it should be (dependency injection is one of the issues).

Meecrowave on the other hand is a small framework which works well with CDI, JAX-RS and Jackson out of the box which is super easy to set up and performing integration tests is as easy as starting a JUnit test. The following tutorial shows an easy example.

The source code for this tutorial is available here (folder micro).

Note: Although the example runs with Java9 it is not modularized. Some of the dependencies are not yet available as Java9 modules and thus creating this example as a module is outof scope for this tutorial).

Setup

In the following, maven and jdk9 (both for compiling and running) is used.

Add the following dependencies to your pom.xml to include the needed libraries for this example.

Server

Starting meecrowave is simple: Just start the meecrowave server. All the rest like scanning classes for endpoints, … is done automatically. Create a class with a main method and add the following code:

If you start the class, you should see some printout and meecrowave is up and running. To start it from Java9 you have to add –add-modules java.xml.bind as argument to the virtual machine.

Note: The main class is not needed at all for running it outside of ides(at least not on Linux machines) since meecrowave can create a whole distribution package (see below).

You should see output like:

JAX-RS endpoints

You can create arbitrary endpoints based on the jax-rs annotations. Each endpoint needs to be annotated with a Path and scope annotation. The following example defines a simple endpoint returning the string „Hello World“:

You can point your browser to http://localhost:8080/test and see the result.

Dependency injection

Dependency injection works like expected. You need a class, which is annotated with the scope and inject it somewhere. Lets test it with a simple object:

Run your main class and check in yout browser (see above) to see that everything works fine.

Testing

Testing in meecrowave is as simple as writing (annotated) unit tests. In order to get it working, you have to add the following dependencies to your pom.xml (okhttp is used for getting the result from the running microservice):

Additionally, you need some tweaking to get the tests working with Java9. Add the following line to the properties section in the pom.xml:

Add the following test class:

And run it either from your IDE or from maven:

During test execution the server should be started and the tests should be executed successfully against your running meecrowave application should be performed:

Distribution

Creating a distribution package (for Linux) is as simple as adding a new goal to the maven call:

Note: The meecrowave:bundle goal creates the distribution and includes what is already in compiled as jar in target directory. A call to meecrowave:bundle without package would result in an empty meecrowave server without your application.

After that, your target directory should contain a file called micro-meecrowave-distribution.zip. The zip archive contains a bin folder in which the executable (….sh) is located. For running this in Java9, the java.xml.bind module needs too be added (remember: We are not using modules here, therefore no module-info and no automatic way for Java to figure this out). Search the line strting with JAVA_OPTS in the start script and add:

Now you can start the service by:

The default port used is 8080 and thus you can test your application easily with curl or the browser. Enjoy!

PS: Cors

PPS: GSON

Update 20200331:Logging

There seems to be a problem in the generated meecrowave.bat file for starting in Windows. It does not include the log4j2 configuration file. The meecrowave.sh for Linux is working. As a workaround the Windows bat file can be edited manually.

Building Java 9 projects with Eclipse and Gradle

What: Using gradle to build and setup a Java 9 project in Eclipse Oxygen (4.7.1)
Why: Automate settings in Eclipse like module path, … for Java 9 projects
How: Using gradle ‚eclipse’/’java‘ plugin

Background

The following is taken from this buildship issue.

If you create a Java 9 project, a module-info.java file is needed describing your module. The dependencies defined in the build.gradle file are not automatically added to the project settings and your module-info.java file will have compile errors in Eclipse. If added manually, the settings are gone if the project is refreshed via gradle or each dependency has to be added manually if no refresh is used. This can be avoided with the gradle ‚eclipse‘ plugin.

At compile time, the dependencies of your project needs to be available as (automatic) modules and at runtime the dependencies should be available somewhere in the module path.

Requirements

Buildship 2.2

You can install the buildship plugin from Eclipse Marketplace or from the update site. You need version 2.2 at least. Currently, only the update site has this version.

Eclipse project nature

Your project needs to be a gradle project. If this is not the case, you can convert it to a gradle project by right-click on the project -> Configure -> Add Gradle Nature.

Modifications of build.gradle

Add the following to your build.gradle file:

Update Eclipse project

Rightclick on your project -> Gradle -> Refresh Gradle Project.

Now, the project is set up in such a way, that all your dependencies are available for use in module-info.java.

Add dependencies as modules for compilation

The dependencies defined by the build.gradle are by default not added to the module path if you are using Java 9 for compilation. They can be added by:

Add dependencies in a folder which can be used as module path

A simple derived copy task can be used to put all the dependencies in a subfolder of the build diretory:

This folder can be used as module path while running your project:

Flashing christmas poems

What: Using a raspberry pi and some LEDs to send christmas poems in morse code
Why: Controlling LEDs with raspberry
How: Using RPi.GPIO library, some simple LEDs and a python script

Morse code

A basic morse code library for python can be found at github. Essentially, it maps lower case characters (no special ones) to a sequence of morse symbols.
It is a python dictionary with the characters as keys:

You can use it in any python program:

Translate text to LED signals

Using two LEDs, one can encode the dit (short symbol) and the other one the dah (long symbol) in morse code. You need additional breaks between characters and words. These can be encoded by not flashing any LED.

Putting it together with the morse alphabet and simple logic for processing arbitrary text (note: the script below does not check, if all characters used are present in the alphabet; just use lower case ones and comma and dot):

You can start it with (assume you saved the above code in a file called morseflash.py and assigned execution rights):

See it in action:

Chess in space, wondrous stuff and IT