Archiv der Kategorie: Visualization

Fly me to the stars II: Explorative data analysis in Jupyter with python

What: Use Dygraphs within Jupyter notebooks
Why: Interactive explorative data analysis
How: Use jupyter_dygraphs for dygraphs plots within Jupyter

Installation

Jupyter_dygraphs can be easily imported directly from github:

1
pip install git+https://github.com/frankdressel/jupyter_dygraphs.git#egg=jupyter_dygraphs

Usage

Import the jupyter_dygraphs module into your notebook:

1
from jupyter_dygraphs import jupyter_dygraphs

Call the _dygraphplot_ function with dictionaries containing the pandas data frame and plot options:

1
2
3
4
5
6
7
8
9
jupyter_dygraphs.dygraphplot({
    'df': pandas.read_csv('https://moduliertersingvogel.de/wp-content/uploads/2018/05/temperatures.csv'),
    'options': {
        'title': 'Title for test plot',
        'xlabel': 'Date',
        'ylabel':
        'Temperature'
    }
})

Example plot

Further information

See github.

Make a small animation based on existing images

What: Concatenate images to a video
Why: Small animation as logo, simple stop and go movie
How: Using ffmpeg to concat existing images into mp4 file

Requirements

The following is using ffmpeg in debian linux. There are also builds for other platforms available. Depending on the platform you maybe have to install the codec. The following is tested with a default debian system.

Install ffmpeg with:

1
apt-get install ffmpeg

Create some images

Create a sequence of images. Lets assume we want to have a spinning wheel of the following kind for this tutorial:










The important point is the numerical ordering of the image names like img1.png, img2.png, ….

Create the video

Create the video according to the documentation (assuming an image numbering like mentioned above):

1
ffmpeg -i %d.png -pix_fmt yuv420p output.mp4

The above command works because there are less than 10 images. If you have another image numbering (for example 001-999) you have to change the pattern:
– img%02d.png for 01-99
– img%03d.png for 001-999
– …

The final result will look like:

Fly me to the stars: Interactive graphs with Docker, Jupyter and R

What: Documented data analysis on the fly
Why: Keep documentation, implementation and operation in one document; create publication ready reports; explore data interactively
How: Use Docker (containerization), Jupyter (documenting and code execution), R (analysis) and Plotly (interactive plots, very interesting also for web-frontend Javascript graphs)

You have to install Docker to run this tutorial. All dependencies are automatically installed in the container. Otherwise, you have to install the dependencies manually. Additionally, you need a browser to get access to Jupyter. The interactive plots work best in Chrome. I had some issues running it with firefox.

Copy the following content in a file called Dockerfile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
FROM ubuntu:xenial
 
RUN apt-get update
RUN apt-get install -y wget
RUN apt-get install -y bzip2
 
# Install python
RUN wget https://3230d63b5fc54e62148e-c95ac804525aac4b6dba79b00b39d1d3.ssl.cf1.rackcdn.com/Anaconda3-4.0.0-Linux-x86_64.sh && bash Anaconda3-4.0.0-Linux-x86_64.sh -b -p $HOME/anaconda
 
# Install R
RUN apt-get install -y r-base
RUN apt-get install -y r-recommended
 
# Install requirements for Jupyter
RUN apt-get install -y libzmq3-dev
# See http://askubuntu.com/a/628288
RUN apt-get install -y libcurl4-openssl-dev
RUN apt-get install -y libssl-dev
RUN apt-get install -y inkscape
RUN apt-get install -y pandoc
RUN apt-get install -y texlive-latex-base
RUN apt-get install -y texlive-latex-extra
 
# Prepare R and Jupyter.
ADD installJupyter.R /root/installJupyter.R
RUN export PATH="$HOME/anaconda/bin:$PATH" && R CMD BATCH /root/installJupyter.R
RUN export PATH="$HOME/anaconda/bin:$PATH" && jupyter notebook --generate-config
# Allow all clients to connect. THIS IS NOT SECURE!
RUN sed -i s/.*NotebookApp\.ip.*localhost.*/c.NotebookApp.ip=\'*\'/g ~/.jupyter/jupyter_notebook_config.py
RUN mkdir /root/jupyternotebooks
 
# Set the locale (german)
RUN echo >> /etc/locale.gen de_DE.UTF-8 UTF-8
RUN echo >> /etc/locale.gen de_DE ISO-8859-1
RUN echo >> /etc/locale.gen de_DE@euro ISO-8859-15
RUN locale-gen
RUN echo "Europe/Berlin" | tee /etc/timezone
RUN dpkg-reconfigure --frontend noninteractive tzdata
 
# Taken from: http://jaredmarkell.com/docker-and-locales/ and adapted
#RUN apt-get update && apt-get install -y language-pack-de-base && locale-gen de_DE && echo >> ~.profile export LANG=de_DE.ISO-8859-1
ENV LANG de_DE.iso88591
ENV LANGUAGE de_DE.iso88591
ENV LC_ALL de_DE.iso88591
 
# Start Jupyter on port 8888
EXPOSE 8888
CMD export PATH="$HOME/anaconda/bin:$PATH" && cd /root/jupyternotebooks && jupyter notebook --no-browser

In the same directory, you need the file installJupyter.R with the following content:

1
2
3
4
5
6
7
8
9
10
install.packages("digest", repos="https://cran.uni-muenster.de/")
install.packages("uuid", repos="https://cran.uni-muenster.de/")
install.packages("jsonlite", repos="https://cran.uni-muenster.de/")
install.packages("base64enc", repos="https://cran.uni-muenster.de/")
install.packages("evaluate", repos="https://cran.uni-muenster.de/")
install.packages("dplyr", repos="https://cran.uni-muenster.de/")
install.packages("stringr", repos="https://cran.uni-muenster.de/")
install.packages("plotly", repos="https://cran.uni-muenster.de/")
install.packages(c("rzmq","repr","IRkernel","IRdisplay"), repos = c("http://irkernel.github.io/", "https://cran.uni-muenster.de/"), type = "source")
IRkernel::installspec()

Now, run the following two docker commands in the same directory where the docker file is located:

1
2
docker build -t docker-jupyter .
docker run --rm -v <Path to a folder for the jupyter notebooks>:/root/jupyternotebooks -p 8888:8888 docker-jupyter

You should now have the running docker instance available on the docker VM, port 8888:
screenshot_jupyter

Via the „Upload“ button, upload the Notebook from here: Interactive graphs with Jupyter and R and open it. Execute all steps via „Cell“ -> „Run all“. Now, it should look like:

screenshot_jupyter_plotly

Did you noticed the interactive plot? You can zoom, export as png, have tooltips, … All without anything to programm. Cool, isn’t it?

Ok. Your customer is impressed but does not like the modern html stuff? He wants to print it out and send it away via mail? No problem. Just change the interactive plots to none-interactive ones and export to pdf via „File“ -> „Download as“ -> „PDF via Latex“ (that ’s why the docker file above contains all the stuff like pandoc, latex, …). You will get nearly publication ready report out.