Agile, oven-fresh documentation. Part 2: PlantUML with JBake

codecentric Blog - Wed, 07-Mar-18 01:00

After looking at the configuration of JBake in the first part, I will now extend the scenario described in the previous post by adding PlantUML. PlantUML makes it possible to describe UML diagrams in simple textual notation.

:Hello world;
Result of

First of all, we will integrate PlantUML into the existing Gradle build file. In order to do so, we need asciidoctorj-diagram as an external library for the build script. With the upcoming release of the Gradle plugin it is no longer necessary. The configuration of JBake needs to be adjusted so that the asciidoctorj-diagram library can be used. The asciidoctorj-diagram library now allows us to use PlantUML diagrams within the Asciidoctor document.

buildscript {
    repositories {
    dependencies {
        classpath 'org.asciidoctor:asciidoctorj-diagram:'
plugins {
     id '' version '1.0.0'
group ''
version '1.0-SNAPSHOT'
apply plugin: 'groovy'
apply plugin: 'java'
sourceCompatibility = 1.8
repositories {
dependencies {
    compile 'org.codehaus.groovy:groovy-all:2.3.11'
    testCompile group: 'junit', name: 'junit', version: '4.12'
jbake {
    srcDirName = 'src/documentation'
    destDirName = 'documentation'
    configuration['asciidoctor.option.requires'] = "asciidoctor-diagram"
    configuration['asciidoctor.attributes'] = [

build.dependsOn bake

We now create a new file, architecture.adoc, within the content folder. The following code block shows the content of the file.

= Architecture documentation
Daniel Kocot
:jbake-type: page
:jbake-tags: documentation, manual
:jbake-status: published

== First draft

[plantuml, "first_draft", "png"]
node "Confluence as content repository" as nodeconf
folder Intranet{
    together {
        node "WordPress Node 1" as nodeiwp1
        node "WordPress Node 2" as nodeiwp2
        node "WordPress Node 3" as nodeiwp3
        node "WordPress Node 4" as nodeiwp4
folder Extranet{
    together {
        node "WordPress Node 1" as nodeewp1
        node "WordPress Node 2" as nodeewp2
        node "WordPress Node 3" as nodeewp3
        node "WordPress Node 4" as nodeewp4
node "LoadBalancer / nginx Intranet" as lbinginx
node "LoadBalancer / nginx Extranet" as lbenginx
node "Content Delivery Network" as cdn
cloud "Intranet" as intranet
cloud "Extra/Internet" as internet
actor "Internal User" as internal
actor "External User" as external
nodeconf --> nodeiwp1
nodeconf --> nodeiwp2
nodeconf --> nodeiwp3
nodeconf --> nodeiwp4
nodeconf --> nodeewp1
nodeconf --> nodeewp2
nodeconf --> nodeewp3
nodeconf --> nodeewp4
nodeewp1 <--> lbenginx
nodeewp2 <--> lbenginx
nodeewp3 <--> lbenginx
nodeewp4 <--> lbenginx
nodeiwp1 <--> lbinginx
nodeiwp2 <--> lbinginx
nodeiwp3 <--> lbinginx
nodeiwp4 <--> lbinginx
cdn <--> nodeconf
cdn --> nodeewp1
cdn --> nodeewp2
cdn --> nodeewp3
cdn --> nodeewp4
cdn --> nodeiwp1
cdn --> nodeiwp2
cdn --> nodeiwp3
cdn --> nodeiwp4
lbinginx <--> intranet
lbenginx <--> internet
external <--> internet
internal <--> intranet

After writing the first draft of our architectural diagram into the document, there is still the question of how the text content is turned into a graphic.

For this purpose, when creating the entire documentation, the first thing we do is generate a picture – in our case a PNG file – via PlantUML. This picture is saved in the assets folder under the file name first-draft with the file extension .png. In a second step, JBake will then create the corresponding HTML file based on the Asciidoctor file.

The PlantUML integration enables us to also maintain the important architecture diagrams in close alignment with the source code.

In the third part of this series, we will take a look at test reports and their integration in JBake.

Check out the demo in my Git Repository: agile-fresh-baked-docs

The post Agile, oven-fresh documentation. Part 2: PlantUML with JBake appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

Cloud Launcher for MongoDB in the Google Compute Engine

codecentric Blog - Sun, 04-Mar-18 23:00

In this post you will learn how to use Google’s Cloud Launcher to set up instances for a MongoDB replica set in the Google Compute Engine.

Replication in MongoDB

A minimal MongoDB replica set consists of two data bearing nodes and one so-called arbiter that only takes part in the election of a new primary in case of failure.

Unlike other distributed databases, MongoDB does not offer auto-discovery of cluster nodes. If all nodes are up, you have to initialize the cluster via an administrative command inside the Mongo CLI that takes list of all replica set members. This fact makes it hard (but not impossible) to script this with the cloud provisioning tool of your choice.

Cloud Launcher for MongoDB

With the help of Google’s Cloud Launcher for MongoDB, the provisioning of a replica set is done in just a few steps.

First, we define a deployment name (that will prefix the names of the instances), a zone and the name of the replica set.

cloud launcher mongodb

Then, we set up the instances that will run in the Compute Engine. We define a minimal replica set with two server instances and one arbiter. All other parameters will use the defaults for now (the server instances will use a n1-highmem-4 machine type).

cloud launcher mongodb

This will finally lead to three running instances.

Compute Engine Instances

The instances will show up in the Compute Engine dashboard where they can be managed:

compute engine instances

If you prefer your CLI, you can list the instances with the gcloud tool:

$ gcloud compute instances list
mongo-0-arbiters-vm-0  europe-west3-b  n1-standard-1       RUNNING
mongo-0-servers-vm-0   europe-west3-b  n1-highmem-4         RUNNING
mongo-0-servers-vm-1   europe-west3-b  n1-highmem-4        RUNNING

Replica Set Status

In order to check if everything is up and running, we open a SSH window to one of the instances from the dasboard and start the mongo CLI:

compute engine ssh

After connecting to the MongoDB, we run the rs.status() to get the status of the replica set:

  "set" : "rs0",
  "date" : ISODate("2018-02-12T12:52:47.562Z"),
  "myState" : 7,
  "members" : [
      "_id" : 0,
      "name" : "mongo-0-servers-vm-0:27017",
      "health" : 1,
      "state" : 1,
      "stateStr" : "PRIMARY",
      "uptime" : 787,
      "_id" : 1,
      "name" : "mongo-0-servers-vm-1:27017",
      "health" : 1,
      "state" : 2,
      "stateStr" : "SECONDARY",
      "uptime" : 787,
        "_id" : 2,
        "name" : "mongo-0-arbiters-vm-0:27017",
        "health" : 1,
        "state" : 7,
        "stateStr" : "ARBITER",
        "uptime" : 801,
        "configVersion" : 3,
        "self" : true
  "ok" : 1

(I ommitted some of the output, so we can focus on the important things.)

We can see the name of the replica set rs0 and our three instances

  • mongo-0-servers-vm-0
  • mongo-0-servers-vm-1
  • mongo-0-arbiters-vm-0

Restore Data Dump from Google Storage

We’ll restore a data dump from a file located in the Google Storage. Since only the the primary node can do write operations, we connect to the instance mongo-0-servers-vm-0 that has to role PRIMARY right now.

Inside that instance, we download a BSON dump from the storage by using the gsutil tool, assuming your storage bucket is called [BUCKET_ID]:

mongo-0-servers-vm-1:/tmp$ cd /tmp
mongo-0-servers-vm-1:/tmp$ gsutil cp gs://[BUCKET_ID]* ./
Copying gs://[BUCKET_ID]
Copying gs://[BUCKET_ID]             
/ [2 files][  5.6 KiB/  5.6 KiB]                                                
Operation completed over 2 objects/5.6 KiB. 

Now the dump is on our disk and we restore its data to the MongoDB replica set:

mongo-0-servers-vm-1:/tmp$ mongorestore pois.bson
2018-02-12T13:12:33.402+0000    checking for collection data in pois.bson
2018-02-12T13:12:33.402+0000    reading metadata for test.pois from pois.metadata.json
2018-02-12T13:12:33.418+0000    restoring test.pois from pois.bson
2018-02-12T13:12:33.481+0000    restoring indexes for collection test.pois from metadata
2018-02-12T13:12:33.590+0000    finished restoring test.pois (7 documents)
2018-02-12T13:12:33.590+0000    done

Finally, we perfom a simple query from the OS command line:

mongo-0-servers-vm-0:/tmp$ mongo --quiet --eval "db.pois.count()" test

to check if our 7 imported documents are really there.


We created a MongoDB replica set and deployed it to the Compute Engine. Then we checked the replica set status and imported some data.

In one of my next posts I will show you how to build a REST-ful microservice in the AppEngine that will access our MongoDB backend in the Compute Engine.

The post Cloud Launcher for MongoDB in the Google Compute Engine appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

ETL with Kafka

codecentric Blog - Fri, 02-Mar-18 05:04

“ETL with Kafka” is a catchy phrase that I purposely chose for this post instead of a more precise title like “Building a data pipeline with Kafka Connect”.


You don’t need to write any code for pushing data into Kafka, instead just choose your connector and start the job with your necessary configurations. And it’s absolutely Open Source!

Kafka Connect


Before getting into the Kafka Connect framework, let us briefly sum up what Apache Kafka is in couple of lines. Apache Kafka was built at LinkedIn to meet the requirements that message brokers already existing in the market did not meet – requirements such as scalable, distributed, resilient with low latency and high throughput. Currently, i.e. 2018, LinkedIn is processing about 1.8 petabytes of data per day through Kafka. Kafka offers a programmable interface (API) for a lot of languages to produce and consume data.

Kafka Connect

Kafka Connect has been built into Apache Kafka since version 0.9 (11/2015), although the idea had been in existence before this release, but as a project named Copycat. Kafka Connect is basically a framework around Kafka to get data from different sources in and out of Kafka (sinks) into other systems e.g. Cassandra with automatic offset management, where as a user of the connector you don’t need to worry about this, but rely on the developer of the connector.

Besides that, in discussions I have often come across people who were thinking that Kafka Connect was part of the Confluent Enterprise and not a part of Open Source Kafka. To my surprise, I have even heard it from a long-term Kafka developer. That confusion might be due to the fact that if you google the term Kafka Connect, the first few pages on Google are by Confluent and the list of certified connectors.

Kafka Connect has basically three main components that need to be understood for a deeper understanding of the framework.

Connectors are, in a way, the “brain” that determine how many tasks will run with the configurations and how the work is divided between these tasks. For example, the JDBC connector can decide to parallelize the process to consume data from a database (see figure 2).
Tasks contain the main logic of getting the data into Kafka from external systems by connecting e.g. to a database (Source Task) or consuming data from Kafka and pushing it to external systems (Sink Task).
Workers are the part that abstracts away from the connectors and tasks in order to provide a REST API (main interaction), reliability, high availability, scaling, and load balancing.


Kafka connect can be started in two different modes. The first mode is called standalone and should be used only in development because offsets are being maintained on the file system. This would be really bad if you were running this mode in production and your machine was unavialable. This could cause the loss of the state, which means the offset is lost and you as a develeoper don’t know how much data has been processed.



The second mode is called distributed. There, the configuration, state and status are stored in Kafka itself in different topics which benefit from all Kafka characteristics such as resilience and scalability. Workers can start on different machines and the attribute in the .properties file will eventually form the Kafka Connect Cluster which can be scaled up or down.


So let’s look in the content of the pretty self-explanatory topic use in the configuration file:

// TOPIC => connect-configs
 "twitter", "twitter.consumersecret":"XXXXXX", 

// TOPIC => connect-offsets

// TOPIC => connect-status

The output shown here of the messages are just the values, the key of the message is used to identify the different connectors.

Interaction pattern

There is also a different interaction pattern normally between the standalone and distributed mode – in a non-production environment where you just want to test out a connector, for example, and you want to set manually the offset of your choice. You can start the standalone mode with passing in the sink or source connector that you want to use, e.g. bin/kafka-connect config/ config/ config/

On the other hand, you can start the Kafka Connect worker in the distributed mode with the following command: bin/kafka-connect config/ After that, you can list all available connectors, start, change configurations on the fly, restart, pause and remove connectors via the exposed REST API of the framework. A full list of supported endpoints can be found in the offical Kafka Connect documentation.


So let’s have a closer look at an example of a running data pipeline where we are getting some real time data from Twitter and using the kafka-console-consumer to consume and inspect the data.


Here is the complete example shown in the terminal recording: Github repository. You can download and play around with the example project.


In this blog post, we covered the high-level components that are the building blocks of the Kafka Connect framework. The latter is a part of the Apache Kafka Open Source version that allows data engineers or business departments to move data from one system to another without writing any code via Apache Kafka’s great characteristics, of which we barely scratched the surface in this post. So happy connecting…

The post ETL with Kafka appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

Agile, oven-fresh documentation. Part 1: JBake

codecentric Blog - Fri, 23-Feb-18 01:00

In software development, one question that constantly arises is how the current project can be comprehensively documented. The most common way to do this is to use Confluence or some other wiki. But why not use the code itself?
This is why, in the first part of this article series, I will give you an understanding of JBake.
JBake is actually a generator for static websites. Dan Allen, Project Lead of the Asciidoctor project, calls it “Jekyll of the JVM”. Using plugins for Maven and Gradle, JBake can easily be integrated into build pipelines.
I will start off by explaining potential use cases of JBake based on an example. First and foremost, we will build a simple Gradle project to provide a starting point for further steps.

group ''
version '1.0-SNAPSHOT'

apply plugin: 'groovy'
apply plugin: 'java'

sourceCompatibility = 1.8

repositories { 

dependencies {
   compile 'org.codehaus.groovy:groovy-all:2.3.11'
   testCompile group: 'junit', name: 'junit',version: '4.12'

Now we extend the build file by the plugin jbake-gradle-plugin.

plugins {
     id '' version '1.0.0'

This gives us another “bake” task that allows us to “bake” the documentation. For this, we also need a JBake basic installation within the sources. Under MacOS, this can be easily done with SDKMAN

$ sdk install jbake

or Homebrew

$ brew install jbake

By default, the Gradle plugin expects the documentation sources to be located in src/jbake. By using parameters in the build file, the source and destination can be adjusted. The target, however, is always located in the build folder. For our example, we use the following values:

jbake {
     srcDirName = 'src/documentation'
     destDirName = 'documentation'

Since JBake is now installed, we can run the command

jbake -i

in the folder


The resulting structure can now be extended and adapted as desired. Content can be saved in the content folder in the form of HTML, Asciidoctor, or Markdown files. For this blog article, we will replace the existing file about.html with about.adoc and adjust the content accordingly.

= Project documentation
Daniel Kocot
:jbake-type: page
:jbake-tags: documentation, manual
:jbake-status: published

Within the Asciidoctor document, it is possible to save JBake metadata such as page type, tags, and the status of each page. The latter controls the visibility of the pages after the “baking process”. If you do not like the standard template, you can customize it according to your own preferences using the template languages Freemaker, Groovy Simple, Groovy Markup, Thymeleaf, and Jade.
Now, looking back at the build process, we can connect the build task with the bake task to complete this first part.

build.dependsOn bake

If we now execute the build task, the documentation will be generated as well. In the second part, we will add more ingredients to our agile, freshly baked documentation.

Check out the demo in my Git Repository: agile-fresh-baked-docs.

The post Agile, oven-fresh documentation. Part 1: JBake appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

Improve your test structure with Lambdas and Mockito’s Answer

codecentric Blog - Thu, 15-Feb-18 00:57

Although the use of mock objects is controversial, we as developers have to use them from time to time. The nearly 6000 stars Mockito has on GitHub indicate that others would agree with this statement. Especially when we are dealing with library classes that we cannot easily instantiate or with classes that establish some connection like HTTP, mocks show their strength. In order to make tests more readable, Java’s lambdas and Mockito’s Answer can help us.

Motivating example

One class that is a good candidate for mocking is Spring’s RestTemplate. In order to have an easy to set up and fast test we usually do not want to ramp-up the complete Spring Application Context. We would rather mock the RestTemplate and return some pre-canned responses. To give you an example I created a simple service that retrieves Chuck Norris facts. You can find the example on GitHub.

A simple approach to mocking the RestTemplate often results in test code that looks like this:

public class ChuckNorrisServiceNeedsRefactoringTest {

    private static final Long EXISTING_JOKE = 1L;
    private static final Map GOOD_HTTP_PARAMS = Collections.singletonMap("id", EXISTING_JOKE);
    private static final Long NON_EXISTING_JOKE = 15123123L;
    private static final Map NON_EXISTING_HTTP_PARAMS = Collections.singletonMap("id", NON_EXISTING_JOKE);
    private static final Long BAD_JOKE = 99999999L;
    private static final Map BAD_HTTP_PARAMS = Collections.singletonMap("id", BAD_JOKE);

    private static final ResponseEntity ERROR_RESPONSE =
            new ResponseEntity<>(new ChuckNorrisFactResponse("NoSuchQuoteException", "No quote with id=15123123."), HttpStatus.OK);
    private static final ResponseEntity ITEM_RESPONSE =
            new ResponseEntity<>(new ChuckNorrisFactResponse("success", new ChuckNorrisFact(1L, "Chuck Norris is awesome")), HttpStatus.OK);

    public void serviceShouldReturnFact() {
        RestTemplate restTemplate = mock(RestTemplate.class);
        when(restTemplate.getForEntity(FACT_URL, ChuckNorrisFactResponse.class, GOOD_HTTP_PARAMS))
        ChuckNorrisService myServiceUnderTest = new ChuckNorrisService(restTemplate);

        ChuckNorrisFact chuckNorrisFact = myServiceUnderTest.retrieveFact(EXISTING_JOKE);

        assertThat(chuckNorrisFact, is(new ChuckNorrisFact(EXISTING_JOKE, "Chuck Norris is awesome")));

    public void serviceShouldReturnNothing() {
        RestTemplate restTemplate = mock(RestTemplate.class);
        when(restTemplate.getForEntity(FACT_URL, ChuckNorrisFactResponse.class, NON_EXISTING_HTTP_PARAMS))
        ChuckNorrisService myServiceUnderTest = new ChuckNorrisService(restTemplate);

        ChuckNorrisFact chuckNorrisFact = myServiceUnderTest.retrieveFact(NON_EXISTING_JOKE);

        assertThat(chuckNorrisFact, is(nullValue()));

    @Test(expected = ResourceAccessException.class)
    public void serviceShouldPropagateException() {
        RestTemplate restTemplate = mock(RestTemplate.class);
        when(restTemplate.getForEntity(FACT_URL, ChuckNorrisFactResponse.class, BAD_HTTP_PARAMS))
                .thenThrow(new ResourceAccessException("I/O error"));
        ChuckNorrisService myServiceUnderTest = new ChuckNorrisService(restTemplate);


In this test, the two Mockito methods mock() and when() are statically imported. mock() creates the RestTemplate mock object and when() records the behaviour that is expected.

This test code is not too bad, but also not too good. We already see some repetition (we should keep our code DRY) and if we would ever switch from the RestTemplate to something else we will have to touch every test. Therefore, let’s see how we can improve this.

We can clearly see that extracting a method could improve the first two tests. This method then takes the answer and the http parameter and configures the mock. The third test method does not fit the schema because it throws an exception instead of returning a ResponseEntity. Next to the duplication, we are actually dealing too much with technical details here. When reading the tests, do we really need to know if GET or POST is being executed? Do we even have to know the type of the response? What we actually care about is how the ChuckNorrisService behaves. The HTTP communication is hidden inside it.

Lambdas to the rescue

This is where Lambdas can help us to improve our test structure. Next to the probably well known Mockito methods thenReturn and thenThrow there is also thenAnswer. This method expects a parameter implementing the generic Answer interface, which can do basically anything. The advantage is that an Answer can compute the value it returns. This differs from the values which thenReturn and thenThrow take because those are fixed. I do not know if it was intentional or not, but Mockito’s Answer interface fulfills the requirements of a Java 8 functional interface. With its single method T answer(InvocationOnMock invocation) throws Throwable; it is equivalent to java.util.function.Function. The only difference is the throws. Having this knowledge, we can get rid of the code duplication and show clearly what our intention in the test is.

To start, I will directly show you the refactored version of the example above:

public class ChuckNorrisServiceStepOneTest {

    private static final Long EXISTING_JOKE = 1L;
    private static final Map GOOD_HTTP_PARAMS = Collections.singletonMap("id", EXISTING_JOKE);
    private static final Long NON_EXISTING_JOKE = 15123123L;
    private static final Map NON_EXISTING_HTTP_PARAMS = Collections.singletonMap("id", NON_EXISTING_JOKE);
    private static final Long BAD_JOKE = 99999999L;
    private static final Map BAD_HTTP_PARAMS = Collections.singletonMap("id", BAD_JOKE);

    private static final ResponseEntity ERROR_RESPONSE =
            new ResponseEntity<>(new ChuckNorrisFactResponse("NoSuchQuoteException", "No quote with id=15123123."), HttpStatus.OK);
    private static final ResponseEntity ITEM_RESPONSE =
            new ResponseEntity<>(new ChuckNorrisFactResponse("success", new ChuckNorrisFact(1L, "Chuck Norris is awesome")), HttpStatus.OK);

    public void serviceShouldReturnFact() {
        RestTemplate restTemplate = restEndpointShouldAnswer(GOOD_HTTP_PARAMS, (invocation) -> ITEM_RESPONSE);
        ChuckNorrisService myServiceUnderTest = new ChuckNorrisService(restTemplate);

        ChuckNorrisFact chuckNorrisFact = myServiceUnderTest.retrieveFact(EXISTING_JOKE);

        assertThat(chuckNorrisFact, is(new ChuckNorrisFact(EXISTING_JOKE, "Chuck Norris is awesome")));

    public void serviceShouldReturnNothing() {
        RestTemplate restTemplate = restEndpointShouldAnswer(NON_EXISTING_HTTP_PARAMS, (invocation -> ERROR_RESPONSE));
        ChuckNorrisService myServiceUnderTest = new ChuckNorrisService(restTemplate);

        ChuckNorrisFact chuckNorrisFact = myServiceUnderTest.retrieveFact(NON_EXISTING_JOKE);

        assertThat(chuckNorrisFact, is(nullValue()));

    @Test(expected = ResourceAccessException.class)
    public void serviceShouldPropagateException() {
        RestTemplate restTemplate = restEndpointShouldAnswer(BAD_HTTP_PARAMS, (invocation -> {throw new ResourceAccessException("I/O error");}));
        ChuckNorrisService myServiceUnderTest = new ChuckNorrisService(restTemplate);


    private RestTemplate restEndpointShouldAnswer(Map httpParams, Answer> response){
        RestTemplate restTemplate = mock(RestTemplate.class);
        when(restTemplate.getForEntity(FACT_URL, ChuckNorrisFactResponse.class, httpParams)).thenAnswer(response);
        return restTemplate;

So, what did improve? Firstly, we can directly see how the HTTP parameter correspond to certain responses. We do not have to skim through the test to match parameters and reponses. Secondly, when reading a single test the details of the REST invocation are now hidden from us. We do not need to know about the URL, HTTP method and response class unless we really have to. Lastly, we managed to unify the handling of the RestTemplate mock by extracting a method. The “normal” answers and the exception are no longer treated differently. Changing the REST call from GET to POST would only require to change one line in the test.

Further refactoring

What we did not solve is spreading the RestTemplate all over the place. By using fields and @Before we can trim down the test even more;

public class ChuckNorrisServiceStepTwoTest {

    private static final Long EXISTING_JOKE = 1L;
    private static final Map GOOD_HTTP_PARAMS = Collections.singletonMap("id", EXISTING_JOKE);
    private static final Long NON_EXISTING_JOKE = 15123123L;
    private static final Map NON_EXISTING_HTTP_PARAMS = Collections.singletonMap("id", NON_EXISTING_JOKE);
    private static final Long BAD_JOKE = 99999999L;
    private static final Map BAD_HTTP_PARAMS = Collections.singletonMap("id", BAD_JOKE);

    private static final ResponseEntity ERROR_RESPONSE =
            new ResponseEntity<>(new ChuckNorrisFactResponse("NoSuchQuoteException", "No quote with id=15123123."), HttpStatus.OK);
    private static final ResponseEntity ITEM_RESPONSE =
            new ResponseEntity<>(new ChuckNorrisFactResponse("success", new ChuckNorrisFact(1L, "Chuck Norris is awesome")), HttpStatus.OK);

    private RestTemplate restTemplate;
    private ChuckNorrisService myServiceUnderTest;

    public void setUp(){
        restTemplate = mock(RestTemplate.class);
        myServiceUnderTest = new ChuckNorrisService(restTemplate);

    public void serviceShouldReturnFact() {
        restEndpointShouldAnswer(GOOD_HTTP_PARAMS, (invocation) -> ITEM_RESPONSE);

        ChuckNorrisFact chuckNorrisFact = myServiceUnderTest.retrieveFact(EXISTING_JOKE);

        assertThat(chuckNorrisFact, is(new ChuckNorrisFact(EXISTING_JOKE, "Chuck Norris is awesome")));

    public void serviceShouldReturnNothing() {
        restEndpointShouldAnswer(NON_EXISTING_HTTP_PARAMS, (invocation -> ERROR_RESPONSE));

        ChuckNorrisFact chuckNorrisFact = myServiceUnderTest.retrieveFact(NON_EXISTING_JOKE);

        assertThat(chuckNorrisFact, is(nullValue()));

    @Test(expected = ResourceAccessException.class)
    public void serviceShouldPropagateException() {
        restEndpointShouldAnswer(BAD_HTTP_PARAMS, (invocation -> {throw new ResourceAccessException("I/O error");}));


    private void restEndpointShouldAnswer(Map httpParams, Answer> response){
        when(restTemplate.getForEntity(FACT_URL, ChuckNorrisFactResponse.class, httpParams)).thenAnswer(response);

Using fields and moving the instantiation of the class under test into the test setup might not be advantageous in every case but we cannot deny that it removes even more repetition. Also, the restEndpointShouldAnswer() method looks cleaner without a return value.


An important point we should keep in mind when writing tests is to make clear what their intention is, i.e. what we actually want to test. If we cannot clearly see what the test actual does and asserts, it will be hard to change the test in the future. Additionally, it can be hard to check whether the class under test is thoroughly tested. Using Lambdas to refactor mocking and to extract duplicated code helps us to improve the test structure as well as the readability.

The post Improve your test structure with Lambdas and Mockito’s Answer appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

Continuous Integration with Drone on AWS

codecentric Blog - Wed, 14-Feb-18 01:20


Drone is a Continuous Delivery system built on container technology and written in Go. It uses a simple configuration yaml, with syntax similar to a docker-compose file, to define and execute pipelines inside Docker containers.

In this article, I show how to set up a Drone build server, running on an AWS EC2 instance, which connects to your (private) GitHub repository and builds and tests your code after each commit.


You need the following for the example setup:

  • A basic understanding of continuous integration
  • Basic knowledge of AWS as well as an AWS account
  • Basic knowledge of docker
  • A github account
  • A golang project (“Hello World” will do), alternatively use the example Java .drone.yml at the end of this article in conjunction with a simple Maven/Java project

Why Drone?

Let’s take a look at the following .drone.yml file, capable of building a simple Go project:

    image: golang
    - go build
    - go test

That’s it already. Intriguing, isn’t it? Of course there aren’t any notifications for when the build fails at this stage. But this file allows you to get started. Yet, how does it work? With Drone you basically put everything you need to build your software in a Docker container and execute the commands to do so. In the case above, we just need the golang binaries to build our go application; so the standard golang docker image is sufficient. The project is then built and tested with go commands. If any of the commands returns with an exit code > 0, the build fails. Drone took a good look at Travis config file syntax but unlike Travis, Drone can be installed on any system of your choice and doesn’t have to be exclusively used as software as a service. The Drone Team is also planning a fee-based concept, but as it is an OpenSource project under Apache licence, you are free to set it up in your infrastructure.

Let’s begin setting up the whole system.

Setup EC2 instance

We launch an EC2 instance with the default AWS Linux image into a public subnet with a security group open for the following tcp ports: 22, 80, 443, 9418 and 8000. Next, we ssh in, and continue with the Docker installation which is described here: AWS docker on EC2 documentation.

Afterwards, we continue the installation process with docker-compose.

Install docker-compose

still in ssh:

sudo su
curl -L "$(uname -s)-$(uname -m)" > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
docker-compose —version

Sometimes it is necessary to logout and get back to the EC2 instance to verify the installation. Now we are set AWS-wise, let’s continue with setting up the repository.

The build file for your project

Our example project is a golang project – pretty easy to build, as you have seen in the introduction. Just add a .drone.yml file into the root directory of your go project with the following content:

    image: golang
      - go build
      - go test

The pipeline will fail if any of the commands returns a nonzero exit status, indicating failure. The next step brings repository and pipeline together.

GitHub OAuth consumer

We need to create a new OAuth consumer with a callback URL, to enable Drone to connect to our repositories. In your GitHub account, go to Settings —> Developer settings —> OAuth Apps —> Register
Fill in the form with sensible information and provide the following callback url:


as website URL you can enter the following:


In our case the drone-port is the exposed port from the docker-compose file, which will be 8000. Adapt the AWS security group for the EC2 instance to allow custom tcp port 8000 and http. It should look like this:
GitHub OAUTH configuration

Now we need to provide a docker-compose file on the EC2 instance, which will start the Drone server and the Drone agent. On the EC2 instance terminal session, create a docker-compose.yml file with the following content and replace values in the square brackets:

version: '3'

    image: drone/drone:0.8.1
      - 8000:8000
      - 9000:9000
      - ./drone:/var/lib/drone/
    restart: always
      - DRONE_OPEN=true
      - DRONE_HOST=
      - DRONE_GITHUB=true

    image: drone/agent:0.8.1
    restart: always
      - drone-server
      - /var/run/docker.sock:/var/run/docker.sock
      - DRONE_SERVER=drone-server:9000

Complete DRONE_GITHUB_CLIENT and DRONE_GITHUB_SECRET with values of the GitHub OAuth application you created. You can use an arbitrary value as DRONE_SECRET, e.g. generate one with the following command:

openssl rand -base64 32

Make sure that the secret is the same for the agent and the server. After that, run the stack with:

docker-compose up

Wait till Drone and the agent have started and can communicate with each other. Relevant log output should be visible in the EC2 terminal. Afterwards visit your EC2 instance at:


Allow access to your account; you can now activate your repository. The only prerequisite for activation is a valid .drone.yml within the root directory of the project. Upon activation, Drone automatically adds webhooks, which will trigger a build every time code is committed to your repository. Manual configuration is not required.

Repository activation

Now you can commit into your repository to trigger a build and monitor the progress in your browser.

Dry run

An additional “nice to have” feature is that you can also „dry“ build your software with the Drone command line tools. Installation e.g. with homebrew:

brew tap drone/drone
brew install drone

The following command executed in the directory with your .drone.yml file will run the build locally, allowing you to check before submitting code to your source control repository:

drone exec

What about Java and other languages?

What do you need to build your Java project? Most of the time you just need Maven, and the „internet“. So here’s the .drone.yml file:

    image: maven:3.5-alpine
      - mvn clean package

I think this is pretty neat. What’s more, you can provide any Docker image you want, either from your own registry or from Docker hub, and use the specific tag in your .drone.yml.

Of course if you want to deploy in addition to build, things get a bit more complicated, as for secrets and configuration and so on, but that’s another story.
Furthermore, having a Drone EC2 instance running unprotected in public might not be a good idea. For a more secure approach you could, for instance, configure Drone behind a nginx proxy.


To expand your build pipeline and offer continous deployment, you can customize Drone with a variety of plugins to integrate further functionalities into your pipeline. There already are plugins for the AWS build stack, Docker, Kubernetes as well as notification plugins for slack and google mail, just to name a few. In addition you can write your own plugins in bash, go, node and python.


Drone has quite an elegant approach to building your software – the simplicity of the build file struck me straightaway. Although still in beta, you should definitely keep an eye on Drone in the future.

The post Continuous Integration with Drone on AWS appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

Deploying Spring Boot Applications in the Google AppEngine Flex Environment

codecentric Blog - Mon, 12-Feb-18 23:00

In this tutorial I will show how to set up a deployment of Spring Boot applications for the AppEngine Flex environment in the Google Cloud infrastructure.


You should be familiar with the Spring Boot ecosystem and should be able to use Maven as a build tool.

Preparations for Google AppEngine Development

There are several general steps to go through in order to be able to deploy an application to the AppEngine environment.

  • Create a Google Cloud account and set up billing.
  • Create a new project. Accept the proposed project ID or choose your own. We will use the expression [PROJECT_ID] for the rest of this tutorial.
  • Download and install the GCP Cloud SDK.
  • Initialize the Cloud SDK so it points to your GCP project:
gcloud init [PROJECT_ID]

Spring Boot Demo Application

With the new AppEngine Flex Environment you are no longer restricted to the Jetty web framework in order to run web applications. So there is no need to tweak your Spring Boot dependencies in order to run in the GAE any more. (Check the differences between the standard and the flex environment.)

Our first Spring Boot application will feature a very simple REST controller:

public class HelloController {

    public String hello() { return "Hello AppEngine Flex"; }

Since the AppEngine performs a health check by default, we are using the actuator starter and bind it to the required path in src/main/resources/application.yml:

   context-path: /_ah
    enabled: false

For the sake of simplicity we are using the GAE legacy health check that will perform a GET request to /_ah/health.

AppEngine Flex Deployment

The AppEngine deployment descriptor has to be located in src/main/appengine/app.yaml. These are the most important settings:

service: sb-gcp # Id of the service
env: flex       # Flex environment
runtime: java   # Java runtime
  jdk: openjdk8 # use OpenJDK 8

Your project holds several services. Each service needs a unique identifier that is set by the service parameter. Then we configure the flex environment and the Java runtime.

In order to deploy the Spring Boot application, we are using this Maven plugin:

Now we are ready to build a deployment:

mvn clean package appengine:deploy

The last Maven goal triggers the deployment which in the end is performed by the gcloud command line client from the Cloud SDK. This will run for some time. Basically, a Docker image containing your Spring Boot JAR is built and deployed to the GCP. After deploymemt is finally finished, your application will show up in the

AppEngine Dashboard

Each deployment creates a new version of a service. Our AppEngine dashboard for the service named sb-gcp may look like this:

appengine-flex versions

You can start and stop your service, go back to older versions etc.

And finally we see our running application at


That’s the output from our simple REST controller:

curl https://sb-gcp-dot-[PROJECT_ID]
Hello AppEngine Flex

What’s next?

In upcoming posts I’ll show you how to access backends in the Google Compute Engine from Spring Boot applications.

The post Deploying Spring Boot Applications in the Google AppEngine Flex Environment appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

Zooming on lenses, with lenses in the JS land

codecentric Blog - Mon, 12-Feb-18 08:32

Functional lenses or simply lenses are functions that point to a specific part of the data structure. Lenses help us to focus on just the individual entity in the complex data structure by providing a functionality for accessing as well as updating the entity without mutation of the original structure.

Let’s get started.

const object = {
    property: 'my value'
} = 'another value'

The example above shows the way how commonly we mutate the object state, and there is nothing wrong with this approach, but if we want to stay pure this definitely is not a way to go. By pure I mean to avoid dirty mutations.

Let’s compare this example with the lenses one.

const object = {
    property: 'my value'

const valProp = lensProp('property')

const obj = setProp(valProp, 'another value', object)

We have one function to focus on the specific property of the data structure the “lensProp”, and one to set the value on lens property in this case the “setProp” function. And the usage, we just applied the “lensProp”, new value, and the object which we want to update, then the function call just returned the new object with an updated property.

Lenses are so powerful but definitely, more verbose. I would suggest not rushing to use them in every possible situation, they are not the solution to every problem if there is no need for immutability you will not gain much if you use them.

Under the hood of the lenses

We will write the most basic lenses library, just to see that lenses don’t use black magic whatsoever. I would not tell you to do the same in the real world situation because in the npm land there is much more mature and robust implementations. Most of these lens libraries from npm would possibly suit all your needs. Moreover, they tackled even the tiny edge cases which we will ignore in our implementation. I suggest taking a look at the ramda library, ramda includes a ton of pure immutable-free functions including everything you need for lensing.

As what is said in the introduction, lenses provide easy functionality for accessing as well as updating the state of the entity, In other words, lenses are something like a getters and setters function, but much more flexible and reusable.
So, let’s dive into creating these functions, first let’s create a lens function.

const lens = (getter, setter) => ({

The lens one is dead simple, but now we need those two getter and setter functions, also I will use the same names like in ramda library, just that in the end hopefully every example used here can be puggled with ramda implementation rather than using this one.
Now the getter function.

const prop = key =>
  object =>

The prop function will act as our “generic” getter also as we can see we “curried” the function just that we can partially provide arguments.
Now we can do something like this

const objA = {name: 'objA'}
const objB = {name: 'objB'}

// prop will wait for data, the last argument to the function to be executed
const nameProp = prop('name')


After we are done with the getter function, we should implement the setter too.

const assoc = key =>
  value =>
    object =>
      Object.assign({}, object, {[key]: value})

Simple as getter, the setter function will just clone the object with provided new value, so that process of setting the new value stays immutable.

And small example how would we use the setter-assoc function.

const objA = {name: 'objA'}
const objB = {name: 'objB'}

const setName = assoc('name')

  setName('new objA name')(objA),
  setName('name objB')(objB)

Now finally we can feed the lens function with our getter and setter functions, but still, the lens function will be rather worthless if we don’t write a few more functions to work with our lens. We need to write functions for viewing, changing and possibly for applying the function to the “focused” value.

const view = (lens, obj) =>

const set = (lens, val, obj) => 

const over = (lens, fmap, obj) => 
    set(lens, fmap(view(lens, obj)), obj)

They are pretty basic, right?
Then we should try it on the simple example, but before we dive deep into the example let’s take look at how would we use our lense function.

const objLens = lens(prop('property'), assoc('property'))

The prop and assoc that we passed to a lens function are using the same argument, and every time we want to write lens like that we would need to pass the prop and assoc functions. But we can reduce that boilerplate by creating the new function which will cut those two. And as before we will use the same name as ramda does to call our functions

const lensProp = property =>
  lens(prop(property), assoc(property))

Now back to the example :

// plain data object
const object = {
  property: 'some value',
  issue: {
    name: 'nested',
    deep: [
      { name: "Brian", lastName: "Baker" },
      { name: "Greg", lastName: "Graffin" }, 
      { name: "Greg", lastName: "Hetson" }]

// variadic pretty object console.log
const logObjs = (...objs) =>
    objs.forEach(obj => console.log(JSON.stringify(obj, null, 2)))

// curried first class 
const mapOver = fn =>
  data =>

// creat a couple of lenses
const issueLens = lensProp('issue')
const nameLens = lensProp('name')

// first class to string to upper case
const toUpper = str => str.toUpperCase()

const lensOverToUpperCase = lens => 
  str =>
    over(lens, toUpper, str)

const nameToUpper = lensOverToUpperCase(nameLens)

const massagedObject = 
  set(issueLens, over(lensProp('deep'), 
     mapOver(nameToUpper), view(issueLens, object)), object)

// log the result

Looks pretty interesting but still “massagedObject” becomes unreadable due to multiple levels of data, and it contains a ton of repetition, as we can see two times we passed the “object” and “issueLens”. Most of the lens libraries solve this problem by providing the way to see through multiple levels of the data structure, ramda contains a lensPath function which accepts an array of properties, the path to a specific property in the structure. Now we just need the lensPath one, and I promise it will be the last one. Shall we implement that one too?

const lensPath = path => lens(pathView(path), pathSet(path))

And that is it, simple right? But, But, we still need those pathView and pathSet functions implemented, they are just like a prop and assoc functions but they will work with an array of properties instead of single property.

const pathView = paths =>
  obj => {
    let maybeObj = obj;
    paths.forEach((_, index) => 
      maybeObj = maybeObj[paths[index]]) 
    return maybeObj

const pathSet = path => 
  value =>
    object => {
      if (path.length === 0) 
        return value
      const property = path[0];
      const child =, property) ?
        object[property] :
        Number.isInteger(path[1]) ? [] : {}
      val = pathSet(path.slice(1))(value)(child);
     return Array.isArray(object) ? 
        Object.assign([...object], {[property]: value}) :

And now our “massagedObject” from example above could be written in a lot more readable fashion, without repetition, like this

const massagedObject = over(lensPath(['issue','deep']),
    mapOver(nameToUpper), object)

Finally, it started to look elegant and it will do a job

Categories: Agile, Java, TDD & BDD

Integration of Microsoft Visual Studio Team Services, Protractor and Sauce Labs

codecentric Blog - Thu, 08-Feb-18 01:00

Continuous Integration and Automated Testing in combination with cloud testing platforms can prove as a good practice in the web application development life cycle. As a result, these can improve the product reliability and quality. Cloud testing platforms provide a vast number of possible environments to run tests against, execution in parallel to improve efficiency, execution recording and detailed reporting and overall a very comfortable experience for QA teams. Based on an example, this post will explain how to integrate these concepts. The example will use the following technologies: Microsoft Visual Studio Team Services (VSTS), the Protractor end-to-end testing framework, and the Sauce Labs cloud testing platform. The web application in this example is based on Angular and created using Angular CLI.

Protractor configuration

To enable test execution forwarding to Sauce Labs, add the following attributes to exports.config object in protractor.conf.js:

exports.config = {
  sauceUser: /*sauceUser string*/,
  sauceKey: /*sauceKey string*/,

Any specific Sauce Labs options go into the capabilities object inside exports.config.
For example, if there is a need to extend the default Sauce Labs test duration time (30 minutes), then add a property maxDuration with the integer value in seconds.
Also, to enable parallel test execution, add the properties shardTestFiles and maxInstances to capabilities. The property shardTestFiles ensures that tests run in parallel, while maxInstances represents the number of tests running in parallel.

exports.config = {
  capabilities: {
    maxDuration: /*integer value*/,
    shardTestFiles: true,
    maxInstances: /*integer value*/,

To have a test result report generated, add Jasmine JUnit XML Reporter in onPrepare function in protractor.conf.js:

onPrepare: function () {
  jasmine.getEnv().addReporter(new JasmineReporters.JUnitXmlReporter(
      consolidateAll: true,
      savePath: 'e2e/testresults',
      filePrefix: 'testresult-' +

The VSTS build definition will use the resulting file of the XML reporter to publish test results.

Microsoft VSTS build definition

Build definition can be created from existing templates or from an empty process with custom tasks.
Microsoft VSTS offers various build definition tasks and tools regarding build, test, package, deploy, utility, etc.
This example will use npm build tasks to run scripts.

The first script is npm install. It will install a project on the build server and fetch dependent npm modules.

Next is a custom script npm run webdriver-update. It will download and install Selenium server and drivers required for end-to-end tests to run.

Next is also a custom script npm run e2e. It actually calls the Angular CLI command ng e2e. This command will serve the application and run end-to-end tests in accordance with the configuration in protractor.conf.js. It is using the Sauce Labs username and key and is forwarding test execution to Sauce Labs.

Therefore, add following two scripts in package.json file:

"scripts": {
  "webdriver-update": "webdriver-manager update",
  "e2e": "ng e2e",

In addition, the build definition should have a task to publish test results. There is a built-in task for this purpose and it requires the test result format and test result file path. This task will use the resulting file from the Jasmine JUnit XML Reporter, so as a test result format choose JUnit and as test result file path input **\testresult*.xml.

Sauce Labs integration plugin

The command npm run e2e requires the Sauce Labs tunnel to be open to forward the test execution to Sauce Labs.

To integrate VSTS and Sauce Labs, install the additional plugin Sauce for Visual Studio Team Services in VSTS. This plugin can be found in the Visual Studio Marketplace.

After installing this plugin in VSTS, two new build definition tasks will be available: Sauce Labs Configuration (Preview) and Sauce Labs Stop Sauce Connect (Preview).
These two tasks provide VSTS with the actions of establishing and closing a tunnel to Sauce Labs, and they should surround the test execution task (npm run e2e) in the build definition.
Furthermore, VSTS requires Sauce Labs credentials in order to establish a tunnel. To do this, add a new Sauce Labs Credentials service endpoint in VSTS. After that, use the newly created Sauce connection in the Sauce Labs Configuration (Preview) build task.

It is good practice to pass the Sauce Labs username and key as build variables. VSTS offers public variables which will be used for Sauce Labs username. VSTS also offers secret variables which are useful for passwords, keys, etc because VSTS encrypts these variables. Sauce Labs key will be set as a secret variable. Build tasks access secret variables through command arguments (for example $(SAUCE_ACCESS_KEY)).
To access public variables in the Protractor configuration, use process.env and for secret variables use process.argv. So in this example it will be:

exports.config = {
  sauceUser: process.env.SAUCE_USERNAME,
  sauceKey: process.argv[3],


Finally, the build definition should contain these tasks:

  • npm install
  • npm run webdriver-update (Command: custom, Command and arguments: run webdriver-update)
  • Sauce Labs Configuration (Preview)
  • npm run e2e (Command: custom, Command and arguments: run e2e — $(SAUCE_ACCESS_KEY))
  • Sauce Labs Stop Sauce Connect (Preview)
  • Publish Test Results (Test result format; JUnit, Test results files: **\testresult*.xml, Search folder: $(System.DefaultWorkingDirectory))

After the build run is complete, a test report will be available in VSTS. The report will contain detailed information on the test run, pass/fail status, and error messages and stack trace in case of failed tests.

After the tests execution is complete, a report will also be available in Sauce Labs. This report will contain test results, logs, metadata and test report media (screenshots and execution video). The QA team can easily retrieve and use test report media in the further QA process.

The post Integration of Microsoft Visual Studio Team Services, Protractor and Sauce Labs appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

Deep Learning Workshop at codecentric AG in Solingen

codecentric Blog - Mon, 05-Feb-18 23:06

Big Data – a buzz word you can find everywhere these days, from nerdy blogs to scientific research papers and even in the news. But how does Big Data Analysis work, exactly? In order to find that out, I attended the workshop on “Deep Learning with Keras and TensorFlow”.

On a stormy Thursday afternoon, we arrived at the modern and light-flooded codecentric AG headquarters. There, we met performance expert Dieter Dirkes and Data Scientist Dr. Shirin Glander. In the following two days, Shirin gave us a hands-on introduction into the secrets of Deep Learning and helped us to program our first Neural Net. After a short round of introduction of the participants, it became clear that many different areas and domains are interested in Deep Learning: geologists want to classify (satellite) images, energy providers want to analyse time-series, insurers want to predict numbers and I – a humanities major – want to classify text. And codecentric employees were also interested in getting to know the possibilities of Deep Learning, so that a third of the participants were employees from the company itself.

The second day began with a tasty Latte Macchiato and a nice “Good Morning” in the big kitchen of codecentric headquarters. The friendly room, with real moss on the only wall that’s not glass, emanated a very creative air and made the start of the day much easier. For this day, Shirin had a Neuronal Net prepared, that we could use to apply and try out what we’ve learned. We spent a lot of time exploring the effects that changing different parameters had on the performance of the Neural Net. During the practical part, many questions came up, that Shirin answered and discussed with us. We also talked about our own projects and ways to advance them with what we learned in the workshop.

Deep Learning with Artificial Neural Nets

Artificial Neural Nets (ANN) can be used to analyse highly complex data. Deep Learning is a subdomain of machine learning, which is again a subdomain of Artificial Intelligence. ANNs are modelled after the human brain, which has been adapted to learn efficiently and transfer knowledge to new scenarios. Researchers tried to mimic the brains ability to learn by building ANN, which are able to learn from training data and transfer this knowledge to new data.

Gehirn vs Artificial Neural Network

Already back in 1958, Frank Rosenblatt [1] presented his concept of ANNs, the so called perceptron. The perceptron works with a numeric input and uses weighted connections in a node or neuron; together with an activation function, it calculates the result.

With increasing computational power and increasing amounts of data, multi-layer perceptrons and deep neural nets became more and more prevalent. Alongside the boom in machine learning, more and more deep learning frameworks, libraries and programming interfaces were developed. Today, there is a plethora of easy-to-use and open-source materials for deep learning. The bar to get your feet wet with this complex topic is as low as never before; you need almost no deep mathematical knowledge any more in order to build neural nets.

TensorFlow and Keras

TensorFlow and Keras are such deep learning APIs. TensorFlow has been developed by Google Brain and has been written in Python and C++. Right now, it is one of the most often used open-source libraries for machine learning. Keras builds on top of it and works specifically with neural nets. Same as TensorFlow, it is open-source and has been written in Python. There are several other libraries that can be used as backend with Keras, e.g. Theano and CNTK. TensorFlow is based on the concept of data flow graphs, meaning that every neural net can be seen as a form of mathematical operation in nodes and multi-dimensional data objects, the so called tensors, as edges. With TensorFlow we can use the integrated TensorBoard. TensorBoard is a collection of visualisation tools, that makes it easy to analyse processes, potential errors and optimize neural nets.

TensorBoard Beispiel

Image classification with Neural Nets

Most of the research into neural nets has so far involved image classification. The most famous data set for this task is MNIST [2], which contains 28 x 28 pixel images of hand-written images and their respective labels. The MNIST dataset is optimal for getting started with image classification and neural nets. Classification with neural nets falls into the category of supervised learning because it uses examples to train models. The hand-written digits of the MNIST dataset can be used to learn a classification task, namely which pixel arrangement corresponds to which digit.

MNIST Dataset

The neural net will be fed input data from the training set, which it will pass on to the next layer of neurons. The initial weights were picked at random and every neuron in a layer will calculate an output based on the incoming data and the weights. The output will then be passed on again to following layers. In the final layer, a softmax function calculated the probabilities, with which it sorts the input image to the classes. The prediction of this input will be the class with the highest probability. Because we know the correct class for every image in our training set, we can calculate the difference between prediction and correct class. This is called cross-entropy. A cross-entropy of 0 would mean that the neural net predicted the correct class with 100% certainty. After every cycle of training, we calculate an error rate, the so called loss, which represents the cross-entropy of the entire training set.

During training, we aim to minimize the loss function. In order to achieve this, the weights of the neurons need to be optimized, e.g. by using backpropagation with stochastic gradient descent. The direction in which to change the weights is found with a gradient; we want to “descend” this gradient to find the smallest error rate. The step size with which we traverse the gradient landscape is defined by the learning rate. We can start with big steps to prevent getting stuck in local minima, but with advanced training we want to reduce the learning rate so that we can – ideally – reach the global minimum. With a loss function of 0, we would have achieved the perfect result.

The final evaluation of our neural net’s performance is measured on independent test data. We want to avoid that our net learned the training set too specifically and isn’t able to generalize any more. There are many different parameters that can be adjusted when training a neural net to find the optimal composition for our classification tasks. We try out different combinations and observe the effect they have on the result.

I want to thank Shirin for a great workshop! You made the complex topic of deep learning and neural nets much more accessible to us.

Maria Hartmann

Workshop description:

[1] Rosenblatt, Frank (1985). The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain. Web:

[2] LeCun, Yann. The MNIST Database of Handwritten Digits. Web:

Guest author Maria Hartmann studies Digital Humanities at the University of Trier. Through her love of linguistics she came to the classification of semi-structures texts with neural nets – which she now works on for her Master’s thesis. TensorFlow and Keras help her in her studies.


The post Deep Learning Workshop at codecentric AG in Solingen appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

How development for AWS changes the local setup

codecentric Blog - Fri, 02-Feb-18 03:56

When a new project is set up in the cloud these days, this usually means that AWS is used, that the backend is split into multiple services (e.g. microservices), the frontend and backend communicate using REST and multiple managed AWS services are used. Such a setup allows to easily leverage many services AWS provides and reduces the effort and time to implement the desired functionality. To fully use this potential, the development process has to be reviewed and adjusted.

In the old days, a single monolith was developed locally on the developer computer. Dependencies like database, message queue or SMTP server were mostly installed locally or, if this was not possible, mocked or a central installation was used. In most cases the developer could be certain that if the application runs on his computer it will also run on other environments (e.g. test stage or production environment). This approach is not feasible with how applications are developed these days. The differences stem from multiple sources:

  • Splitting the application into multiple parts (e.g. microservices) creates a fully distributed application in which the glue which connects the parts is becoming more important. One side effect is, for example, that the glue also has to be tested. Another consequence of a distributed application is that the application becomes unwieldy in the way that multiple programs have to be running to experience the full application. The development team also has to think about distributed failures and aspects like resilience are getting important.
  • Using cloud services introduces new APIs and a new way of thinking. So, for example, the database is not just there, but you also have to think about request capacities for your database access.
  • Using AWS especially adds the burden of not being able to install the cloud solution on your local computer(s). Other solutions like Kubernetes allow you to do this. While not a real limitation these days, you lose the ability to develop offline. More important is that the development becomes more expensive since these AWS services also have to be paid for the developers, and this sometimes means that the cost will scale with the number of developers.

DevOps can solve the first two issues, but the last one could be described as vendor lock-in.

Let’s take a look at some scenarios and assess them. In the following, “stage” means a self-contained version of the software under development. This could be (for AWS): a set of EC2 instances, an ECS cluster, a unique URL, a set of DynamoDB tables identified by a prefix, a set of CodePipelines and other AWS services.

Use a shared dev stage

In a very easy setup, there is one stage for development. If you want to do a change, you commit and push your change, wait for the build and deployment and then test your change. While it’s easy to get this setup running, it has certain drawbacks:

  • The long turnaround times since you have to wait for the build and deployment. This usually takes a few minutes.
  • Debugging your services is cumbersome or impossible.
  • Since AWS CodePipelines do not support branches, the commit has to be performed in the master. This may be O.K. if the team develops in the master using feature toggles. This, however, prevents other teams to use feature branches and perform code reviews using pull requests.
  • Potential problems because of conflicts between developers which use the same code areas and the same stage.

Use personalized stages

One idea to reduce the drawbacks it to create a stage for each developer. This also means the code pipelines have to be duplicated and adjusted to listen to developer-specific branch names. This allows us to review the code with pull requests in addition to developing independently. This is especially important if there are incompatible code changes in the backend services or changes in the state itself. However, there are some existing drawbacks and a new one:

  • Long turnaround times
  • Debugging is hard or impossible
  • Extra costs for each stage

Run application services locally

To improve the turnaround times and allow for debuging of the application services easily, it is possible to run the application services locally. This may mean that only the one service under development is started or that all services are running locally. While it’s easy to run the application services locally, it’s not that easy or even impossible to get the AWS services (i.e. SQS, S3) running on the local machine. So the AWS services of one stage have to be used. While this setup can be combined with one shared dev stage or with the personalized stages mentioned above, the shared dev stage has two disadvantages:

  • If there are structural changes to the AWS services or the data stored / transferred extra effort is required to ensure that other developers using the same stage are unaffected. It may not be worth the effort and a personalized stage is the better solution.
  • There may be too little control over the execution: if during the test one application service adds a message to a queue (e.g. SQS) and another locally running application service is the consumer of this queue, it is not guaranteed that the locally running consumer will get this message.

Another aspect is how these application services are reached:

  • Accessed by the frontend or other application services: This is usually through a load balancer. In such a case a locally running load balancer replacement is required. We have successfully used nginx for this with a script which converts AWS load balancer rules into a nginx config file.
  • Activated by AWS services: An example here is SQS consumption. This is a problem with a shared stage but works for personalized stages.

In general this approach is possible and is recommended if the conditions are right (low frequency of structure changes, either no AWS service activation or personalized stages are used).

Running as much as possible local

It is possible to run more, in addition to the load balancer, AWS services locally. This allows to further reduce the dependency for AWS services during development and reduce the costs. Besides DynamoDB which is available for local installation the other AWS services are not available for local installation. One can view a replacement software as locally running AWS service or one can also see it as a mock for the real AWS service. This also depends on how sophisticated the service implementation has to be to successfully run the application on the local computer. In general it also means that the fact that the application works locally doesn’t mean necessarily that it also works on a full AWS stage.

There are replacements for multiple AWS services:

  • moto: Originally a mocking library but also allows to start as a separate server.
  • localstack: “A fully functional local AWS cloud stack. Develop and test your cloud apps offline!”
  • Eucalyptus (doc): complete private cloud implementation which claims to be API compatible with AWS.

Out of these three Eucalyptus looks not active maintained. It also looks like the owner of the backing company changed recently. LocalStack uses moto but aims to provide a more complete solution. There is also the beginning of a commercial company backing LocalStack.

For specific services (an older list):

  • For Kinesis:
    • kinesalite: “An implementation of Amazon’s Kinesis built on LevelDB”
  • For S3:
    • minio: “Minio is an open source object storage server with Amazon S3 compatible API. “ It supports multiple underlying storage providers.
    • ceph: “Ceph is a unified, distributed storage system designed for excellent performance, reliability and scalability.” It also offers an S3 interface.
    • Fake S3: “A lightweight server clone of Amazon S3 that simulates most of the commands supported by S3 with minimal dependencies”
  • SQS:
    • elasticmq: “Message queueing system with an actor-based Scala and Amazon SQS-compatible interfaces”
    • goaws: “AWS (SQS/SNS) Clone for Development testing”
  • Lambda and API gateway:
    • serverless-offline: “Emulate AWS λ and API Gateway locally when developing your Serverless project”
    • emulambda: “Python emulator for AWS Lambda”


There are surprisingly few experience reports about local AWS development. LocalStack, on the other hand, was developed by Atlassian and while there is no official press release it is reasonable to assume that they felt a problem and developed a solution for it.

If money is not a concern, the recommendation is to provide a personalized stage for each developer and run the application services locally. In such a case a stripped down stage without the active application services is enough.

If the number of developers is larger or a stage consists of many costly elements, the recommendation is to try to use replacements which run locally, for example LocalStack. If there is a setup which avoids AWS services, you may also want to consider using this setup in CI/CD. Still plan for the case that the real AWS services may behave differently.

The post How development for AWS changes the local setup appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

AWS CodePipeline Dashboard

codecentric Blog - Mon, 29-Jan-18 23:42

In modern stacks, the application often consists of multiple microservices. For one of our projects we had to manage about 20 of these. Not only because of the partnership between codecentric and AWS we decided to use the AWS platform. That also led to the decision to use infrastructural components provided by AWS like CodeCommit and CodePipeline. In this blog post we want to speak about our experience with CodePipeline.

Unfortunately, the web interface of CodePipeline is rather basic. It is difficult to get an overview which pipelines are running, building or failing when you have a pipeline for each microservice. Failing? Yes, that happens! And that is exactly what we want to see immediately. To summarize: We found the existing web interface insufficient to help us in our daily work and looked at alternatives.

Before we dive into the graphical user interface (UI) we prefer, we want to mention alternatives to visualize pipeline issues. Sometimes you may prefer getting an email or notification on Slack when something goes wrong. This kind of feedback can be implemented by using the CodePipeline API.

Why do we prefer a graphical UI to visualize our set of pipelines? One reason is that human beings can easily work with graphical representations and their changes. The latter one is the reason that makes GUIs so intuitive. The following example comes from our daily project work. Please note that some information has been blurred to protect the privacy of the customer. Imagine a screen like this one:

Do you see anything special? No? You’re right! This screen shows a pipeline system without any problems. It shows all implemented pipelines with their build steps, the last date of being built and their status in colours. Green means that everything is okay. Just imagine this on a big screen: a short glance tells you “Everything is green, no problems.”. Great!

Now check this screen:   

Ah, there is something different: First of all, there is this red bar in the middle. This project needs your attention. The build step failed. 

And of course you also noticed the blue bar on the bottom. Well, it’s not that eye-catching. That’s by intention: there is a build in progress. The takeaways are: Everything is fine, the pipelines are working and something is currently working there.

Summary: you can see this information with one glance. Just react on red bars and the blue bars show that the whole system is alive.

Of course, we don’t want to just show you this tool in a blog post. You can also use it. The application is available on github.

We also provide a docker container for this application on docker-hub.  Please read the instructions in our README file for further information.

The post AWS CodePipeline Dashboard appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

DRY in the 21st Century

codecentric Blog - Mon, 29-Jan-18 02:30

It seems that nowadays, the “Don’t Repeat Yourself”(DRY) principle is one of the foundations of programming that is criticized the most. You can find tweets and blog posts questioning it. Also, it seems that critical voices are increasing.

But why is this happening right now? DRY, amongst other principles, is often used when distinguishing good from bad code. To me it was taught at university as an extension of SOLID (making it SOLIDD basically). The principle itself is easy to understand and apply. Examples showing the importance of the principle can be found all over the Internet. Still, there is some rejection.

I would like to offer my opinion here and state why I do not agree with the critics completely. But first, let’s start by looking at where and when the DRY principle was formulated.

The Pragmatic Programmer

As Wikipedia tells us, DRY was firstly formulated in “The Pragmatic Programmer” [1]. In chapter 7 “The evils of duplication” (pp. 26-33) the DRY principle is formulated as

“Every piece of knowledge must have a single unambiguous, authorative representation within a system.”

The authors also mention that

“[…] it is one of the most important tools in the Pragmatic Programmer’s tool box.”

As pointed out by several people, the principle is about “knowledge” – not code, not classes, not methods, not functions,… The principle is basically agnostic of code. But if we keep on reading the book we see that for the authors code is also a form of knowledge; next to comments and documentation. So, whoever points out that DRY is not about code at all did not read the complete chapter. Or, as programmers like to say:

The truth is in the code.

And therefore the knowledge is in the code.

Following reasons for duplications and their explanations, the authors point out that we should “Make it easy to reuse” our code. Regarding the structure of the book, this tip (all of the advice in the book is presented as tips) is as important as DRY, it just did not receive such a nice abbreviation (MIETR is hard to pronounce I guess). So, for the following discussion, I want to look at DRY and MIETR.

Historical Context

Talking about the Pragmatic Programmer, it is helpful to take a look at the time when it was written. Published in 1999, it will see its twentieth birthday next year. Regarding the rapid development in Computer Science 1999 might not be the Stone Age, but something like the Renaissance. Since I was more interested in my PlayStation back then, I asked my older colleagues what some of the hot topics those days were:

  • OOP
  • JavaScript
  • PHP
  • J2EE
  • CVS

Next to the technological aspects, there are also some other things worth mentioning:

  • Java 1.2 was the most recent Java version.
  • The Internet Bubble was still rising.
  • Netscape lost the Browser Wars.
  • The Apache Foundation was founded.
  • Apache Ant and Maven were not invented yet.

So what can we say about this time? The majority of applications were monolithic. Vertical scaling dominated over horizontal scaling. OOP was probably seen as the solution for all problems (which lead to huge class hierarchies, which we try to solve now by favoring delegates over inheritance). Building/packaging applications was most probably a bigger issue than it is nowadays.

All in all, I would say that reusing code, either within a project/company or outside of it, was a much bigger problem in 1999 than today. No Apache, no GitHub, SourceForge just started. I do not want to imagine having the problems of unreliable dependencies that you cannot look into because some closed-source library was bought from some vendor. Copying everything into big /lib directories is also an issue from the past.
In this context, having code that is easy (and fun) to reuse might have been a rare occasion. Therefore, raising developers’ awareness of MIETR seems more important than advising them to follow the DRY principle.

Having seen the past, let us go back to today.

Microservices, DDD, Serverless…

What we can easily agree on is that times have changed since 1999. Nowadays, we have Microservices or even Pico- and Nanoservices. Using a serverless approach for an application is also rising to the top on the Gartner Hype Cycle. Splitting your application into its domains and bounded contexts is also more mainstream these days.

From my point of view, as Microservices rose up, skepticism about DRY increased, too. Sharing code between different Microservices is considered bad (not without reason, of course). But still, extracting code into a shared library also encounters resistance, even within a project. So, the solution is obvious: code has to be copied. And this violates the DRY principle. One could argue that by trying to make DRY irrelevant, developers only try to retrospectively justify their code duplication, since they were either too lazy or too ignorant to create a shared library. But I think this argumentation falls short.

Philosophically put, the problem might be within our education as developers. Everything is binary, DRY is either ON or OFF. There is no 30 % DRY. MIETR is not (easily) measurable, and what we cannot measure, we cannot control and therefore is ignored.

Having the current context and a philosophical thought, let’s take a look at some concrete examples.


In this section we will take a look at two examples to see how DRY helps us keep our code clean. Hopefully, I can convince you that the application of DRY, at least in those examples, is advantageous.


A class that I have used a lot recently is the UriComponentsBuilder. This class helps us transform a URL with placeholders like http://{host}:{port}/{username} into its expanded form http://localhost:8080/admin. Additionally, it can add query parameters to the request.

If I was not using this class, I would have to implement my own string replacement logic. And this replacement logic would have to be copied across all places where I would need a URL with placeholders. Changing e.g. the markers of a placeholder to something different than braces would require a lot of changes in the code. Of course I could create my own Helper/Util/… class that contains the replacement, but this would be DRY, nearly as DRY as using the Spring class.

But why would I want to violate DRY here? This class serves me very well. MIETR is also “high” because the API is well documented and I can use the class without looking into its internals.

Table Name

Another example would be a table name that I need in my code. Following the recommendations from The Pragmatic Programmer, I would create a constant somewhere in my code (or even extract it from the SQL file that creates it) that contains the table name. Whenever I have to access the table, I use the constant. If the table name changes, there is only a single place where the code has to be changed. Again, simply following DRY. Without this constant, several files would need to change in case of a renaming. This would be quite error-prone because I could easily forget a place to change it (and according to Murphy’s Law, I would). Importing a constant somewhere in my code is also quite easy to do, so MIETR is taken into account, too.

Putting DRY into Context

I hope that you agree with me so far on the DRY principle.

Of course you might argue now that those are only technical examples. The UriComponentsBuilder is a helper regarding my HTTP communication and a table name is also technical, nothing businessy. But still, they represent knowledge. The knowledge about what placeholders in my URLs look like and the knowledge about the table name.

So, to put it in a more abstract way, within the context of URLs, the UriComponentsBuilder is my single source of truth for replacements. And within the context of my database, the constant with the table name is the single source of truth for that name. I got to admit that my knowledge in this area is rather limited but this sounds like a Bounded Context to me.

So, what if – and only if – we should no longer apply DRY “globally”, but rather within a Bounded Context?

Why DRY is still relevant

In my opinion we will return to darker times if we start praising the rejection of DRY as being a trait of experienced programmers. The principle is still relevant and should be taken into account when writing code.

What I also think is that we should adjust DRY to the current times. If your shopping application needs to represent a customer in the context of purchase and in the context of shipping – go on, copy it. Create two customer representations. As the DRY opponents state, those two representations will differ more and more over time and forcing it into the same class for the sake of DRY would cause more harm than good. Or, as the author of “Domain Driven Design” [2] stated (p. 344):

“Code reuse between Bounded Contexts is a hazard to be avoided.”

Also, if I have to instantiate an object with parameters I only use in half of my application or which I set to null just because something has to be passed into the constructor, it is no convenient reuse. It is rather a burden and a possible source of errors.

Blindly applying DRY is as false as ignoring it. I personally like the rule of thumb that if something occurs identically three times or more, then it should be refactored and stored in a common place.

To summarize, apply DRY within your context or unit of deployment. Don’t DRY out everything up front but rather look for reccurring patterns. And if you extract and abstract something, keep MIETR in mind.

Lastly, to repeat myself: Don’t repeat yourself within your context.


[1] Andrew Hunt and David Thomas. 2010. The Pragmatic Programmer: From Journeyman to Master. 25th Edition. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.
[2] Evans. 2004. Domain-Driven Design: Tackling Complexity in the Heart of Software. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.

The post DRY in the 21st Century appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

Different ways of unsubscribing from RxJS Observables with Angular

codecentric Blog - Tue, 23-Jan-18 03:55

Angular uses RxJS as a backbone of the Angular application. RxJS uses the concept of Observables and Observers, where an Observable is a source of data and Observer is the one who use the data. Every time an Observable produces new values, it informs an Observer and the Observer handles those values inside subscribe operator. Dealing with Observables can be dangerous because there is the possibility of creating a memory leak. How’s that even possible?


Every time a component/directive is destroyed, all subscriptions to custom Observables still remain active.


Manually unsubscribe from all custom Observables when a component/directive gets destroyed. The best place to unsubscribe is inside functions that handle the OnDestroy lifecycle hook. Some subscriptions like router and http don’t need manual unsubscribe, for the rest of them there are various solutions:

  • execute unsubscribe over the subscription object
  • using takeUntil operator
  • using async pipe


export class UnsubscribeCardComponent implements OnInit, OnDestroy {
  message: string;
  subscription: Subscription;
  constructor(private upperCaseService: UpperCaseService) {}

  ngOnInit() {
    this.subscription = this.upperCaseService.getUpperCaseMessage()
      .subscribe((message: string) => this.message = message);

  ngOnDestroy(): void {this.subscription.unsubscribe();}

Subscription represents a disposable resource, it has an unsubscribe method that can be used to dispose the resource held by the subscription.


export class TakeUntilCardComponent implements OnInit, OnDestroy {
  message: string;
  private unsubscribe$ = new Subject();
  constructor(private upperCaseService: UpperCaseService) {}

  ngOnInit() {
      .subscribe((message: string) => this.message = message);

  ngOnDestroy(): void {

TakeUntil takes a second Observable as an argument, it monitors the second Observable and discard subscription after it emits a value or terminates.


export class AsyncPipeCardComponent implements OnInit {
  messageSubscription: Observable;
  constructor(private upperCaseService: UpperCaseService) {}

  ngOnInit() {
    this.messageSubscription = this.upperCaseService.getUpperCaseMessage();

{{messageSubscription | async}}

The async pipe subscribes to an Observable and returns the latest value it has emitted. When a component is destroyed, the async pipe unsubscribes automatically.


When a component/directive is destroyed, all custom Observables need to be unsubscribed manually. Async pipe is a good solution, because it does everything automatically, but keep in mind that it can’t be used in all scenarios.
The RxJS lead developer Ben Lesh recommends the takeUntil approach. He explains his arguments in this article.

The sourcecode for the complete example can be found here.
The demo example can be found here.

The post Different ways of unsubscribing from RxJS Observables with Angular appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

Implementing a simple Smart Contract for Asset Tracking [blockcentric #6]

codecentric Blog - Thu, 18-Jan-18 00:27


Our article series “blockcentric” discusses blockchain-related technology, projects, organization and business concerns. It contains knowledge and findings from our work, but also news from the area.We are looking forward to your feedback on the column and exciting discussions about your use cases.

In previous posts we gave an overview of the challenges of supply chains, an overview of a blockchain-enabled supply chain platform, as well as a  hardware implementation for production integration. Now we would like to give some examples for implementing smart contracts that allow asset tracking in the blockchain.

A simplified asset tracking case

In our simplified case we would like to track assets throughout the supply chain. In reality, the data structures for the assets might be more complicated, but we will focus on a few aspects only: name, description and manufacturer. It should be guaranteed that there is only a single instance of an asset at any time which leads to a uniqueness requirement of the ID. We would like to keep track of origin and history of the asset. The key actions that should be supported are the creation of new assets as well as transfers of assets. Furthermore, there should be a possibility to check whether a person is actually in the possession of the asset ownership certificate or not.

The implementation environment

As an easy starting point to implement smart contracts, we use the Ethereum blockchain. The key feature of Ethereum is the ability to execute programs on the nodes of the blockchain. It can be seen as an application platform that runs contracts as unstoppable programs on the blockchain. Ethereum has its own virtual machine (Ethereum virtual machine) that executes programs on all nodes the same way. The programming language that defines these programs is called Solidity, which is a object contract-oriented, high-level language for implementing smart contracts. Solidity supports inheritance, libraries, complex types and reads quite similarly to object-oriented programming languages, such as C++, Python or JavaScript.

Once you have implemented your smart contract functionality, you are ready to either test the contract locally, by deploying it to a test blockchain such as truffle-ganache or to an Ethereum test network such as the Rinkeby test network. Both options will be covered by follow-up articles. The interaction with the contract is done by using either command-line tools such as geth or via web frameworks such as web3.js. To manage your wallets in your browser, you would need a wallet store such as Metamask that connects to either your local or a public test chain as well as the Ethereum network.

Structure of the smart contract

Let’s have a look a the basic structure of a smart contract written in Solidity.

pragma solidity ^0.4.2;
import "assets.sol";

contract AssetTracker {
    string id;

    function setId(string serial) public {
          id = serial;

    function getId() public constant returns (string) {
          return id;

The contract file opens with a section on the applicable Solidity language version. Pragmas are known from compilers such as gcc. They give the compiler additional information on how to process the given code. This given pragma tells the compiler to use the ‘^’ highest available language version above ‘0.4.2’.

Furthermore, you are able to import code from other files at the global scope level by using the well-known ‘import’ statement, just as in most other languages.

In this example we have added some public getter and setter functions that can be used both as an internal and as an external function. Beyond this, there is the possibility to create events along the way that can be used to log processes or states. We will make use of the events in order to keep track of supply chain events such as goods receipt or asset creation.

A brief excursion to types and costs

The static types that are available, such as string, int, bool …, will come with the typical unary and binary operators. You can find a full list of types here. You shouldn’t be lazy about learning and using the types properly, because this will have impact on the running cost oft your transactions. The transaction effort, described in the unit gas, will depend on the operations executed and the data stored. This effort is priced by you. If you pay more for the execution of your code, your transaction will be preferred by the network, hence executed sooner.

If you want to drill down into the economics of transactions, you should have a look at ‘Calculating Cost in Ethereum Contracts‘ and ‘Ethgasstation‘ to get a feeling about the associated costs with your smart contract. The details of gas cost for opcodes, e.g. the formerly used SSTORE opcode, can be found in the Ethereum yellow paper. Another way to have a look at the cost is to use the online Solidity compiler that allows you to have a look at the Ethereum Virtual machine code  generated from your contract. It estimates the gas price and reveals the inner workings of how the different data types are handled on the stack.

Tracking data structure

The following  struct describes our simplified asset.

struct Asset {
    string name;
    string description;
    string manufacturer;
    bool initialized;    

We use members such as describing properties such as name, description and process control variables such as initialized and manufacturer. They are used to check whether this asset was already manufactured and who the manufacturer is.

In order to store the assets, we create two mappings that will allow us to store asset properties as well as the relation between assets and wallets based on asset uuids.

mapping(string  => Asset) private assetStore;

is used later on to store assets under their respective uuid:

assetStore[uuid] = Asset(name, description, true, manufacturer);

For the wallet store, we use the following mapping:

mapping(address =>; mapping(string =>; bool)) private walletStore;

is used later on to make the assignment of an asset to a wallet

walletStore[msg.sender][uuid] = true;

Declaring the events

For different real-world events in the supply chain, such as asset creation or asset transfer, we define counterparts in the smart contract.

event AssetCreate(address account, string uuid, string manufacturer);
event RejectCreate(address account, string uuid, string message);
event AssetTransfer(address from, address to, string uuid);
event RejectTransfer(address from, address to, string uuid, string message);

Declaring the functions

The first function that we would need is the create asset function. It takes all information needed to specify the asset and checks if the asset already exists. In this case we trigger a formerly declared event – RejectCreate(). If we have a new asset at hand, we store the data in the asset store and create the relation between the message sender’s wallet and the asset uuid.

function createAsset(string name, string description, string uuid, string manufacturer) {

    if(assetStore[uuid].initialized) {
        RejectCreate(msg.sender, uuid, "Asset with this UUID already exists.");

      assetStore[uuid] = Asset(name, description, true, manufacturer);
      walletStore[msg.sender][uuid] = true;
      AssetCreate(msg.sender, uuid, manufacturer);

In order to transfer the asset, we create a function that takes the address of the target wallet along with the asset id. We check two pre-conditions: The asset is actually existing and the transaction initiator is actually in possession of the asset.

function transferAsset(address to, string uuid) {

    if(!assetStore[uuid].initialized) {
        RejectTransfer(msg.sender, to, uuid, "No asset with this UUID exists");

    if(!walletStore[msg.sender][uuid]) {
        RejectTransfer(msg.sender, to, uuid, "Sender does not own this asset.");

    walletStore[msg.sender][uuid] = false;
    walletStore[to][uuid] = true;
    AssetTransfer(msg.sender, to, uuid);

We would also like to have access to the asset properties by just giving the uuid. Since it is currently not possible to return structs in Solidity, we return a list of strings.

function getAssetByUUID(string uuid) constant returns (string, string, string) {

    return (assetStore[uuid].name, assetStore[uuid].description, assetStore[uuid].manufacturer);


Furthermore, we would like to have a simple way to prove the ownership of an asset without the need to fiddle around the transaction log. So we create a helper function isOwnerOf().

function isOwnerOf(address owner, string uuid) constant returns (bool) {

    if(walletStore[owner][uuid]) {
        return true;

    return false;

Once the contract is deployed, we can interface with the smart contract by using web3.js. The following example is an excerpt for creating a new asset.

export const createAssetInContract = async (assetData, publicKey) =>; {
    console.log('Creating asset...');
    const atContract = await AssetTracker.deployed();
    const asset = atContract.createAsset(,
      { from: publicKey },

    return asset;

This simple example shows a basic asset tracking functionality in the blockchain. There are many ways to improve the design and the functionality, such as on-chain uuid generation. If you want to discuss tracking solutions in the blockchain, feel free to start a discussion.

If you are interested in this topic, please let us know. We would like to hear your comments and amendments. Feel free to drop me a few lines: and follow me on Twitter:

Previously published blockcentric posts


The post Implementing a simple Smart Contract for Asset Tracking [blockcentric #6] appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

BigchainDB – The lightweight blockchain framework [blockcentric #5]

codecentric Blog - Mon, 15-Jan-18 23:00

With BigchainDB we see one of the first complete but simple blockchain frameworks. The project strives to make blockchain usable for a large number of developers and use cases without requiring special knowledge in cryptography and distributed systems.

According to benchmarks (whose scripts are also included in the repository), a simple BigchainDB network is able to accept and validate 800 transactions per second (cf. 3-10 tx/s at Bitcoin). This high data throughput is due to the selected Big Data technologies for data persistence. Database systems from this environment use proven mechanisms based on Paxos algorithms to reach a consensus on the status of the data.
MongoDB or RethinkDB can be selected as the underlying database for a BigchainDB. Both options are document-oriented NoSQL databases, which can be scaled horizontally by replication and shaping. They are also schema-free, so that data can be stored in them without the need to globally define a uniform schema.
The throughput of both systems alone can generally be named with more than 80,000 write operations per second.

Transactions in BigchainDB

The framework makes uses of the consensus of the used database cluster. In order to validate transactions, it is also possible to implement your own validation mechanisms.
Two types of transactions are available in BigchainDB. CREATE transactions create an asset, i. e. a collection of data, and link it to at least one owner (its public key). In addition, all assets are “divisible” and can therefore be broken down into different proportions for different owners.
With TRANSFER transactions, complex instructions can be created that link existing assets (and their shares) to conditions of further transactions. In this way, assets and parts of the network can be easily moved between subscribers in the network.
As usual in a blockchain, there is no way to delete an asset or modify its properties.

For a transaction to be validly processed in the network, several conditions must be met.
Once the transaction has been received, the nodes check it for the correct structure. For example, CREATE transactions must contain their asset data while TRANSFER transactions reference the ID of an asset created. Both types also differ in the way they have to deal with inputs and outputs. Of course, each transaction must be signed and hashed before it is transmitted to the network.
If the structure of a transaction is valid, the validity of the contained data is checked. In short, the so-called “double spending” is prevented in this step. This prevents the transaction from repeatedly transferring the same asset or issuing assets that have already issued other transactions.
In addition, you can implement your own validation mechanisms that could check asset generation and transfers for correct functionality, for example. For example, whether realistic dimensions have been assigned for a car part or whether the color code of an automotive paint job exists.

For a deeper understanding of the idea as well as the technical and architectural details of the project, it is recommended to read the BigchainDB Whitepaper, which was maintained until June 2016.

Scenarios and operation of a blockchain

Since the consensus in a BigchainDB is not implemented via public mechanisms such as Proof of Work, Proof of Stake or similar, the technology is more suitable for private blockchains. This means that some parties will form a consortium in order to jointly execute their transactions among themselves without the need for an intermediary. To this end, each participant of this association adds some infrastructure on which at least one node of the blockchain solution is operated. Therefore, each transaction that occurs in the network must be validated and confirmed by a technical and organizational party of the consortium. This approach is very lightweight and does not require participants to be rewarded for their validations. The reward for the participants is, after all, to build up a trusting network without questionable and costly middlemen.

Due to these circumstances, the operation of a private blockchain based on BigchainDB is relatively easy. Each member of the consortium must take care of setting up and maintaining a database and BigchainDB cluster distributed in its infrastructure. In addition, there is of course the holding of a private key to sign his messages to the network. Each organization participating in the network can be identified and verified by in-built certificate management and registry.

One example would be a merger of a number of banks operating payment transactions and information exchange among themselves. Usually, these participants do not fully trust each other and must involve third parties to verify the transactions. However, if this network of banks were to form a consortium that would automate and cryptographically secure each transaction, a third instance would be superfluous and could therefore be excluded.
With a BigchainDB solution in place, each bank would operate its own cluster in its infrastructure that is linked to the network.

BigchainDB is therefore particularly suitable for private blockchain networks with high activity and data volume. This stack is also suitable for archiving solutions in which many data records have to be stored in a trustworthy way for many years. This can be used, for example, to make instances obsolete that call up high service, hardware and license fees for legally compliant data archiving.
Tracking steps in a supply chain can also cover the Bigchain transaction model excellently.

Getting started on the local machine or in IPDB

As a Managed BigchainDB Blockchain network, the Interplanetary Database (IPDB) is now also offered, with which one can interact as a registered organization.

Locally, Bigchain can either be installed directly on the host or operated with a docker. The docker variant is well suited for starting and testing.

In order to develop client applications against the started network, some official and community-maintained drivers are available. From the wide range of Python, JavaScript, Java, Ruby, Haskell and Go, every developer will probably find the right library.

We wish you a lot of fun trying it out.

Our article series “blockcentric” discusses Blockchain-related technology, projects, organization and business concerns. It contains knowledge and findings from our 20% time work, but also news from the area.

Blockcentric Logo
We are looking forward to your feedback on the column and exciting discussions about your use cases.

Previously published blockcentric-Posts

The post BigchainDB – The lightweight blockchain framework [blockcentric #5] appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

Change Streams in MongoDB 3.6

codecentric Blog - Sun, 14-Jan-18 23:00

MongoDB 3.6 introduces an interesting API enhancement called change streams. With change streams you can watch for changes to certain collections by means of the driver API. This feature replaces all the custom oplog watcher implementations out there, including the one I used in the article on Near-Realtime Analytics with MongoDB.

For a start, we need to install MongoDB 3.6.0 or a higher version. After setting up a minimal replica set we connect to the primary and set the feature compatibility to 3.6 be able to use change streams (this will hopefully be the default in future version):

use admin
db.adminCommand( { setFeatureCompatibilityVersion: "3.6" } )

I will use the Java driver for the following examples. In order the be able to watch for documents changes, we write a simple program the inserts a bunch of document to a collection

MongoCollection eventCollection =
    new MongoClient(
        new MongoClientURI("mongodb://localhost:27001,localhost:27002,localhost:27003/test?replicatSet=demo-dev")

long i = 0;
while (true) {
  Document doc = new Document();
  doc.put("i", i++);
  doc.put("even", i%2);
  System.out.println("inserted: " + doc);
  Thread.sleep(2000L + (long)(1000*Math.random()));

The output of this Java process looks like this:

inserted: Document{{i=1, even=0, _id=5a31187a21d65707e8282fa7}}
inserted: Document{{i=2, even=1, _id=5a31187d21d65707e8282fa8}}
inserted: Document{{i=3, even=0, _id=5a31187f21d65707e8282fa9}}
inserted: Document{{i=4, even=1, _id=5a31188121d65707e8282faa}}

In another Java process, we use the same code for retrieving the collection. On that collection we call a method watch with takes a list of aggregation stages, just like the aggregate operation:

ChangeStreamIterable changes =
    Aggregates.match( and( asList(
      in("operationType", asList("insert")),
      eq("fullDocument.even", 1L)))

We register only for insert operations on the collection and additionally filter for documents with the field even being equal to 1.

Change Streams in MongoDB 3.6

When we iterate over the cursor we just print out the matching documents:

changes.forEach(new Block>() {
  public void apply(ChangeStreamDocument t) {
    System.out.println("received: " + t.getFullDocument());

The result looks like this:

received: Document{{_id=5a311e2021d657082268f38a, i=2, even=1}}
received: Document{{_id=5a311e2521d657082268f38c, i=4, even=1}}
received: Document{{_id=5a311e2a21d657082268f38e, i=6, even=1}}

With change streams, the MongoDB API grows even wider. Now you can quite easily build things that resemble triggers you know from traditional databases. There is no need for external event processing or your own oplog watcher implementation anymore.

The full source code can be found at GitHub.

The post Change Streams in MongoDB 3.6 appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

Gamma-TicTacToe – Neural Network and Machine Learning in a simple game

codecentric Blog - Fri, 12-Jan-18 06:52

This post is about implementing a – quite basic – Neural Network that is able to play the game Tic-Tac-Toe. For sure there is not really a need for any Neural Network or Machine Learning model to implement a good – well, basically perfect – computer player for this game. This could be easily achieved by using a brute-force approach. But as this is the author’s first excursion into the world of Machine Learning, opting for something simple seems to be a good idea.


The motivation to start working on this post and the related project can be comprised in one word: AlphaGo. The game of Go is definitely the queen of competitive games. Before the age of AlphaGo it was assumed that it would take a really long time until any computer program could beat the best human players, if ever. But unlike the predominant chess programs, AlphaGo is based on some super-advanced – the word really fits here – Neural Network implementation. With this it simply swept away any human top player in the world. Depending on the viewpoint this is amazing, sad, scary or a bit of all.

If this is about the game of Go then why is there a video embedded about playing chess? The engine behind AlphaGo has been developed further. Its latest incarnation is called AlphaZero and it is so generic that it can teach itself different games based only on the rules. There is no human input required anymore, but learning is completely performed using self-play. This is really fascinating, isn’t it? AlphaZero had already easily defeated all its predecessors in the game of Go when it was trained to conquer the world of chess. After only four hours (!) of self-training it crushed the best chess engine around, which in turn would beat any human chess player.

So far for the motivation to start this project, which obviously cannot – and is not intended to – even scratch on the surface of what has been achieved with AlphaZero. Though the project name is clearly inspired by it ;-).


Then what should be achieved? Learning about and implementing a Neural Network with some kind of self-learning approach to start with. As the rules of Tic-Tac-Toe are very simple – just play on an empty field – not too much time must be spent implementing the game mechanic as such. This allows focusing on the Neural Network and the learning approach.

Ideally the program should play the game perfectly in the end. This would mean it will never loose to any human player and win if that player does not play the best moves. Tic-Tac-Toe cannot be won by any player if both players are playing decent moves.

The basic – and a little bit less ambitious – objective is that it wins a fair amount of games when playing against a random computer player after some amount of self-learning.

Playing a random computer player

Playing a random computer player is the first assessment of what has been achieved. Then we are going to take a closer look at the implementation, the ideas that did not work and the ideas that worked out in the end.

The complete implementation of Gamma-Tic-Tac-Toe can be found here: That page also includes instructions on how to compile and run it.

Self-play against the random computer player is implemented in a way that allows independent matches with any amount of games. The Neural Network is re-initialized and trained again between two matches. By default each match consists of 10.000 games and 50 matches are performed. All these values are configurable. The amount of training games are of course also configurable as this is an interesting parameter to test the ability of the Neural Network to learn.


The match between two random computer players is used to crosscheck the implementation. It is expected that the results are almost totally even as can be also seen in the following chart.

It is easy to make mistakes when validating the results using self-play. In the beginning the Neural Network was always playing the first move. Of course in a game like Tic-Tac-Toe this has let to wrong results. Having two random computer players playing the game this could be detected as it was quite suspicious that one random player was winning far more often than the other one.

chart showing the results of two random players


The next match is the random computer player vs. an untrained gamma-engine (the fancy name used instead of writing “the Neural Net playing Tic-Tac-Toe”). This is interesting as the matches are going back and forth, but without a clear overall winner or loser. The individual matches are often won quite clearly in comparison to the games played between two random computer players.

chart showing the results of a random computer player vs. an untrained gamma-engine


Now we are having a gamma-engine that is trained in 50 games against the random computer player before each match. It can be seen that the amount of matches won is clearly increasing in comparison to the untrained version. But there are still quite some matches lost, sometimes even pretty clearly.

chart showing results of a trained gamma-engine


With 250 training games things are improving a lot. All but one match is won and often quite clearly.

chart showing results of an increasingly trained gamma-engine


Interestingly the results are pretty much the same as with 250 training runs. This time even two matches are lost. Still it is obvious that the training has a positive effect on playing.

chart showing results from a gamma-engine with 500 trained games


So let’s perform 1500 training games before each match. The result is again not changing dramatically, but there is still some improvement.

charts showing some improvement after 1500 training games


Finally let’s make a huge jump to 15000 training runs before each match. With this amount of training the Neural Network is winning very consistently and on a high level. This result has been double-checked by executing it several times. The same is true for the other results as well.

chart showing high winning rate after 15000 training runs

The journey to gamma-engine stage-1

The results presented in the previous chapter are based on stage-1 of the gamma-engine. The different stages are intended to differ regarding the amount of learning that is applied. This is not related to the number of training runs, but the factors used to learn something about the game. In stage-1 the “learning algorithm” is based on the following factors: If a game is won the first and the last move of that game are “rewarded”.

This “rewarding the right decisions” is a kind of backpropagation that is often used to train Neural Networks. Even though what has been done here seems to be a bit simpler than what is described in that article.

Therefore the output weights of the corresponding neurons triggering those moves are increased. This does not seem to be a lot of learning at all, but it is enough for the results shown above. Of course this is only possible due to the fact that Tic-Tac-Toe is such a trivial game to play.

There are a lot of articles dealing with Neural Networks and Machine Learning. The corresponding Wikipedia page for example is quite extensive. Therefore this article is focusing on the practical approach towards the specific problem at hand and not so much on the theoretical side of Neural Networks. Still we need some theoretical background to start with.

A Neural Network is composed of different layers. It has an input layer, any amount of hidden layers and an output layer. Theoretically each layer can have any amount of neurons. But the amount of input and output nodes are precluded by the data and the task at hand. Thus practically only hidden layers can have any number of nodes. In this implementation dense layers are used where each neuron of one layer is connected to each neuron of the next layer. There are lots of other layer types where this is not the case. The input to a neuron is an input value representing (a part of) the task to be solved and a weight assigned to that connection. By modifying those weights the Neural Network can learn. The following diagram shows an example of such a Neural Network.

Different layers of a Neural Network

The input layer and the output layer are defined by the values to be processed and the result to be produced. It is pretty clear that there will be one output neuron as we need to generate one move in the end. That move will be the output of that single output neuron.

For the input neurons things are not that straightforward. It is clear that the game state must be passed to the input neurons. On first sight the different possible board representations after making each valid move have been considered as the input. But it is hard to do something meaningful with this in the hidden layer. Furthermore the input neurons would have a different semantic every time. This makes the learning difficult. The next idea has been to map one field from the board to one input neuron. That worked to some extend. The final solution has three input neurons for each field on the board. Those are representing the possible game states: empty, occupied by computer player and occupied by opponent. With this approach it is important that the same field – with its corresponding state – is assigned to the same input neuron every time. This is depicted in the following diagram.

Input Layer with 27 Input Neurons

In addition some input value is required. This is defined based on the different fields and whether or not that field is empty, occupied by the computer player or occupied by the opponent.

Neurons in the hidden layer are calculating a “positional score” for the candidate moves. This is done based on the field values and the input weights. Hereby each neuron in the hidden layer always exactly represents a move to a certain field on the board.

In the beginning every neuron in the hidden layer was calculating a candidate move out of all possible moves. But this approach felt too much like an algorithmic solution through the backdoor.

That’s why there are nine neurons in the hidden layer, as there is at any time a maximum of nine possible moves.

Nine neurons in the hidden layer

Thus the first neuron in the hidden layer stands for a move on the first field, neuron two for a move on the second field and so on. This implies that some neurons cannot “fire” a valid move as the corresponding field is already occupied. This is the equivalent to a threshold that decides whether or not a neuron is activated (fires) or not. If no neuron in the hidden layer can be activated the game is anyway finished as there are no more valid moves anymore.

Activation functions

Activation functions are a vital part of any Neural Network implementation. They are using input values and input weights to calculate output values. Those are the input to the neurons of the next layer or the result computed by the Neural Network. Common to all layers is the randomized generation of output weights when the Neural Network is (re-)initialized. All neurons of one layer are sharing the same implementation of the activation function.

input layer

The activation function of this layer is rather simple. It stores the field information that it has retrieved as an input. Based on this it calculates a value depending on the field state and location on the board. Basically this includes a kind of threshold function. Only one of the three neurons reflecting one field is used as an input in the hidden layer.

hidden layer

For each neuron in the hidden layer a so-called position value is calculated based on the input weights and values. This is done applying the formula below where the sum over all input neurons is created. By doing so the complete board state is considered.

For every set of input neurons that are reflecting the same field the input weight and value of the corresponding neuron is used. This depends on whether the field is empty, owned by the computer or owned by the opponent. Thus from the 27 input neurons always only the nine relevant neurons from the input layer are used for this calculation.


Then the sigmoid function is applied to Z. The sigmoid function is quite commonly used in activation functions of Neural Networks.

S(Z) = 1 / 1 + e-Z

The resulting value is the positional score for this neuron and thus this candidate move.

output layer

In the output layer again a value Z is calculated. But this time not as a sum, but for each of the candidate moves.


Then again the sigmoid function is applied to Z. The candidate move where S(Z) is the maximum is chosen as the move to execute.

Summary and Outlook

This has been one of the most fun projects for quite some time. Not having any idea where to start was a really interesting experience. Playing around with different parameters like number of neurons and weight changes applied and then seeing how this affects the outcome of playing the game was really fascinating.

Luckily there is still plenty of room for improvements. First of all a more thorough training algorithm can be applied like rewarding all moves that lead to a win and not only the first and the last one. Another idea is to decrease the output weight of neurons if a move has let to a loss.

Then the structure of the Neural Network can be evolved by introducing additional hidden layers and thus increasing the number of neurons and connections between those.

Pretty sure there will be a follow-up to this blog post as one of the main objectives is not yet achieved: A Neural Network that learns to play Tic-Tac-Toe flawlessly :).

The post Gamma-TicTacToe – Neural Network and Machine Learning in a simple game appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

Interview with IOTA Co-Founder Dominik Schiener [blockcentric #4]

codecentric Blog - Thu, 11-Jan-18 01:29

Dominik Schiener is one of the co-founders of IOTA – a German non-profit organization that implements a completely new Distributed Ledger Technology competing with blockchain. I talked to Dominik about his early days in the world of Distributed Ledger Technologies, Tangle technology, the IOTA ecosystem, and the future.

Jannik Hüls: Thank you for your time. You are traveling a lot. Where am I catching you right now?

Dominik Schiener: I’m in the headquarters of Deutsche Telekom in Bonn. We are set up internationally, which involves a lot of traveling. We have offices in Berlin, Oslo, Chicago, and are in the process of opening one in Singapore.

Let’s talk about your history: The world of Distributed Ledger Technologies (DLT) is not that old yet. When was your first encounter with that topic – how did you hear about blockchain?

My first point of contact was in 2011, when I came across the Bitcoin whitepaper and tried to understand it. Back then, I was 14 or 15 years old, my English was not that good yet, and I didn’t fully grasp the concept. But I was quickly fascinated – I’ve always been the curious type – so that I have worked on it full time since 2012. I started by mining. Everyone kept telling me this was how you made big money, and of course that’s why I was really ambitious from the outset. I wanted to become an entrepreneur and started my own projects at the age of 14. With AWS credits I mined and thus earned quite a lot of money. I then used this money to implement my first own projects.

So basically mining was your main field of application. Or did you actually implement something back then, did you see other options besides cryptocurrencies?

The main focus back then was on cryptocurrency. It was not until 2013 that I really understood the value added, applications that are interesting not just from a financial point of view, but where blockchain itself creates added value.

Before we talk about IOTA: currently, the topic of DLT covers a wide range of things. To you, what defines a use case that lends itself to applying DLT?

If we look at time-to-implement or time-to-market, a blockchain solution for supply chain management is the one most likely to be integrated within the next two to five years. In supply chains, it creates the greatest added value. Thanks to the underlying transparency, we can identify inefficient processes and improve things like insurance.

“A blockchain solution for supply chain management is the one most likely to be integrated within the next two to five years.”

Tangle is the foundation of IOTA. How did the idea for the Tangle come about?

Since we had a startup developing new hardware for IT, or Fog Computing, to be precise, the basic idea was to have the machines pay each other – machine payments to buy and sell resources. Since we are also blockchain experts, our technical know-how was good enough to realize that none of the existing blockchain architectures were able to cater to the demands of the IoT space. Many had fundamental technical flaws and were too slow or too costly. So we did some research on Directed Acyclic Graphs to solve the existing problems. We did the mathematical proofs, and that’s how we developed IOTA.

Can you briefly outline the differences between IOTA and blockchain?

There are two major differences: The architecture is no longer a chain, but a Directed Acyclic Graph. And the way consensus is reached is different: [with blockchain,] miners use the Competitive Proof of Work or another consensus algorithm that validates transactions in cycles. In IOTA there are no cycles like in blockchain. When transactions are executed in IOTA, two older transactions must be confirmed. There are no more miners, but whenever someone in the network performs a transaction, they also contribute to the consensus of the network, which is one of the main benefits.

Another advantage is scalability. IOTA scales horizontally: the more network participants, the faster transactions are confirmed. Plus, there are no transaction fees because there are no miners left to substitute. This means that no expenses have to be paid because everyone participates in the validation process. Other benefits include partition tolerance and the fact that quantum computers can no longer attack our hashes.

You mentioned the IoT as possible use case – is there a difference between Full Nodes and Small Nodes? Does every IoT device need to store the entire tangle? And how big is it when it’s stored?

Right now there are only Full Nodes, but we are also developing Light Nodes and Small Nodes. Small Nodes are clusters of devices that are combined. In this cluster, they will use a Full Node that meets, for example, the more demanding requirements. However, these very questions depend very much on the architecture of future systems. So, what do they really look like in the future? How do the IoT devices interact?

That’s why the concept of Fog Computing is so relevant to us. The interesting thing about IOTA is that you can really outsource all processes that are involved in running a transaction to different devices. Every IoT device can have its own signature, which means: every IoT device can have a wallet. Even my coffee machine. The second step is Tip Selection. IOTA can find two transactions that need to be confirmed. For that I have to be a Full Node or a Light Node. This Tip Selection algorithm can then really be executed by the node, and thus the coffee machine can also make a transaction by interacting with the Full Node. We imagine this as a SmartHub within our own four walls. The final process is the proof of work, which is a bit more computation-intensive. That’s why we work on special hardware – especially for Fog Computing.

You said that the big advantage of IOTA is that there are no transaction costs. What’s the incentive for the miner then – why should I run a Full Node?

That’s one of the biggest misconceptions: There is no incentive to run a Full Node in Bitcoin and Ethereum. A Full Node is not necessarily a miner. There are about 5,000 to 6,000 Full Nodes in Bitcoin, but only a small fraction of these Full Nodes are miners. The advantage of IOTA is that the effort related to the validation process is much lower compared to Ethereum or Bitcoin. This means it’s better to run an IOTA Full Node than a Bitcoin or Ethereum Full Node. A node runs to be able to participate in the network.

“The advantage of IOTA is that the effort related the validation process is much lower compared to Ethereum or Bitcoin. This means it’s better to run an IOTA Full Node than a Bitcoin or Ethereum Full Node.”

In terms of business organization, IOTA deliberately sets itself apart from other DLT startups. You are a German non-profit organization. What’s the rationale behind this step?

It was of course a strategic move. We realized that the potential of the technology is simply too big to be limited by patents – that’s a conflict of interest. This is why this foundation idea makes so much sense, because the base layer, i.e. IOTA, should be free to use and open source. It should be used as widely as possible. In our opinion, the foundation is the best way to promote adoption. That’s why non-profit makes sense. Our goal is to bring together big companies, startups, and governments to build an ecosystem and invest. Since it is agnostic and independent, other companies are very interested in working with us rather than, for example, IBM.

Initially you also did an ICO [Initial Coin Offering], but you sold all IOTA tokens. How is the foundation funded?

We sold 100 percent of the tokens, then said to the community: if you want a foundation, you’ll need to donate money. As a result, the community got together and donated five percent of the tokens, which currently makes up about 200 million Euros. This is how the foundation is funded.

In other words, the value of the currency IOTA is also fundamental to the financial resources of the foundation.

Exactly. Now we bring companies on board, who then donate to the foundation. And we work with governments.

For example, the Data Marketplace is currently implementing a use case based on IOTA. There, data can be paid with micropayment.

Interesting. How do you explain the volatility of the market? How can I sell this use case better? When I buy something today, it costs, let’s say, 1000 IOTAs, which is perhaps 5 Euros, and tomorrow it might cost 50 Euros.

This is one of the biggest problems in IOTA and one of the biggest problems of cryptocurrencies in general. Volatility is in direct conflict with usability. One could think about an additional layer in which the use of the cryptocurrency is abstracted and that allows for payment in Euros, for example. However, our vision for IOTA is that the tokens are actually used. With other cryptocurrencies, the token is useless. We do not want a network in which each institution maintains its own token. This leads to a way too fragmented ecosystem. Nevertheless, the usability of the token is problematic and remains an unresolved issue.

Currently there are a lot of news about you after you announced Masked Authentication Messaging, Payment Channels, the Data Marketplace, and many other things. Can you roughly tell us what direction you are headed in? I assume you have more things in the pipeline.

We focus on the announcements, especially the partnerships we start with big companies. There, we are able to integrate IOTA into large-scale existing ecosystems. This is how we really get a scaling effect – where we can deploy thousands of nodes. But I can’t say more about this right now.

At codecentric, we are developers. Are there any SDKs [Software Development Kits] for IOTA?

As a matter of fact, we are working on this, especially for the modules that we develop. In terms of IOTA development, we are currently at the point where we have the IRI client to join the network and execute the transactions. Over the past few months, we’ve been working on a completely new system architecture which implements microservices and which is enterprise-oriented. You see, right now we just have one monolithic block, just as Ethereum or Bitcoin.

In addition, the IRI client is becoming much more modular. So as a company you can decide what communication protocol you want to use, and what database, be it SAP Hana or Redis. This really is the future of IOTA and it will be one of the best releases ever. Hopefully it’ll come in February, right now it’s still being developed and thoroughly tested.

There is a sandbox and a test net for developers. Are these the best ways to validate a proof of concept, or what’s the easiest way to go?

We are presently improving the entire sandbox environment. Our goal is that developers only need to send out an API call, and we then take care of the deployments. We are currently cooperating with some companies because they are so interested in IOTA that they also help the network by managing the deployments.

Looking at the Data Marketplace as use case for application developers: what is really stored in the Tangle? Or is it just about paying for the sensor data? Can you roughly outline the architecture?

Of course, the sensor data are represented in the Tangle. In IOTA, a transaction can contain about 1.2 kilobytes of data. In other words, if I have a sensor just for temperature logging or a small dataset, I can use IOTA for data transfer. This is how IOTA also ensures the integrity of the data. If someone wants to buy the data of a sensor, this will be billed directly by micropayment. The data is then not read by the sender, but from the tangle.

So basically the sensor pushes data into the Tangle, and I as a consumer can use the data from the Tangle to read sensor data. Pretty cool.

Finally, to reiterate: you talked about the IRI earlier. I’ve already seen that’s open source. What else is?

Everything. The Data Marketplace will also be open source. We also want to make other use cases, such as SatoshiPay, available to the community. We are not done with that, though, we are still working on it.

Thank you for your time. Cool stuff you’re working on, and fun to follow. Have a great time in Bonn!


Thank you, Jannik!


The interview was conducted in German and then translated into English.

Our article series “blockcentric” discusses Blockchain-related technology, projects, organization and business concerns. It contains knowledge and findings from our 20% time work, but also news from the area.

Blockcentric Logo
We are looking forward to your feedback on the column and exciting discussions about your use cases.

Previously published blockcentric-Posts

The post Interview with IOTA Co-Founder Dominik Schiener [blockcentric #4] appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

Looking beyond accuracy to improve trust in machine learning

codecentric Blog - Tue, 09-Jan-18 04:00

Traditional machine learning workflows focus heavily on model training and optimization; the best model is usually chosen via performance measures like accuracy or error and we tend to assume that a model is good enough for deployment if it passes certain thresholds of these performance criteria. Why a model makes the predictions it makes, however, is generally neglected. But being able to understand and interpret such models can be immensely important for improving model quality, increasing trust and transparency and for reducing bias. Because complex machine learning models are essentially black boxes and too complicated to understand, we need to use approximations to get a better sense of how they work. One such approach is LIME, which stands for Local Interpretable Model-agnostic Explanations and is a tool that helps understand and explain the decisions made by complex machine learning models.

Accuracy and Error in Machine Learning

A general Data Science workflow in machine learning consists of the following steps: gather data, clean and prepare data, train models and choose the best model based on validation and test errors or other performance criteria. Usually we – particularly we Data Scientists or Statisticians who live for numbers, like small errors and high accuracy – tend to stop at this point. Let’s say we found a model that predicted 99% of our test cases correctly. In and of itself, that is a very good performance and we tend to happily present this model to colleagues, team leaders, decision makers or whoever else might be interested in our great model. And finally, we deploy the model into production. We assume that our model is trustworthy, because we have seen it perform well, but we don’t know why it performed well.

In machine learning we generally see a trade-off between accuracy and model complexity: the more complex a model is, the more difficult it will be to explain. A simple linear model is easy to explain because it only considers linear relationships between variables and predictor. But since it only considers linearity, it won’t be able to model more complex relationships and the prediction accuracy on test data will likely be lower. Deep Neural Nets are on the other end of the spectrum: since they are able to deduce multiple levels of abstraction, they are able to model extremely complex relationships and thus achieve very high accuracy. But their complexity also essentially makes them black boxes. We are not able to grasp the intricate relationships between all features that lead to the predictions made by the model so we have to use performance criteria, like accuracy and error, as a proxy for how trustworthy we believe the model is.

Trying to understand the decisions made by our seemingly perfect model usually isn’t part of the machine learning workflow.
So why would we want to invest the additional time and effort to understand the model if it’s not technically necessary?

One way to improve understanding and explain complex machine learning models is to use so-called explainer functions. There are several reasons why, in my opinion, model understanding and explanation should become part of the machine learning workflow with every classification problem:

  • model improvement
  • trust and transparency
  • identifying and preventing bias

Model Improvement

Understanding the relationship between features, classes and predictions, thereby understanding why a machine learning model made the decisions it made and which features were most important in that decision can help us decide if it makes intuitive sense.

Let’s consider the following poignant example from the literature: we have a deep neural net that learned to distinguish images of wolves from huskies [1]; it was trained on a number of images and tested on an independent set of images. 90 % of the test images were predicted correctly. We could be happy with that! But what we don’t know without running an explainer function is that the model based its decisions primarily on the background: wolf images usually had a snowy background, while husky images rarely did. So we unwittingly trained a snow detector… Just by looking at performance measures like accuracy, we would not have been able to catch that!

Having this additional knowledge about how and based on which features model predictions were made, we can intuitively judge whether our model is picking up on meaningful patterns and if it will be able to generalize on new instances.

Trust and Transparency

Understanding our machine learning models is also necessary to improve trust and provide transparency regarding their predictions and decisions. This is especially relevant given the new General Data Protection Regulation (GDPR) that will go into effect in May of 2018. Even though it is still hotly discussed whether its Article 22 includes a “right to explanation” of algorithmically derived decisions [2], it probably won’t be enough for long any more to have black box models making decisions that directly affect people’s lives and livelihoods, like loans [3] or prison sentences [4].

Another area where trust is particularly critical is medicine; here, decision will potentially have life-or-death consequences for patients. Machine learning models have been impressively accurate at distinguishing malignant from benign tumors of different types. But as basis for (no) medical intervention we still require a professional’s explanation of the diagnosis. Providing the explanation for why a machine learning model classified a certain patient’s tumor as benign or malignant would go a long way to help doctors trust and use machine learning models that support them in their work.

Even in everyday business, where we are not dealing with quite so dire consequences, a machine learning model can have very serious repercussions if it doesn’t perform as expected. A better understanding of machine learning models can save a lot of time and prevent lost revenue in the long run: if a model doesn’t make sensible decisions, we can catch that before it goes into deployment and wreaks havoc there.

Identifying and Preventing Bias

Fairness and bias in machine learning models is a widely discussed topic [5, 6]. Biased models often result from biased ground truths: if the data we use to train ours model contains even subtle biases, our models will learn them and thus propagate a self-fulfilling prophecy! One such (in)famous example is the machine learning model that is used to suggest sentence lengths for prisoners, which obviously reflects the inherent bias for racial inequality in the justice system [4]. Other examples are models used for recruiting, which often show the biases our society still harbors in terms of gender-associations with specific jobs, like male software engineers and female nurses [5].

Machine learning models are a powerful tool in different areas of our life and they will become ever more prevalent. Therefore, it is our responsibility as Data Scientists and decision makers to understand how the models we develop and deploy make their decisions so that we can proactively work on preventing bias from being reinforced and removing it!


LIME stands for Local Interpretable Model-agnostic Explanations and is a tool that helps understand and explain the decisions made by complex machine learning models. It has been developed by Marco Ribeiro, Sameer Singh and Carlos Guestrin in 2016 [1] and can be used to explain any classification model, whether it is a Random Forest, Gradient Boosting Tree, Neural Net, etc. And it works on different types of input data, like tabular data (data frames), images or text.

At its core, LIME follows three concepts:

  • explanations are not given globally for the entire machine learning model, but locally and for every instance separately
  • explanations are given on original input features, even though the machine learning model might work on abstractions
  • explanations are given for the most important features by locally fitting a simple model to the prediction

This allows us to get an approximate understanding of which features contributed most strongly to a single instance’s classification and which features contradicted it and how they influenced the prediction.

The following example showcases how LIME can be used:
I built a Random Forest model on a data set about Chronic Kidney Disease [7]. The model was trained to predict whether a patient had chronic kidney disease (ckd) or not (notckd). The model achieved 99 % accuracy on validation data and 95 % on test data. Technically, we could stop here and declare victory. But we want to understand why certain patients were diagnosed with chronic kidney disease and why others weren’t. A medical professional would then be able to assess whether what the model learned makes intuitive sense and can be trusted. To achieve this, we can apply LIME.

As described above, LIME works on each instance individually and separately. So first, we take one instance (in this case the data from one patient) and permute it; i.e. the data is replicated with slight modifications. This generates a new data set consisting of similar instances, based on one original instance. For every instance in this permuted data set we also calculate how similar it is to the original instance, i.e. how strong the modifications made during permutation are. Basically, any type of statistical distance and similarity metric can be used in this step, e.g. Euclidean distance converted to similarity with an exponential kernel of specified width.
Next, our complex machine learning model, which was trained before, will make predictions on every permuted instance. Because of the small differences in the permuted data set, we can keep track of how these changes affect the predictions that are made.

And finally, we fit a simple model (usually a linear model) to the permuted data and its predictions using the most important features. There are different ways to determine the most important features: we typically define the number of features we want to include in our explanations (usually around 5 to 10) and then either

  • choose the features with highest weights in the regression fit on the predictions made by the complex machine learning model
  • apply forward selection, where features are added to improve the regression fit on the predictions made by the complex machine learning model
  • choose the features with smallest shrinkage on the regularization of a lasso fit on the predictions made by the complex machine learning model
  • or fit a decision tree with fewer or equal number of branch splits as the number of features we have chosen

The similarity between each permuted instance and the original instance feeds as a weight into the simple model so that higher importance is given to instances which are more similar to the original instance. This precludes us from using any simple model as an explainer that is able to take weighted input, e.g. a ridge regression.

Now, we can interpret the prediction made for the original instance. With the example model described above, you can see the LIME output for the eight most important features for six patients/instances in the figure below:

LIME explanations of example machine learning model

Each of the six facets shows the explanation for the prediction of an individual patient or instance. The header of each facet gives the case number (here the patient ID), which class label was predicted and with what probability. For example, the top left instance describes case number 4 which was classified as “ckd” with 98 % probability. Below the header we find a bar-plot for the top 8 most important features; the length of each bar shows the weight of the feature, positive weights support a prediction, negative weights contradict it. Again described for the top left instance: the bar-plot shows that the hemoglobin value was between 0.388 and 0.466, which supports the classification as “ckd”; packed cell volume (pcv), serum creatinine (sc), etc. similarly support the classification as “ckd” (for a full list of feature abbreviations, see This patient’s age and white blood cell count (wbcc), on the other hand, are more characteristic of a healthy person and therefore contradict the classification as “ckd”.

Links and additional resources

This article is also available in German:

  1. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16). ACM, New York, NY, USA, 1135-1144. DOI:
  2. Edwards, Lilian and Veale, Michael, Slave to the Algorithm? Why a ‘Right to an Explanation’ Is Probably Not the Remedy You Are Looking For (May 23, 2017). 16 Duke Law & Technology Review 18 (2017). Available at SSRN:
  4. and

The post Looking beyond accuracy to improve trust in machine learning appeared first on codecentric AG Blog.

Categories: Agile, Java, TDD & BDD

Thread Slivers eBook at Amazon

Syndicate content