Feed aggregator

A Different Kind of Java Security Framework

Javalobby Syndicated Feed - Mon, 18-Jan-16 02:46

A commonly asked question about the open-source OACC security framework for Java is how it compares to <insert your favorite framework here>? I always wish I could provide a satisfactory answer as short as that question. The TL;DR seems to be:

OACC solves the same problem, but with a comprehensive security model that is different.

Categories: Java

Deciding Between Ease and Extensibility

Javalobby Syndicated Feed - Mon, 18-Jan-16 00:01

In the day-to-day tasks of development, a very common question arises: Should I spend the time to make this flexible and extensible ("do it the right way") or should I do it the quickest way possible ("get it to work")? In the former case, you will end up with a very extensible solution that makes future development easier, or you may end up spending time now to create a solution that will either not be used in the future or will become encumbering (the complexity of the solution means that you have more software complexity to maintain in the future). In the latter case, you may get by with the simplest solution and get the job done on time, or if you are smooth, ahead of schedule, but maybe you start to accrue technical debt or create a concoction that makes every other developer cringe when they get near it. 

With these two options and any degree between these two poles, we need to make a decision between ease of development (getting the job done) or extensibility (doing the right way, so that future development is easier). We should also not lure ourselves into thinking these choices are a dichotomy or are mutually exclusive: Even if we decide we need to be more extensible than swift, we are making the decision to favor one factor over another. As with most questions in software engineering, there is no single answer and no silver bullet, but that does not mean there are not points we can examine to make a better decision. These are by no means the only points to consider, but these are some points that will steer us in the correct direction (think selecting between North, South, East, or West rather than between a heading of 0 degrees or 5 degrees).

Categories: Java

Java 9 Jigsaw – A Missing Piece

codecentric Blog - Mon, 18-Jan-16 00:00

Some excellent blog posts have already been written about Java 9 – Jigsaw, not the least by my colleague Florian Troßbach in two parts here; part 1 and here; part 2.
During this post I would like to dive deeper into some lesser known details and quirks, specifically a RuntimeException thrown after incorrectly building a
custom modular run-time image.

Service Resolution

Since Java SE 6 already, the platform provides the ServiceLoader API to extend existing applications. The ServiceLoader makes it possible to detect existing implementations of a certain interface / abstract class and load them to be used. This solution still works nicely with Java modules. Where previously the JAR-files had to be present on the class-path, it is now sufficient to have the modules present on the module-path. See the following example.

The NetworkSocketProvider is our abstract class that needs to be implemented by the various instances we would like to lookup:

public abstract class NetworkSocketProvider {
    protected NetworkSocketProvider() {}
 
    public abstract NetworkSocket openNetworkSocket();
}

The FastNetworkSocketProvider provides one implementation, it will create a FastNetworkSocket for us. Naturally, we could have multiple implementations.

public class FastNetworkSocketProvider extends NetworkSocketProvider {
    @Override
    public NetworkSocket openNetworkSocket() {
        return new FastNetworkSocket();
    }
}

And now for the code that will do the lookup:

public static NetworkSocket open() {
    ServiceLoader sl = ServiceLoader.load(NetworkSocketProvider.class);
 
    Iterator iter = sl.iterator();
 
    if (!iter.hasNext()) {
        throw new RuntimeException("No service providers found");
    }
 
    NetworkSocketProvider provider = iter.next();
    for (NetworkSocketProvider networkSocketProvider : sl) {
        System.out.println(networkSocketProvider.openNetworkSocket().getClass());
    }
    return provider.openNetworkSocket();
}

This static method will, using the ServiceLoader, give us a list of possible implementations and in this case simply return the first one found.

Module Resolution

While service resolution provides not much new, something that is new, is the Layer API and related classes. The Layer API is available in the java.lang.reflect package and represents a ‘layer’ of modules. The following code makes it possible to get a list of loaded modules, specifically those with names starting with “nl.codecentric”:

private static void printModules() {
    Layer layer = Layer.boot();
    layer.modules().stream().filter(m -&gt; m.getName().startsWith("nl.codecentric")).forEach(m -&gt; {
        String name = m.getName();
        Optional version = m.getDescriptor().version();
        System.out.println("Loaded module: " + name + " - version: " + version);
    });
}

Run-time Image – A Missing Piece

Java 9 allows for custom modular run-time images to be created as described by Florian here. What’s amazing about the JLink utility is that it will resolve all required modules automatically. But there’s also a catch.

Given the above example with our sockets, we have three modules:

src/
├── nl.codecentric.test
├── nl.codecentric.socket
├── nl.codecentric.fastsocket

The test module contains the Main method and will request a socket. The socket module contains the abstract NetworkSocketProvider and the class to resolve the socket implementations. Finally the fastsocket module contains the FastNetworkSocketProvider and FastNetworkSocket implementations.

When compiling these modules and running with the following command (with all modules on the module-path):

java -mp mlib -m nl.codecentric.test

We get the following output:

Loaded module: nl.codecentric.socket - version: Optional[1.0]
Loaded module: nl.codecentric.fastsocket - version: Optional[2.0]
Loaded module: nl.codecentric.test - version: Optional.empty
class nl.codecentric.fastsocket.FastNetworkSocket
 
class nl.codecentric.fastsocket.FastNetworkSocket version: 2.0

However, if we now create a custom modular run-time image from this using the command:

jlink --modulepath $JAVA_HOME/jmods:mlib --addmods nl.codecentric.test --output linkout

And running this will generate the following output:

Loaded module: nl.codecentric.test - version: Optional.empty
Loaded module: nl.codecentric.socket - version: Optional[1.0]
Exception in thread "main" java.lang.RuntimeException: No service providers found
	at nl.codecentric.socket.NetworkSocket.open(nl.codecentric.socket@1.0/NetworkSocket.java:20)
	at nl.codecentric.test.Main.main(nl.codecentric.test@/Main.java:15)

As you see, we’ll get a RuntimeException because it cannot find the FastNetworkSocketProvider. Listing the packaged modules:

./bin/java -listmods

Will only show:

java.base@9.0
nl.codecentric.socket@1.0
nl.codecentric.test

What happened?? While the JLink util will resolve all direct references and dependencies, it will not link providers. No module requires the fastsocket module, and thus it is not included. While this is somewhat logical, the risk is that this issue only shows up during run-time. And might be hard to debug even.

The solution is to include all required modules explicitly:

jlink --modulepath $JAVA_HOME/jmods:mlib --addmods nl.codecentric.test,nl.codecentric.fastsocket --output linkout

There might be an option in the future for JLink to bind explicitly; https://twitter.com/mreinhold/status/665122968851382273, which might at least make it more explicit.

Hope you liked this blog and would love to hear your comments!

The post Java 9 Jigsaw – A Missing Piece appeared first on codecentric Blog.

Categories: Agile, Java, TDD & BDD

How to Create a Heap Dump for Your Java Application

Javalobby Syndicated Feed - Sun, 17-Jan-16 01:14

The heap (in a JVM) is the place where a JVM keeps all its runtime objects. The JVM creates a dedicated space for the heap at the JVM startup, which can be controlled via the JVM option -Xms<size> eg: -Xms100m (this will allocate 100MBs for the heap). The JVM is capable of increasing and decreasing the size of the heap [1] based on the demand, and the JVM has another option which allows to set a max size for the heap: -Xmx<size>, eg: -Xmx6g (this allows the heap to grow up to 6GBs).

The JVM automatically performs Garbage Collection (GC) when it detects that the JVM is about to reach the max heap size. But the GC can only clean the objects which are eligible for GC. If the JVM can't allocate required memory even after GC, JVM will crash with "Exception in thread "main" java.lang.OutOfMemoryError: Java heap space".

Categories: Java

Elm Friday: Type System Basics – Type Aliases and Records (Part IX)

codecentric Blog - Sat, 16-Jan-16 06:57

We have already touched the topic of Elm’s type system briefly (for example in post VI about type annotations) but Elm provides a few type constructs that we have not examined yet. We also talked about the advantages of having a strong type system, namely the stronger guarantees it enables as compared to dynamic languages like JavaScript. This boils down to “If it compiles, it’ll never throw a runtime exception”. In this episodes, we’ll revisit tuples and introduce type aliases and records.

About This Series

This is the ninth post in a series of short and sweet blog posts about Elm. The stated goal of this series is to take you from “completely clueless about Elm” to “chief Elm guru”, step by step. If you have missed the previous episodes, you might want to check out the table of contents.

Tuples

We have already used tuples in previous examples but since they are one of the basic building blocks for types, let’s review the concept shortly.

Tuples are similar to lists as they represent collections of multiple items. However, in a list, all elements need to have the same type. The number of elements is variable. In contrast, the elements of a tuple can all have different types but all tuples of the same type have the same length.

Here are some examples for a tuples (representing human beings with a name, weight in kilograms and their height in meters):

alice : (String, Int, Float)
alice = ("Alice", 61, 1.68)

bob : (String, Int, Float)
bob = ("Bob", 78, 1.82)

So, tuples are pairs (or triplets, quadruplets, …) of values. They are enclosed in ( and ) in Elm.

You can use pattern matching to access individual parts of the tuple. The following function takes a 3-tuple like in the example above and returns the name by doing a pattern matching on the incoming tuple. Since we do not care about the weight and the height here, we use the underscore (_) for those values.

getName : (String, Int, Float) -> String
getName (name, _, _) = name

Type Aliases

You can assign type aliases to make your code more readable. If the first element of a tuple is meant to represent a name, why not call it just that?

type alias Name = String
type alias Weight = Int
type alias Height = Float

alice : (Name, Weight, Height)
alice = ("Alice", 61, 1.68)

bob : (Name, Weight, Height)
bob = ("Bob", 78, 1.82)

Here, we used type aliases for basic types that are provided by Elm out of the box. You can use type aliases for any type construct you like, as we’ll see later.

Records

Tuples are one way to represent data structures. They are best suited for structures with only a few attributes, like pairs of values. For more structured data, Elm offers an alternative called records. This is how records looks like:

type alias Person =
  { name : String
  , weight : Int
  , height : Float
  }

alice : Person
alice =
  { name = "Alice"
  , weight = 61
  , height = 1.68
  }

bob : Person
bob =
  { name = "Bob"
  , weight = 78
  , height = 1.82 }

Note how we used a type alias here to have an identifier for the record type. This identifier (Person) can be used in type annotations then. The two values alice and bob show how to create new records.

There are a few more things that you can do with records and we will get to that in the next sections.

Access Record Attributes

Record attributes are accessed using a dot notation. To access the name of a Person record, you would write person.name, or person.height to access the person’s height attribute.

The dot notation can even be used as a standalone function. This is valid Elm code which converts a list of Person records into a list of their names:

toNames : List Person -> List String
toNames list =
  List.map .name list

Update Record Fields

Elm also provides a mechanism to update records. Since all values in Elm are immutable, “updating” a record translates to creating a copy of the original record and changing one (or several) attributes in the process.

The following example demonstrates how to update one attribute.

rename : Person -> String -> Person
rename person newName =
  { person | name = newName }

The next example updates multiple attributes at once.

grow : Person -> Int -> Float -> Person
grow person weightIncrease heightIncrease =
  { person |
      weight = person.weight + weightIncrease,
      height = person.height + heightIncrease }

That’s it for today. Now that we have reviewed tuples, type aliases and records, there is only one major type construct missing, which is called union types. This will be covered in the next episode. See you next time!

The post Elm Friday: Type System Basics – Type Aliases and Records (Part IX) appeared first on codecentric Blog.

Categories: Agile, Java, TDD & BDD

Robot Framework Tutorial 2016 – Keywords

codecentric Blog - Fri, 15-Jan-16 03:25

Robot Framework Tutorial 2016

Part 1: Installation
Part 2: Keywords

The “old” Robot Framework Tutorial.

The Robot Framework really comes with a very comprehensive User Guide. Sometimes this makes me think that additional tutorials are almost not needed. But then again I think that a blog post can be more specific in digging into one aspect of a tool than the documentation can. So here we go digging into the central testing concept of the Robot Framework and that is: Keywords.

This blog post will entirely deal with available Keywords and how to write new Keywords out of existing ones directly using Robot Framework features. For a lot of use cases this approach is already sufficient due to the huge amount of existing Test Libraries and thus Keywords. The next blog post in this series will then deal with writing new Keywords in Java (just in case you are waiting for that one). One could also use other programming languages, but for me as a Java developer this is a quite natural choice. Writing new Keywords this way is needed if you have to test specific technologies that are not yet covered by the existing Test Libraries.

A first Example

Let’s jump into the cold water right away with the smallest possible example I can think of for a Test written with the Robot Framework. (Yes, actually it is not even testing anything, but that is nitpicking.)

*** Settings ***

*** Test Cases ***
Test Robot Framework Logging
  Log   "Test Logging"

The Settings section is empty because we are not importing any external Test Libraries. I could have omitted that section of course, but somehow for me it belongs to a Test Case File even if it is empty.

We defined one testcase below the Test Cases section. That one is executing one Keyword “Log” and we are passing one parameter to this Keyword, namely the String “Test Logging”. Easy as that, but still there are a few stumbling blocks. There must be at least two spaces to separate arguments from Keywords and arguments from each other. Furthermore there must be always an indention of two spaces in the beginning of each line below a testcase. Both should be visible from the example, but it is still easy to miss these things.

The test can be executed with pybot or Java – depending on the type of your installation – directly from the directory where the Test Case File is stored.

pybot --outputdir ./report sample-0-trivial.txt
java -jar /usr/local/opt/robotframework/robotframework-2.9.2.jar --outputdir ./report sample-0-trivial.txt

There are lots of command line options possible. Here we are using “–outputdir” to define the place where the resulting Log- and Report-Files are stored. Just take a look at the Report-File by opening it in a Browser. We will still have a closer look on the Report- and Log-File at the end of this blog post.

All examples shown in this blog post can be found from GitHub here. This way they are hopefully easier to access than through this blog post and there are more examples than shown in this blog post. Furthermore I am planning to enhance that site with more examples in the future.

The Robot Framework is supporting different formats for writing Test Case Files. The format used throughout this blog post (and blog series) is the Plain Text Format. I simply like it the most as it is very easy to read and understand once you have been getting used to the thing with the “additional whitespaces” mentioned above. You could also use HTML, which allows you to have a nice kind of “presentation view” on your test cases. But it is simply more code to write and it is more likely to run into merge conflicts when putting the files under version control (which should always be the case). There is also the possibility to use BDD-style test case descriptions. Well, that could probably be a blog post of its own, but for the time being I would like to omit that. It only makes things more complicated while we want to learn the basic concepts here.

Keywords – A Programming Language

Working with Keywords is like learning a new programming language. So you basically need to understand the syntax of it, which is luckily not too complicated. And hopefully this will no longer be a problem after reading through this blog post and the examples. And of course you need to know what a new language offers. For the Robot Framework this means how to implement certain structures like doing loops or conditional execution of keywords. And it means knowing which Test Libraries and Keywords are available.

Thus it is definitely worth taking a look at the list of available Test Libraries on the Robot Framework homepage to get an idea what is already available. Please note that the Standard Libraries are included in each Robot Framework installation right away (compare previous blog post on installation), while the External Libraries require additional installation.

Build-In, Standard and External Test Libraries. Well, one could get confused by this, but no worries. The Build-In Library is part of every Robot Framework installation and Keywords from that Library can be used without even having to import that Library in the Settings section. Standard Libraries are part of every Robot Framework installation, but to use Keywords from them they must be “imported” in the Settings section. External Libraries must be installed separately. They come with own installation instructions and they must be imported the same way as the Standard Libraries.

Ok, the next example:

*** Settings ***
Library     String


*** Test Cases ***
Test Robot Framework Logging
  Log   Test Logging
  Log Many  First Entry   Second Entry
  Log To Console  Display to console while Robot is running

Test For Loop
    : FOR    ${INDEX}    IN RANGE    1    3
    \    Log    ${INDEX}
    \    ${RANDOM_STRING}=    Generate Random String    ${INDEX}
    \    Log    ${RANDOM_STRING}

I like this example as it is still pretty short, but it shows a few of the features that are available when working with Keywords at one glance. First of all we extended the first test case by adding more rows to it calling different Keywords. That is basically how we implement Tests – even though here we are only logging – by executing any amount of Keywords (with corresponding parameters) below one test case definition. The example also shows how to define two different test cases in one Test Case File. The second test case shows how to implement a loop and how to assign results from executing a Keyword to a variable. As the Keyword “Generate Random String” comes from the String Library we need to import that one under the Settings section. As it is a Standard Library it comes with the Robot Framework installation.

Writing own Keywords

So far we have seen how to use existing Keywords. Now it is time to write the first new Keyword.

*** Settings ***

*** Test Cases ***
Test Robot Framework Logging
    Log    Test Logging

Test My Robot Framework Logging
    My Logging    My Message    WARN

*** Keywords ***
My Logging
    [Arguments]    ${msg}    ${level}
    Log    ${msg}    ${level}

Own Keywords in Test Case Files are written below the Keywords section. The syntax is pretty much the same as when writing test cases. Major difference ist that you can pass parameters in to your own Keywords.

A common example for an own Keyword is for example a Login that would get a username and password (maybe also a URL) and then hides all the technical details (technical Keywords) needed to perform the Login.

Organizing Keywords – Resource Files

Of course you do not want to pollute your Test Case Files with your Keyword definitions. Obviously that makes those also less reusable. The solution to this is defining new Keywords in so-called Resource Files. Those can then be included similar to the way Test Libraries are included.

*** Keywords ***
My Logging
    [Arguments]    @{arg}
    Log Many    @{arg}

Example “Resource File”:

*** Settings ***
Resource        resource-2.txt

*** Test Cases ***
Test Robot Framework Logging
    Log    "Test Logging"

Test My Logging
    My Logging   "Test My Logging 1"   "Test My Logging 2"

Example “Test Case File”:

Well, not too much to add here as this is pretty straightforward. Beside using Resource Files the example also shows how to use arguments from type array for own Keywords. In this example the Test Case File and the Resource File must be located in the same directory. Of course it is also possible to store Resource Files to sub-directories and then use relative path-information when including the Resource File.

Note: Never use absolute paths to include Resource Files as this path might differ for different team members and the test environment. Always use only relative path definitions here.

Beyond Keywords

Basically you should have now all the basics to start writing own Keywords, Resource Files and Test Case Files. Still there are a lot more possibilities beyond those basics. Some of them are briefly depicted in the following.

Setup and Teardown

This feature allows to execute certain Keywords to prepare tests or clean up after testing. The following shows how to define setup/teardown for a complete Test Case File.

*** Settings ***
Suite Setup       Setup Actions
Suite Teardown    Teardown Actions

*** Test Cases ***
Test Robot Framework Logging
    Log    Test Logging

*** Keywords ***
Setup Actions
    Log    Setup Actions done here

Teardown Actions
    Log    Teardown Actions done here

There are more possibilities to also have setup/teardown defined to be running for each test case.

Tagging

Tagging is not really related to Keywords, but it is a nice way to get a better grip on what you are testing from looking at the report.

*** Settings ***

*** Test Cases ***
Test Robot Framework Logging
    [Tags]    sample   logging
    Log    "Test Logging"

The tags are summarized in the report and thus you can check how many test cases are related to certain features of the system under test. But please note that his requires some discipline in applying the tags.

This leads to the final part of this blog post that is dealing with the Report- and Log-Files.

Report- and Log-Files

The reporting- and logging of the Robot Framework is definitely very strong.

reporting

As can be seen on the above screenshot there are different statistics on the results. You can see how many test cases have a certain tag. Furthermore you see the statistics by Test Suite, which here means Test Case File. By clicking on “Sample-0-Tagging” you get a detailed list of the results from that Test Case File.

reporting-1

The great thing is that you can now dig deeper into the results by clicking on an individual test case. This opens a new page with the log. Here you see the result of each of the executed Keywords that are forming a test case. This way if a test would fail you can exactly see which Keyword has failed and also additional logging information could be added here (we had a lot of logging examples already :-)).

logging

Conclusion

With the concepts of Keywords the Robot Framework offers a rich ecosystem for writing test functionality using existing Test Libraries. This can and should be used to decouple technical concepts from functional ones. Using the domain vocabulary in the test cases makes them more readable and understandable while hiding the technical details in own Keywords and Resource Files.

This blog post cannot show everything that is possible and available. Best advice is (again) to take a look at the existing Test Libraries. There are also more examples in the already mentioned tutorial on GitHub.

Next week we will take a look at writing new Keywords in Java and there is still a lot more to explore around the Robot Framework :-).

The post Robot Framework Tutorial 2016 – Keywords appeared first on codecentric Blog.

Categories: Agile, Java, TDD & BDD

Playing With Spring Boot, Vaadin, and Kotlin

Javalobby Syndicated Feed - Fri, 15-Jan-16 00:42

It’s no mystery that I’m a fan of both Spring Boot and Vaadin. When the Spring Boot Vaadin add-on became GA, I was ecstatic. Lately, I became interested in Kotlin, a JVM-based language offered by JetBrains. Thus, I wanted to check how I could develop a small Spring Boot Vaadin demo app in Kotlin – and learn something in the process. Here are my discoveries, in no particular order.

Spring Needs Non-final Stuff

It seems Spring needs @Configuration classes and @Bean methods to be non-final. As my previous Spring projects were in Java, I never became aware of that because I never use the final keyword. However, Kotlin classes and methods are final by default: hence, you have to use the open keyword in Kotlin.

Categories: Java

Book Review: Thinking in Java

Javalobby Syndicated Feed - Fri, 15-Jan-16 00:37

I recently read the 1000-page tome Thinking in Java, written by Bruce Eckel, with the goal of getting my feet wet in the parts of the language that were still obscure to me. Here is my review of the book, containing its strong and weak points.

Basic Topics

This is a book that touches on every basic aspect of the Java language, keeping an introductory level throughout but delving into deep usage of the Java standard libraries when treating a specific subject.

Categories: Java

Meet SnoopEE, a Discovery Service for Java EE

Javalobby Syndicated Feed - Thu, 14-Jan-16 22:50

SnoopEE is a very interesting grass roots open source Java EE ecosystem project. Akin to NetFlixOSS Eureka it enables microservices discovery, lookup and registration. SnoopEE is developed by Java EE advocate, JCP expert, speaker and blogger Ivar Grimstad. Ivar presented on SnoopEE at JavaOne 2016. One very nice aspect of Ivar's talk is that it actually discusses the basic motivation for service discovery and SnoopEE instead of simply covering the features of the project. You can see his well done session below (click here if you can't see the embedded video).

The slide deck for his talk can be found here. The project itself is on GitHub. SnoopEE is still pretty nascent and under active development. You should consider giving it a try, using it and contributing back to it.

Categories: Java

CUBA Platform Filters — the Fast Food of Business Software?

Javalobby Syndicated Feed - Thu, 14-Jan-16 01:14

Last time I wrote about the generic filter and its opportunities that the CUBA Platform provides out of the box. I left off with two requirements which are important for developers, because they have to be 100% sure that the generic solution does not fall down with real world problems.

Going From Fast Food to a Real Healthy Meal

After going through the CUBA filter opportunities for the first time, it felt a little strange to me. The first thing I thought, was something like “OK, so WTF did I spent my time on the last few years?”. Because, at first glance, it seemed too good to be true. Don’t need to write this simple, boring and similar filter mechanisms by yourself anymore?

Categories: Java

The Machinery behind Machine Learning – A Benchmark for Linear Regression

codecentric Blog - Wed, 13-Jan-16 23:00

This is the third post in the “Machinery behind Machine Learning” series and after all the “academic” discussions it is time to show meaningful results for some of the most prominent Machine Learning algorithms – there have been quite a few requests from interested readers in this direction. Today we are going to present results for Linear Regression as prototype for a Regression method, follow-up posts will cover Logistic Regression as prototype for a Classification method and a Collaborative Filtering / Matrix Factorization algorithm as prototype for a Recommender System. It is worth noting that these prototypes for Regression, Classification and Recommendation are relying on the same underlying Optimization framework – despite their rather different interpretation or usage in the field of Machine Learning.

To be a bit more precise: In all three applications the training part is driven by an algorithm for Unconstrained Optimization, in our case Gradient Descent. As we want to emphasize the tuning options and measure the impact of the stepsize in particular we provide a comparison of the standard Armijo rule, the Armijo rule with widening and the exact stepsize. In later articles – after we learned something about Conjugate Gradient and maybe some other advanced methods – we are going to repeat this benchmark, but for now let’s focus on the performance of Gradient Descent.

Linear Regression

Linear Regression is a conceptually simple approach for modeling a relationship between a set of numerical features – represented by the independent variables x_1,...,x_n – and a given numerical variable y, the dependent variable. When we assume that we have m different data points or vectors x^{(j)} and values or real numbers y_j, the model takes the following form:

y_j \approx c_0 + c_1 x^{(j)}_1 + ... + c_n x^{(j)}_n

with c_i,\ i=0,..,n, being some real-valued parameters. Using the matrix-vector notation from Linear Algebra we derive a more compact formulation. We put the parameter values c_i in the parameter vector c of length n+1, collect the row vectors =[1,x^{(j)}_1,...,x^{(j)}_n] into a m\times (n+1)-matrix X – the leading 1 in each vector belongs to the coefficient c_0 – and end up with something close to a linear system of equations:

Xc\approx y.

The interpretation is that we want to find a parameter vector c that satisfies the linear system of equation as good as possible, thus we are looking for the best approximate solution because an exact solution does not exist in general if m>n+1, which is assumed to be the case here.

The objective function for Linear Regression

One mathematical translation of as good as possible is to minimize the residual or error measured by the (squared) Euclidean norm:

=\|Xc - y\|^2.

The Euclidean norm is the usual notion of distance so nothing spectacular here. We square the expression to get rid of the square root that hides within the norm, it’s simpler and better from a computational point of view. Of course it does influence the concrete value of the error but does not change the solution or optimal parameter vector c^* that we are looking for.
Now we have arrived at an unconstrained minimization problem, the process of minimizing the error theoretically involves all possible values for the parameters c_i, there are no restrictions. It’s time to show what we have learned so far. First, let’s unfold the compact expression to see what exactly is measured by the objective function f defined above:

f(c)=\|Xc - y\|^2 = \sum_j \left( c^Tx^{(j)}-y_j \right)^2.

In the implementation we have included an optional scaling factor of 1/(2m), that normalizes the value of the objective function with respect to the number of data points. It’s presence or absence does not change the concrete solution, it’s an implementational detail that we have omitted here. The important ingredients are the summands \left( c^Tx^{(j)}-y_j \right)^2 that quantify the pointwise deviation of the model from the input data.

Visualizing Linear Regression

Effectively we sum up the squared prediction errors for every data point, as is illustrated in the following plot. This example has been generated using the mtcars dataset that comes with R. The point-wise (squared) error is the (squared) length of the vertical line between the true data point (black) and the predicted point (blue) on the regression line.

Linear Regression 1D

The variables for the optimization algorithm are the coefficients c_i,\ i=0,...,n, and the objective function f(c) is nothing but a polynomial of degree 2 in these variables. In fact there is a structural similarity to some of the simple test functions from part 2 of this blog series, but now let us look at the concrete test case. As usual you can find all information and the R code in the github repository, the code is self-contained.

The algorithmic setup

As we want to check the scaling behaviour in higher dimensions I have decided to create artificial data for this test. The setup is as follows:

  • The first column of the m\times (n+1)-matrix X contains only 1s, the remaining n columns contain random numbers uniformly distributed in the unit interval [0,1].
  • We define an auxiliary vector \hat{c} of length n+1 by = [1,2,3,...,n+1].
  • We define a random vector z of length m – the number of data points – containing random numbers uniformly distributed in the unit interval, scaled to unit length. z is used as noise generator.
  • We define a weight \varepsilon – a real number that allows us to scale the noise vector z.
  • Finally, we define the vector y by = X\hat{c} + \varepsilon z.

The vector \hat{c} is by construction an approximate solution of y\approx Xc as long as the noise \varepsilon z is small. This setup might look somewhat complicated but it gives us a nice parametrization of the interesting things. The main parameters are

  • The number of data points: m.
  • The number of parameters of the linear model: n+1.
  • The amount of noise, i.e. a trivial upper bound of the expected value of the objective function: \varepsilon.

The last parameter allows for a quick sanity check, as \varepsilon^2 always is a trivial upper bound because we already know the possible solution \hat{c} with the property
f(\hat{c})=\varepsilon^2.

Furthermore it makes the scenario somewhat realistic. Typically you want to reconstruct the unknown solution – modeled by \hat{c} – but what you get from any algorithm almost always is a perturbed solution c^* that still contains some of the noise. Using this parametrization you can get a feeling for how much of the added noise actually is present in the solution, which might lead to a deeper understanding of Linear Regression.

Benchmarking Linear Regression

We compare the performance of the standard Armijo rule, the Armijo rule with widening, the exact stepsize for several choices of m,n and \varepsilon. We also include a single comparison for some choices of a fixed stepsize at the end, that indicate what you can expect from this choice and where it can fail. In all cases, the algorithm terminates if the norm of the gradient is below 1e-6 or if the number of iterations exceeds 100,000.

Impact of data complexity

The first test is about the simplest possible model, one-dimensional Linear Regression, i.e. the number of model parameters is 2. Don’t be confused by this, there is always one parameter for the “offset” c_0, such that the n-dimensional model depends on n+1 parameters. The optimization algorithm is initialized with the parameter vector c=(c_0,c_1)=(0,0). We vary the number of data points from 5 to 10,000 in order to experience the scaling with respect to the amount of data.

LinearRegressionTable1

That’s more or less the expected behaviour. The number m of data points can grow but the underlying optimization problem still has the same dimension n+1=2, thus the number of iterations remains roughly constant – given that there are sufficiently many data points – and the runtime increases linearly with m. It is interesting that the exact stepsize needs so few iterations making it almost as fast as the standard Armijo rule, this indicates the simple structure of the problem. Nevertheless, the Armijo rule with widening is the best choice here.

Impact of model complexity

The second test varies the number of model parameters from 2 to 16, the number of data points is fixed at 1,000. Here, we expect the scaling behaviour to be somewhat different.

LinearRegressionTable2

Still, the exact stepsize produces the smallest number of iterations but the difference is much smaller for more complex models which indicates that inexact stepsizes are a good choice here. On the other hand, the price for this slight advantage is prohibitively high. We can also see that not only the runtime per iteration, but also the number of iterations is increasing with the model complexity, this is something to be considered when choosing a model for a real-world problem.

Fixed stepsize

As promised we provide some results for fixed stepsizes as well. Conceptually, it does not make sense to use a fixed stepsize at all unless you do know in advance a good value that definitely works – which you typically don’t. In reality, you have to test several values in order to find something that allows the algorithm to converge in a reasonable time – which is nothing but a stepsize rule that requires a full algorithmic run in order to perform an update. But let’s look at the numbers.

LinearRegressionTable3

Only for the one-dimensional – and almost trivial – Linear Regression problem there is a value of the stepsize for which the performance is competitive, but this choice is way too risky for non-trivial settings. Even for slightly more complex Linear regression problems and a fixed stepsize of 1e-1 the performance is worse by a factor of 10. And for more complicated objective functions, e.g. the Rosenbrock function that has been introduced in the last blog post, the value would have to be smaller than 1e-6 implying that the number of iterations and the runtime would explode.

Summary

The results for Linear Regression already indicate that it is indeed useful to keep an eye on the underlying machinery of Optimization methods. The Armijo rule with widening shows the best performance, but the exact stepsize leads to the smallest number of iterations. These two findings imply that the direction of steepest descent is a good or at least reasonable choice for Linear Regression. The reasoning behind this is twofold. Remember that one argument for inexact stepsizes was that they can help avoiding the risk of overfitting that the exact stepsize cannot avoid in case the search direction is bad. Overfitting – which can be interpreted as significant deviation between the “local” search direction and the “global” optimal direction pointing directly to the solution – should lead to a higher number of iterations or at least not to less iterations. So in reverse, if the number of iterations is smaller, there can be no overfitting and the search direction should be a reasonable approximation of the unknown optimal direction.
The second argument – which directly applies to widening but also to the exact stepsize – considers the step length. Widening leads to larger steps which again only make sense if the local search direction is a reasonably good approximation of the global optimal direction in a larger environment of the current iterate. As a general rule: Larger steps imply that the model is a “good fit” in a larger environment of the current point. The exact stepsize also did produce larger steps which we did not mention here but you can check it on your own.
Feel free to take a look at the code in the github repository and give it a try yourself, you can even apply it to your own data. For other choices of the parameters the numbers can look differnt, sometimes the widening idea has no effect and sometimes it clearly outperforms the exact stepsize in terms of number of iterations, but my goal was to show you the “average” case as this is what mostly matters in practice.
The results for Logistic Regression are coming soon and here the impact on the runtime and the overall performance will be even more visible. Thanks for reading!

The Machinery behind Machine Learning – Part 2
The Machinery behind Machine Learning – Part 1

The post The Machinery behind Machine Learning – A Benchmark for Linear Regression appeared first on codecentric Blog.

Categories: Agile, Java, TDD & BDD

Extending the Stream API to Maps

Javalobby Syndicated Feed - Wed, 13-Jan-16 22:31

One of the bigest features in Java 8 is the ability to stream over collections of objects. By adding the .stream()-method into the Collection interface, every collection in the java language is suddenly expanded with this new ability. Other data structures like the Map-interface, do not implement the method as they are not, strictly speaking, collections.

In this article I will show you how Speedment Open Source streams efficiently over standard Java maps, expanding the Stream interface into something called a MapStream! This addition will make it easier to keep your streams concrete and readable even in complex scenarios. Hopefully this will allow you to keep streaming without prematurely collecting the result.

Categories: Java

Elixir, Phoenix and CouchDB — An Introduction

codecentric Blog - Wed, 13-Jan-16 12:45

The Elixir MVC Framework Phoenix

Marcel WolfMarkus Krogemann

Abstract

This blog post will show how a functional web application can be built with Elixir and Phoenix in a few, simple steps, without requiring a deeper understanding of functional programming or the Erlang/OTP system.
A small demo application will then help us understand how the Phoenix framework works and appreciate the technical merits of the Elixir ecosystem.
Follow–up blog posts will provide deeper insights into the framework and its components, such as Plug or Ecto. Special attention will be paid to connecting the web application with the document database CouchDB.

Phoenix, Elixir, CouchDB

Introduction

Elixir is a dynamic, functional programming language that leverages the Erlang VM (BEAM). It has a modern syntax that borrows from languages such as Ruby and supports advanced meta programming features, facilitating a smooth entry into the world of Erlang and its interesting runtime characteristics.

Erlang is a functional programming language that has been developed to build highly available, “always–on” systems in the telecommunications domain.
It is well known for its resilience, its transparent approach to building distributed systems and a developer friendly handling of concurrency (Actor Model).

The ecosystem around Elixir is still comparatively small, but is being quickly developed by a growing community of enthusiasts.

Phoenix

The web application framework Phoenix is one of the products that this community has spawned. It greatly simplifies the development of high–performance HTML5 apps, API–Backends and websocket applications.

CouchDB

When we first checked out Phoenix, it became apparent that database adaptors exist mainly for relational databases and MongoDB.
In order to stay within the Erlang ecosystem and to leverage the features of a document database, we decided to integrate with CouchDB.
CouchDB is a NoSQL database, developed in Erlang, that stores JSON documents and exposes its functionalities through an HTTP API.

Having found no readily available database connector that fulfilled all of our requirements triggered our decision to develop our own CouchDB connector in Elixir.
The current release of this connector supports basic create, update and read operations, as well as rudimentary view operations.
The code has been published as an open source project on Github.

Starting a Phoenix Project

In order to bootstrap a new Phoenix project, we make use of the Elixir command line tool mix and apply one of the tasks defined by Phoenix, in this case:

mix phoenix.new <app-name>

As we do not want to make use of a relational database, we exclude the default persistence layer Ecto by adding the –no-ecto flag. So, in order to create a project with name ‘foo’, we type:

mix phoenix.new foo --no-ecto

start phoenix project

After creating the files and folders that make up the project structure, the mix tool prompts us to download and install a set of dependencies — later on we will see where these are declared.

download dependencies

Change into the project root:

cd foo

We now have several options to start the application. To allow interaction with the application’s modules at runtime, we start the application within a REPL (Read–Eval–Print Loop), using Elixir’s interactive shell,  iex:

iex -S mix phoenix.server

phoenix server running 1

This brings up the (as yet empty, but fully functional) application, so that we can now reach it through a web browser on port 4000:

We will be greeted with the page shown below.

phoenix server running 2

Live–Reload

Phoenix has a very useful Live–Reload feature. A change that we make in the currently shown page will immediately become visible in the browser, without us having to manually trigger a reload of the page.
Assuming the application is in the state shown above and we change the underlying HTML template in

web/templates/page/index.html.eex

from this

welcome to phoenix

to this,

phoenix hello world

the web browser will immediately show the effect of that change and render the new state as shown below:

server running hello world

In the iex console, we will see the change being logged:

phoenix live reload

Project structure

Let’s now look into some of the newly created folders:

├── config

The config folder contains project configuration files. Some dependencies also expect their configurations to be stored here, an example being credo, a static code analysis tool.

The configuration mechanism also supports different settings per environment by simply storing them in the appropriate config files: dev.exs, test.exs, etc.

These .exs files (Elixir scripts) are compiled in memory each time they are run, which makes them ideal for storing configuration details and scripts that change often during development.

Some other, less frequently changing configuration artifacts live inside the lib folder.

├── lib
│   └── foo

The .ex files in this folder are compiled to .beam files and then loaded into the Erlang Virtual Machine (BEAM), which makes these files useful for storing higher level, less frequently changing configurations, such as endpoint and OTP supervisor/worker config. For performance reasons, such types of configurations should therefore be stored in lib or subfolders thereof.

├── priv

The priv folder is reminiscent of the Erlang/OTP heritage of Elixir. The OTP specification expects this folder to exist. Its name suggests private content, but in a Phoenix project, it actually contains static resources such as (minified, compile–time generated) JavaScript and CSS files. Database seeding scripts are also commonly stored in priv.

└── web

The Phoenix framework implements the well–known Model–view–controller architectural pattern (MVC).
The web folder contains MVC artifacts like controllers, models and views. In contrast to other MVC frameworks, such as Ruby on Rails, Phoenix differentiates between templates and views. The Templates are getting pre–compiled, which helps to reach the high performance that Phoenix has to offer.

Routing information is stored in router.ex. It contains a mapping of request paths to controller functions.

Static content sources (JavaScript, CSS, images) go into the static subfolder of web.

Phoenix is fully compatible with JavaScript and CSS frameworks as they are typically being used in modern web applications. Just drop them into the appropriate subfolders of web/static and use the configuration mechanism provided by brunch to influence their loading sequence (e.g. load jquery.js before bootstrap.js).

An example configuration (brunch-config.js) can be found in the demo project, which we will introduce next.

“Bird watch”

We have thus far looked at an essentially empty, yet functional walking skeleton project, which already showed some of the ways of working with Phoenix.
However, a real–world demo application allows us to address more of Phoenix’s features and characteristics.
Such a demo application has been made available in a Github repository.

Check out the source code:

git clone https://github.com/mkrogemann/bird_watch

cd bird_watch
mix deps.get

Here, we have made use of the mix build tool to resolve and download the project’s dependencies. mix can also facilitate creating, compiling and testing Elixir projects. We will see more of it in future episodes.

We can also use mix to pre–populate the database by calling an application specific seeding script:

mix run priv/repo/seeds.exs

Now we can start the application

mix phoenix.server

and open it in a web browser:

phoenix bird watch running

Conclusion

In a small number of simple steps, we have created a functional web application and shown how easy it is to make changes that are immediately visible, thanks to the perfect implementation of Live–Reloading. This feature is a real productivity boost.

Another nice trait is the excellent performance of Phoenix web applications. Check out these impressive benchmarks to appreciate its throughput and latency figures.

We will continue development of the demo application “Bird watch” in follow–up blog posts and show more of the advanced functionalities and characteristics of Phoenix, Elixir and OTP.

 

The post Elixir, Phoenix and CouchDB — An Introduction appeared first on codecentric Blog.

Categories: Agile, Java, TDD & BDD

Ted Neward's 2016 Tech Predictions

Javalobby Syndicated Feed - Wed, 13-Jan-16 01:29

As has become my tradition now for nigh-on a decade, I will first go back over last years’ predictions, to see how well I called it (and keep me honest), then wax prophetic on what I think the new year has to offer us.

In 2015…

As per previous years, I’m giving myself either a +1 or a -1 based on a purely subjective and highly-biased evaluational criteria as to whether it actually happened (or in some cases at least started to happen before 31 Dec 2015 ended).

Categories: Java

Markus Eisele Answers Questions About Java EE and Microservices

Javalobby Syndicated Feed - Wed, 13-Jan-16 01:19

I took the chance to catch up with Markus Eisele, Developer Advocate JBoss Middleware, this week about microservices and containers in the context of Java EE. There's been a lot of discussion around whether Java EE is really up to the task, or if the Spring ecosystem hasn't already outpaced EE. In this interview, Markus gives us the Java EE side of the discussion.

You've been talking about microservices in an Enterprise and Java EE context a lot lately. Why?

Before joining Red Hat, I worked as a consultant for various big enterprises and helped them built individual solutions based on Java technology both on the client but mostly server side. This has been a tremendously challenging, but also rewarding time of my life. Working in such environments quickly turns all the easy shopping applications into toy apps. You start to talk about integrating host systems, accessing age old databases and processing large amounts of data. And thanks to platform strategies and optimizations in operations most of the solutions we could propose weren't allowed to use the right tools for the job. It mostly burned down to tweaking the best out of what we had. What I learned in those years is something I want to pass on. I want to help developers put recent buzz-words (e.g. Microservices, Cloud, Container, DevOps) into perspective. Especially into one that works in enterprise settings. And those are for sure different.

Categories: Java

New Languages Will Not Fix Maintainability Issues

Javalobby Syndicated Feed - Wed, 13-Jan-16 00:16

Have you ever run into a new or trending programming language (Rust, nim, Kotlin, etc.), promoting all sorts of great "new" abstraction mechanisms (traits, mixins, multiple-inheritance, monads, lambdas, macros, you name it), thinking: "I could finally get rid of all this copy-pasted or boilerplate code we have in our codebase!"?

I discussed whether newer/trendier languages will lead to less copy-pasted code (or even better code quality in general) with a few colleagues over a few beers recently. Based on our experience as consultants for software quality and Teamscale developers, we quickly came to the conclusion: No, they will not.

Categories: Java

Writing a Slack Plugin is One of the Easiest Things to Do... Sort of

Javalobby Syndicated Feed - Tue, 12-Jan-16 23:02
Slack is the little chat app that could. It’s had a meteoric rise to becoming one of the most popular productivity tools amongst developers. Whilst it’s great as a chat app on its own, one of the reasons it’s become so popular is the huge plugin ecosystem that has grown around it. Slack has integration with calendar systems, weather systems, log scrapers, deployment tools and much more. You can really do anything from it.

Where this really gets exciting is in creating your own Slack integrations. You can hook your Slack system up to your internal processes, and best of all it’s exceptionally easy to do. Slack has created a simple yet powerful API- all you need is a REST service. In this tutorial I’ll be showing you how to do just that to create your own slash command to display images of Nicholas Cage in your Slack chat, taking advantage of the PlaceCage API.

Creating a Custom Slash Command

In slack, a slash command is a forward slash, followed by the command, and then some optional parameters. For example, “/remind me in ten minutes to get a drink” will remind you in ten minutes to get a drink. We can create our own slash commands to do almost anything we want.

Categories: Java

My First Rapiro Cyborg

codecentric Blog - Tue, 12-Jan-16 02:57

I love IoT, but I really hate talks in which all the great concepts are explained and finally when you get the chance to actually see something, all you get is some LEDs that light up and change their color. If you attend your first talk on this topic, you’re probably happy, but when you sit in your third or fourth talk, it really starts to get boring. After all it’s IoT not IoL (Internet of Lights).

I wanted to prepare a talk on using Flex as a frontend for an IoT system, so after complaining about blinking lights, I thought: what’s cooler than blinking lights? Well just about anything is cooler, but my own Cyborg – that’s almost as cool as it can get.

my-first-cyborg-1

Hello Rapiro!

So I started to look for some driving, walking, whatsoever robot kits and stumbled over Rapiro. At first glance he’s definitely cute. At second glance I think he’s even cooler as he’s a good platform to play with and demonstrate a lot of different things. He’s big enough to be visible at a conference presentation. Small enough to take with you, even on a plane and with a big hollow head in which you can store all sorts of electronics. When I found out it is possible to extend the relatively primitive Arduino that comes with Rapiro with an Intel Edison by using a carrier board especially designed for Rapiro, it was sold.

Ordering my Rapiro from Germany was pretty easy and extremely fast. I wouldn’t manage to get to Japan in the time Rapiro managed to come to Germany. Anyway, the hardest thing was explaining the customs guys what’s in the box. In Germany you need a classification for EVERYTHING. For $500 (Rapiro + Edison carrier + a little spare parts) they decided it’s not a toy – so what is it? It’s not an industrial robot and it can’t really transport things. There is no “Walking and dancing robot” category in German tax classifications. So it took them about 30 minutes to figure out what he was. In the end they settled with “0000” which seems to be “We don’t know what it is, just apply the maximum tax on it”. In the end I payed my taxes and Raprio and I were good to go.

Note: I recommend ordering directly form Switch-Science, as I have read about some problems with other suppliers that didn’t fill out all of the papers correctly or didn’t add the CE certificates for the electronics. Yet another suggestion would be to order one or two spare servos of both sizes.

Building Rapiro

Note: If you are planning to include the IR Sensor, I would recommend soldering the pins to the Arduino and do the cabling before assembling Rapiro or you will have to disassemble most of him again.

Building Rapiro is extremely straight forward. The documentation is great and it should almost assemble itself automatically.

One thing I did notice, though, was that even if I did follow the instructions (i.e. I connected the servos and made them initialize before assembling Rapiro and took care not to change their position during assembly), Rapiro did some crazy moves when I switched him on for the first time. The reason was that some servos now initialized to different positions. So I had to disassemble parts, adjust the servos and re-assemble them. But now he’s standing tall awaiting my commands.

Connecting to Rapiro’s Arduino

Note: I would recommend NOT connecting the cable from Rapiro’s Arduino to the Edison for now as it seems there are problems using the Arduino directly if this cable is connected. I guess it’s as if you connected two devices to the same COM port. I had a lot of trouble with this prior to noticing this.

The micro USB port on Rapiro’s right hand side is the serial terminal to Rapiro’s Arduino. Now it was possible to attach Rapiro to a computer and immediately start talking to him. I am using a Mac, so I needed to install the driver for the virtual COM port first. After installing the driver and Arduino IDE I was able to upload my first sketch. The first tasks are to iteratively adjust the trimming values for the servos in the Arduino sketch so the little guy isn’t shivering because the servos are in some half ideal position.

One thing I noticed was that after uploading a sketch I sometimes had to disconnect the USB connection prior to uploading a new version (I had to do this quite often during the initial calibration). Without doing this, all I got was some nasty looking errors when trying to upload for the second time.

So after all servos were correctly calibrated, it was time to have the little guy do his first steps. For this I simply opened a “Serial Monitor” in the Arduino IDE. Inside this I then sent the initial #M1 which makes Rapiro walk forward and what should I say? He started walking. So I played around a little with the other “movement” presets, and was happy.

Here is a list of all commands of the initial sketch: http://wiki.rapiro.com/page/serial-command/

Setting up the Edison

my-first-cyborg-2

After the Arduino had been correctly configured and Rapiro was able to do his basic things, my next step was to setup and configure Rapiro’s brain. First I connected my computers USB cable to Edison’s micro USB (the one on the left hand side if you face Rapiro – as seen on above image) and had the Intel Phone Flash Tool Light install an image of the latest firmware. After the Edison had the latest firmware running, I needed to connect the micro USB cable to the other micro USB port of the Edison (the one on the right hand side if you face Rapiro). This is also a serial terminal port. To connect to the Edison, simply use this command:

screen /dev/xx.usbserial-XXXXXXXX 115200 –L

If the screen is simply clear, be sure to press “Enter” once or twice as the “screen-saver” might be on. In it’s initial state you can simply logon with the root user.
After doing this my first thing to do was to configure the WiFi connection

configure_edison --wifi

and to enable the WiFi SSL access by setting a password

configure_edison --password

Now I was able to disconnect the USB cable and log in to the Edison using a normal SSH client. That was easy!

Setting up the Edison to communicate with Rapiro’s Arduino

The Edison comes with four COM ports enabled:

  • /dev/ttyGS0: Mini-USB Port at the back with which you can directly log in to Rapiro from PC/Mac using a virtual serial connection
  • /dev/ttyMFD0: BlueTooth serial connection
  • /dev/ttyMFD1: Serial UART connection using the on-board Pins (the one needed to communicate with Rapiro’s Arduino)
  • /dev/ttyMFD1: Linux Kernel Debug (no idea what this is)

In order to send commands from the Edison from the command-line, you need to install some sort of terminal. Later on I’ll send the commands from Java directly where I don’t need a terminal, but for testing it’s always good to have one. Edison’s default Linux uses OPKG to install packages. But in order to find packages we first need to add a package repository:

  1. Log in to your Edison
  2. Edit /etc/opkg/base-feeds.conf on the Edison
  3. Add the following lines to the (usually empty) file:
    src/gz all http://repo.opkg.net/edison/repo/all
    src/gz edison http://repo.opkg.net/edison/repo/edison
    src/gz core2-32 http://repo.opkg.net/edison/repo/core2-32
  4. Run opkg update to fetch the list of available packages. (You should resist the temptation to run “opkg upgrade” as I have read that this will break things)
  5. With opkg install screen you install the terminal program “screen”

So after installing the terminal program we can start sending commands to the Arduino:

  1. With
    screen /dev/ttyMFD1 57600 -L

    you open a terminal connection to Rapiro’s Arduino (no worries if the screen is empty … that’s the way it should be … and be sure to use the 57600 baud rate as the Arduino sketch configures the COM port to use that rate)

  2. Now try typing #M1 and your Rapiro should start walking forward
  3. With #M0 it should stop.

I did notice that as soon as you type # Rapiro looks like he’s frozen or having mechanical problems. So in this case, don’t worry, it’s just the software not being implemented too well. I am certain that I’ll write a new sketch for the Arduino that is more optimized for sending commands from Edison over time.

References:
Description of Edison’s COM ports: https://communities.intel.com/thread/54236?tstart=0
Where I got a hint on what “ttyMFD0” actually is: https://android.googlesource.com/platform/hardware/bsp/intel/+/b4e503caf3329e5230441ff6c2a2849a0fa4932f%5E!/
How to seup the OPKG Edison repo: http://alextgalileo.altervista.org/edison-package-repo-configuration-instructions.html

Memory

As I want do to some quite sophisticated stuff with my Rapiro and the Edison carrier board comes with a microSD slot, I got my hands on a really fast 64 GB microSD card. This should be enough for playing around. After all I don’t want a walking Hadoop/Cassandra cluster following me everywhere – even if that might sound cool.

Since the Edison’s Linux distribution doesn’t directly support FAT32, I decided to re-format the SD card with EXT4. But if you are used to formatting stuff with FAT16/32 or NTFS, formatting a 64GB SD card with EXT4 does take a minute or two … or several hundred. The first time I tried, my little friend ran out of energy before finishing the job so the second time I connected the Edison’s terminal port to an USB port for energy and restarted the formatting process. In this way, the little guy doesn’t wear out his servos, as he tended to get nervous tremors in some of his servos every now and then. Doing some math I guess spending the extra bucks on the super-duper SD card was a waste of money as the Edison and/or carrier board might not provide the speed to utilize this. In the end I think formatting my 64 GB took about 3-4 hours.

First I needed to find out under what name my SD card is available:

fdisk -l

This prints out a long list:

Disk /dev/mmcblk0: 3909 MB, 3909091328 bytes
1 heads, 16 sectors/track, 477184 cylinders
Units = cylinders of 16 * 512 = 8192 bytes
        Device Boot      Start         End      Blocks  Id System
/dev/mmcblk0p1               1      477185     3817472  ee EFI GPT
Partition 1 does not end on cylinder boundary
Disk /dev/mmcblk0boot1: 4 MB, 4194304 bytes
4 heads, 16 sectors/track, 128 cylinders
Units = cylinders of 64 * 512 = 32768 bytes
Disk /dev/mmcblk0boot1 doesn't contain a valid partition table
Disk /dev/mmcblk0boot0: 4 MB, 4194304 bytes
4 heads, 16 sectors/track, 128 cylinders
Units = cylinders of 64 * 512 = 32768 bytes
Disk /dev/mmcblk0boot0 doesn't contain a valid partition table
Disk /dev/mmcblk1: 63.8 GB, 63864569856 bytes
255 heads, 63 sectors/track, 7764 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
        Device Boot      Start         End      Blocks  Id System
/dev/mmcblk1p1               3        7765    62351360  83 Linux

So mmcblk1 seems to the the name of the SD device when using it on the Edison carrier board. At least judging from the 63.8GB of space.

Next I needed to switch the FAT32 partition on the SD card to an EXT4 one:

fdisk /dev/mmcblk1

Here you need to hit “t” to “change a partition’s system id” and then use the id “83” to switch the partition type to “Linux”. To save the changes, hit “w” (write) at the end.

After that I started the formatting process:

mkfs.ext4 -m 0 -L memories /dev/mmcblk1p1

and then mounted it with:

mkdir -p /mnt/memories
mount -rw -t ext4 /dev/mmcblk1p1 /mnt/memories/

OK … this way you still need to mount it every time you start, so add this to your /etc/fstab file:

/dev/mmcblk1p1  /mnt/memories  ext4  defaults  0 0

This way you can use mount /mnt/memories and the system will know what to do, but it doesn’t quite do it yet. In order to have the partition auto-mounted at startup you finally need to create a script that is executed while starting up. I couldn’t find one in the typical places, and it seems you have to manually do this:

mkdir -p /etc/init.d
touch rc.local
chmod +x rc.local
vi rc.local

It doesn’t actually have to be called that way, but this way it’s at least a little aligned with established defaults and it has to be executable.

Inside rc.local all we need is one line:

mount /mnt/memories

To tell Edison’s OS about the script we finally registered it to be called at startup:

cd /etc/init.d
update-rc.d rc.local defaults

Done … next time you reboot, the SD card should be automatically mounted.

Summary

At this point we have a working Rapiro with an embedded Intel Edison running on the Edison carrier board, we extended the available disk space by utilizing a microSD card inserted into the SD card reader of the carrier board and this is automatically mounted when the system starts. Edison is accessible using SSH via WiFi and we can log on to it remotely and send commands to Rapiro’s Arduino and thereby remote-control him … I think that’s quite nice for a start, but it’s only the start.

To be continued …

I’ll continue this little series with probably a little more demanding tasks, but my toys have to be delivered first. In part 2 I’ll cover:

  • Connecting the Rapiro IR proximity sensor to his Arduino
  • Adding sound I/O to Edison using a WM8731 circuit board
  • Adding vision including object detection to Edison using an insanely cool Pixy CMUcam5

So get your soldering irons and 3d printers ready.

In part 3 I’m going to deal with software running on the Edison and probably my optimized Arduino sketch.

The post My First Rapiro Cyborg appeared first on codecentric Blog.

Categories: Agile, Java, TDD & BDD

Using Wiremock to Test Against 3rd Party Services Offline

Javalobby Syndicated Feed - Tue, 12-Jan-16 01:45

I’ve recently been playing Stockfighter, the programming game released by Patrick McKenzie. It’s great fun and you should totally try it. The game is based mostly around a REST API, and so I ended up needing to create a basic Java abstraction of the API.

Being a bit obsessed with TDD I wanted to create some tests for this first. Writing code to call third party APIs and testing by calling the external service can be immensely frustrating and slow, so I wanted a better way to stub out standard responses. I’m also currently travelling a lot, often without internet, but wanted to be able to keep coding up against the API having saved the docs offline (thanks evernote!)

Categories: Java

One Small Error in Java Code, One Giant Leap for Java Code Quality

Javalobby Syndicated Feed - Mon, 11-Jan-16 22:42

In this series of posts I am covering some of the many ways you can benefit from XpoLog and especially on how to get the most valuable information from your log4j event logs.

By running the XpoLog Analytic Search on your log4j data, you can among other things measure your application performance and thread activity, measure code activity with class and method analytics on log4j, build security analysis, and make use of various visualization gadgets for maximum analysis.

Categories: Java

Thread Slivers eBook at Amazon

Syndicate content