RESTful web services with JAX RS 2

Recently I had the chance to present to the Brisbane JVM group on JAX RS 2 which was released last year. I gave a brief overview of the basics of JAX RS followed by a run through of some of the new features in JAX RS 2 - client api, filters and interceptors and ayschronous resources.

Thanks to Rob the video is now up at vimeo - http://vimeo.com/88095270, the slides are available on slide share - http://www.slideshare.net/benjaminedwinmorgan/jax-rstalk and the code is on github - https://github.com/nebnagrom/qldjvmJaxRs2

Jersey and Maven Jetty plugin

Recently I ran into an issue working with Jersey and the Jetty Maven plugin. The application is built for Glassfish so I had marked the Jersey libraries as provided. In the Maven Jetty plugin configuration I put Jersey in the configuration for the dependency but when I went to start the server I got the following error:

2014-02-04 10:12:25.320:WARN:oejut.

QueuedThreadPool:qtp1736120767-24:
java.lang.IncompatibleClassChangeError: org/eclipse/jetty/annotations/AnnotationParser$MyClassVisitor
    at org.eclipse.jetty.annotations.AnnotationParser.scanClass(AnnotationParser.java:971)
    at org.eclipse.jetty.annotations.AnnotationParser.parseJarEntry(AnnotationParser.java:953)
    at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:906)
    at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:828)
    at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call(AnnotationConfiguration.java:111)
    at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run(AnnotationConfiguration.java:472)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
    at java.lang.Thread.run(Thread.java:744)

 

After a bit of digging around I found that changing the Jersey dependencies to compile resolved this issue. I figure I must be doing something bad to the Jetty class loader where by loading in the container's classloader is breaking something.

In the end I settled for setting the dependency scope to 'test' and adding the test classpath on the Jetty classpath. This is not ideal as it doesn't really express the dependencies properly of the application.

Hopefully this helps someone else who runs into this issue!

Functional Programming Principles in Scala and Principles of Reactive Programming on Coursera

At the end of 2014 I completed two Coursera courses on Scala and functional programming. I have been slowly trying to learn Scala and these courses were a great help for me to learn the language. This post is about my experience with the course and thoughts on it.

The first course I did was Functional Programming Principles in Scala - https://www.coursera.org/course/progfun which goes through the basics of functional programming in Scala. I found this course to be quite fun and the assignments were good. There was one week, I think week six where I felt the assignment was hard and took more time than the others.

The topics that I found particularly interesting were:

• immutable collections
• pattern matching
• for comprehensions and fold, map and filter
• lazy evaluation and infinite streams

The second course was Principles of Reactive Programming - https://www.coursera.org/course/reactive which goes through reactive programming in Scala. The format was the same as the first course but had three different lecturers. This course felt more like a run through of several different frameworks which was good but not quite what I expected.

I found the lectures in week three and four to be interesting and they covered futures and stream processing with the Reactive Extensions for Java (https://github.com/Netflix/RxJava) which is a port of the .Net RX framework. The RX framework is definitely worth investigating and offers some very nice ways of handling Futures and asynchronous code. I was also lucky enough to see the talk on RX at the 2013 Yow conference by Ben Christensen and this course really helped me understand how it is being used at Netflix.

Weeks five, six and seven were focused on the Akka framework which uses the Actor model. I found the first assignment for Akka to be quite challenging, mainly around having to learn a bunch of ancillary stuff for the framework which was not related to the assignment. This was good however as it gave me a feel for what Akka would be like to configure and setup, although I'm not sure that I am smart enough to right concurrent code that is correct!

Overall I thought the courses were very good and I enjoyed them for the most part. This session was the first time the Principles of Reactive Programming has been run and I think that it showed a little as the assignments were not as polished as the first course. It is a real privilege to be able to enroll in these courses for free and many thanks must go to the organisers for putting this on.

Govhack Brisbane 2013

A month ago I attended the Govhack (http://www.govhack.org/) event which was run around Australia to see what could be done with Government data in a weekend.

This was quite a bit of fun although I only had a day available to work on it the team had a cool idea for planning a trip using the Australia tourism data. We called it travelhackz and the summary is available at http://hackerspace.govhack.org/?q=groups/travelhackz

Big thanks to the rest of the team for their efforts on this!

Lamdba Jam 2013 (Brisbane)

A month ago I went to the YOW Lambda Jam conference in brisbane - http://www.yowconference.com.au/lambdajam/. This was quite a bit of fun and had plenty of interesting talks.

One thing that I do plan on looking up is the iteratee pattern in the play framework (http://www.playframework.com/documentation/2.0/Iteratees). This was presented by Nilanjan Raychaudhuri and looks very interesting, I plan on having a look at this sometime soon.

Hack and Heckle

Recently I joined Darren and Leigh in their Hack and Heckle podcast (http://hackandheckle.com/). We talk about technology and the news that we've seen in the week as well as the local meetups and whatever side projects we happen to be working on.

This has been quite a bit of fun so tune in if you are interested.

Jersey, JAXB and json with generics and inheritance

A month or two ago I ran into an issue where I was marshalling Objects to json using Jersey with JAXB annotations on my pojos but only the Super Class attributes were being marshalled. I have uploaded some code to github to demonstrate this:

https://github.com/nebnagrom/jaxbInheritance

In this contrived example we have a Super Class House which has a member variable which is a String and an @XmlRootElement annotation. There are three child Classes of House:

• BlueHouse which has an additional Integer
• RedHouse which has an additional String
• GreenHouse which has an additional Boolean

We then also have a HousingEstate<T extends House> which wraps a typed collection of houses. In this example it has no behavior but in the situation I had we had behavior on this collection as well.

The issue occured once I started trying to return HousingEstate objects from my RESTful resource. When you invoked the resource you only got back member variables from the House class, i.e. something like this:

 {"houses":[{"houseName":"first blue"},{"houseName":"2 blue"}]}

 When what I wanted was more like:

{"houses":[{"houseName":"first blue", "blueAttribute":1},

           {"houseName":"2 blue", "blueAttribute":2}]}

After a bit of digging I found that I needed to add another annotation on the House class - @XMLSeeAlso like this:

@XmlSeeAlso({ GreenHouse.class })

After I added that I started getting the results I wanted.

I also tried this without generics or just using the Super Class but this did not work either. Interestingly I found that:

1. Using no generics meant that I got House member variables only, even when I annotated BlueHouse with annotation and added BlueHouses to the collection
2. Using the parent class as the Generic gave me Marshalling exceptions, probably to be expected though.

I haven't really tried to dig into what is going on here, I assume that there is something with generics and type erasure that makes it impossible for JAXB / Jackson to know that there is a child class that it should be looking at for the marshalling.

Stack trace of error you get with the House based version over the fold:

Cryptography in Java

I spent an evening recently playing around with cryptography in Java. I had to look up something similar for work and then it piqued my interest so I had a bit of a play with it.

What I was looking to do was AES encrypt a file using a key derived from a passphrase. After some digging around it seems that the way to create a key is to use the PKCS #5 algorithm and hash the result. This can be done in Java with a key factory using the algorithm PBKDF2WithHmacSHA1:

    private static SecretKey generatePBKKey(String password) throws NoSuchAlgorithmException, InvalidKeySpecException {

        SecretKeyFactory keyFactory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
        SecureRandom rand = SecureRandom.getInstance("SHA1PRNG");
        byte[] salt = new byte[16];
        rand.nextBytes(salt);
        KeySpec pbeKeySpec = new PBEKeySpec(password.toCharArray(), salt, 2, 128);
        SecretKey generatedKey = keyFactory.generateSecret(pbeKeySpec);

        SecretKey encKey = new SecretKeySpec(generatedKey.getEncoded(), "AES");

        return encKey;
    }

This key can then be used to initialise a AES Cipher which you can then encrypt the file with using a CipherOutputStream. Creating the AES Cipher was done using AES with ECB (Electronic Code Book) and PKCS5 padding.

        Cipher cipher = Cipher.getInstance("AES/ECB/PKCS5Padding");
        cipher.init(Cipher.ENCRYPT_MODE, key);

 I wasn't sure what block mode and padding to use and this seemed to be what most use.

The best doco I could find from Oracle was the Java 6 guide which ran through the main classes - http://docs.oracle.com/javase/6/docs/technotes/guides/security/crypto/CryptoSpec.html

 The full code of what I ended up with is over the jump.

Build failure in Windows with Maven

I had an 'interesting' problem with Maven and Windows today where the build was broken on a Windows machine but worked on Linux. All we were getting were weird cryptic errors that a class in the test directory could not resolve a class from the src directory. The build worked fine in the ide on windows just not when run on the command line.

Turns out the issue was that we had a lower case name for the package in src/main/resources but an upper case name in src/main/java. During the build the resources are copied over first this means that the directory was created as lower case. Then when the source was compiled it placed the class files in the lower case directory as Windows paths are not case sensitive.

For example we had src/main/java/FOO and src/main/resources/foo when this is built with maven you will get your class files compiled into the package 'foo' even though anything that imports this class will be looking for 'FOO'.

Hopefully this helps someone else so they don't spend a day pulling out their hair over this!

Hibernate caching with multiple objects mapped to one table

I recently had to do some performance tuning on an old application which for some reason had a database table that was mapped to two different domain objects. The mapping is bizarre because the domain objects are actually the same thing but for some reason there are two representations in this application!

One thing I wanted to setup was the second level cache to see if this would improve performance however once I discovered the crazy mappings I wondered how hibernate was going to cache this.

I didn't have enough time to really dig into what Hibernate was doing but based on the cache logs it appears that it knows which tables the entities it is caching are from and invalidates the cache for both when one is updated. Interestingly it also maintains the query cache appropriately so that a fetch for one entity will prime the cache for the other!

I was pretty impressed by this as I was expecting these crazy mappings to confuse hibernate but it seemed to handle it fine. This was an older version though so hopefully it is still valid in the later releases.

If you are interested this was done using Hibernate 3.2 and eh cache 1.3 (ancient!).

The Business Rules Curmudgeon

Its been a while since my last post, in that time my wife and I have had a baby boy who was born in the 9th of June.

Something that I have been pondering for a while is how to reduce the number of small business rules that seem to creep into requirements. Taken individually these seem minor but as they add up you find yourself maintaining dozens of different small rules that in aggregate are complex. This is particularly the case where you have a feature that grows over time and you steadily add more pieces to it.

The part I struggle with is how to say no to these things without sounding like a curmudgeon. The upfront development cost is usually small but when multiplied out across analysis, testing and training materials. I haven't really worked out a way around this, small tasks will disappear within estimates and people often become fixed on their vision and persuading them that it is wrong is difficult or impossible.

Perhaps its time to embrace being a curmudgeon.

Font Crashing / Terminating jvm in Linux with X11 forwarding

I ran into an issue recently where when a certain report was run on a specific server the jvm would disappear without a trace! There were no core dumps, no errors in logs just a 500 response from Apache  and the process was gone.

I struggled with this for a while as I could regularly reproduce the issue but only on one server. By pure co-incidence I could see the terminal when the application was started and there was a cryptic X11 error message:

X connection to localhost:10.0 broken (explicit kill or server shutdown)

This server also turned out to be one where we had to sudo su to a different user to start the application and had hacked around with the Magic Cookie for X11 forwarding. This mean that although I had X11 enabled on my session it was broken.

Removing the X11 forwarding from my ssh session fixed the issue. I never managed to work out if it was a bug in the JVM or if it was my broken X11 session sending a kill signal to the process,  full details of my digging over the jump.

Java Puzzles

In the Java Specalists newsletter they had a link to some java puzzles from http://wouter.coekaerts.be/puzzles which were rather interesting. The first one had me for a while till I read the hints about hashset and I'm still not quite sure about the second puzzle.

My original solution used Thread.sleep which did not work but after some googling I changed it to wait on the sleeper which did work. I'm not exactly sure why this is but it is to do with Thread.sleep not releasing its lock on the synchronised block whereas the sleeper.wait means that the current thread is now waiting on the original thread.

Hopefully when they publish the solution it will be a bit more clear what is going on here.

package dream;

import sleep.Sleeper;

public class Dream {

    public void dream(Sleeper s) {

        DreamThread dreamThread = new DreamThread();
        dreamThread.sleeper = s;

        new Thread(dreamThread).start();
        try {
//             Thread.sleep(100);
            s.wait(100);
        } catch (InterruptedException e) {
        }
    }

    private class DreamThread extends Dream implements Runnable {

        private Sleeper sleeper;

        @Override
        public void run() {
            sleeper.enter(this);
        }

        public void dream(Sleeper s) {
           
            try {
//                 Thread.sleep(500);
                s.wait(500);
            } catch (InterruptedException e) {
            }
        }
    }
}

Hibernate Sequence generation

A while ago I was working with an older version of hibernate (3.3 I think) and had to do a few thousand inserts. Whilst hibernate is not necessarily the tool for this job I was using it to do the insertion and found it spent a large amount of time fetching the primary key id one by one from the database sequence.

To get around this you can use the TableHiLoGenrator but this means that your sequence values are not what you are actually inserting into the database which means that if you wanted to use the sequence later you will need to do some stuffing around.

I had a look at implementing my own sequence generator that relied on the sequence using an increment n and then the generator is configured with this value so that it only fetches the sequence value every n times. I got this working but then we ended up not using it as we found a better way of managing the bulk insertions.

Deadlocks in oracle 11g

We ran into an issue recently where we would get deadlocks on Oracle 11g but not on 10g. After some investigation it turns out there is a change in how Oracle locks rows in 11g which means that insertions to a table with a foreign key also lock the candidate key in the referenced table!

The fix was to add indexes to the column in the referenced table so that Oracle can use this instead of locking the row.

This blog post by Richard Foote had a good explanation of the issue.

Polymorphism and if statements

This post probably falls into the category of 'stating the obvious' and has been covered many times before but I figured I'd put the thoughts rolling around my head up here anyway.

Basically whenever I see a big block of code like this:

Car car

if (car.isBlue()) {
    car.doBlueStuff();
} else if car.isRead()){
    car.doRedStuff();
}

I kind of cringe and wonder if it could be done polymorphically with something like

BlueCar extends Car {

    doStuff();

}

RedCar extends Car {

    doStuff();

}

so the if statement becomes

car.doStuff();

Polymorphism 101.


Obviously things are not always as simple or clear cut as this but big switching if blocks with several branches should at least make you pause and think if this is the best solution.

Stuffing around with LVM on ubuntu

A while ago I installed Ubuntu and went a bit nuts with my partition table with separate partitions for /var /home /tmp etc. etc. I also setup raid and LVM with the intention of having a raid 1 backup partition and a logical volume for myth tv.

One of the results of this was that I didn't put enough space on /home which I have been working around by creating symlinks and directories that live in the large LVM partition I setup for myth tv. My plan to fix this is to remove one of the physical volumes from the myth LVM, create a new volume group for home and move my data into this group. I could create one big volume group with two logical volumes for home and myth data but I wanted to try creating a group from scratch.

This has been irking me for a while so I am finally going to fix it, below is what i ended up doing:

Removing a physical volume from the lvm group

As I used JSF for the format of the logical volume I couldn't reduce the size of the file system so I ended up just running

lvreduce --size 148G /dev/mapper/mythLVMGroup-mythData

which would have the result of losing any data that happened to be in the rest of the volume, I then reformatted it with

mkfs.jfs /dev/mapper/mythLVMGroup-mythData

at this point I could then modify the logical volume by removing the device from the group:

pvmove /dev/sdb8
lvreduce --size 148G /dev/mapper/mythLVMGroup-mythData

pvmove is rather slow so I started writing this post whilst I waited for that to complete. I should have just used the -A n flag as I had deleted the data prior to doing this anyway, also as there was no data it failed. One side result of this was the UUID of the device changed (not sure if was the mkfs or the change to the lvm) so I had to update my /etc/fstab with the new id which I found by running
ls -la /dev/disk/by-uuid/
After this I remove the physical volume from the group:
vgreduce mythLVMGroup /dev/sdb8
At which point I have completed my first step of removing and resizing the mythtv volume.

Setting up the volume group
There are plenty of other guides around about setting up LVM etc. but here is what I ended up doing:
Create the volume group from two partitions:

vgcreate -s 32M homeLVMGroup /dev/sdb1 /dev/sdb8

then create the logical volume:
lvcreate --size 259G -n home_lv homeLVMGroup
now to format it

mkfs.ext4 /dev/homeLVMGroup/home_lv

at which point I mounted it, copied my old home files across and then edited fstab to mount it as home when I rebooted. I did make one stupid mistake here where I had formatted it as ext4 but then had the fstab entry as ext3!

References
I mainly used the following two pages and google to help me get this working:
http://www.walkernews.net/2007/07/02/how-to-create-linux-lvm-in-3-minutes/ 
http://tldp.org/HOWTO/LVM-HOWTO/reducelv.html

badness = 10000

Well I have been meaning to setup a blog for a couple of years now and am finally getting around to it.

The title of the blog is taken from the error messages that LaTeX produces when it runs into trouble formatting, for some reason I always found this error message amusing.