Joel's Blog

There were some issues with some of the classification tests I ran on the chessboard pictures from last week (you can find pictures here).  The ones that were taken from a distance of a couple of meters or less, classified into the correct directories.  But the ones that were taken from far away had this strange error that stopped the analysis and left the pictures were they were (in an un-detected state).

On further study I found out that the reason for the error was that the chessboard points were being inexplicably scattered all over the chessboard.  That is, even though the algorithm had found the chessboard it placed the corners in strange places.  It even calculated a chessboard that did not comply with the initial size given.

I followed the error until the cornerSubPix function call.  The initial findChessboardCorners call was doing its job; but when I tried to further increase accuracy with cornerSubPix, the calculation went wrong for chessboards that were “small”.  After some reading to the opencv documentation and a little experimentation, I found out that the culprit was the winSize argument cornerSubPix.  I don’t know exactly what this argument defines; but if it’s too big, it will damage the calculations made by findChessboardCorners.  To solve the problem I decreased the argument from 11 to 5.

I believe that this argument defines some sort of window size.  If two points manage to fit in one window -because the window is too big, or because the squares are too small (far away)-, then the calculations in cornerSubPix damage what findChessboardCorners calculated.  I’m also guessing that decreasing this argument will make the algorithm more complex.  But all this is just a hunch.

In any case; after making the change to the ilac source I re-ran the test on the images that were left undetected in the previous experiment.  All but three of the remaining pictures went into their respective class directory.  There were two images where the chessboard could not be found.  This is because the picture is taken from very far away (5 meters or more).  I was expecting the algorithm to fail at these distances and was not surprised.  Moreover, we are worried about the accuracy at around 2 meters.

There was one case that misclassified the image.  Though the chessboard was found, the id was miscalculated.  I think this was due to the distance of the picture.  Since the chessboard was so far away from the camera image plane, the number of pixels that made up a square were too small to serve as an accurate calculator.  I’m assuming that this happens only with pictures taken from afar and will ignore this for now.

After discovering that the accuracy of the chessboard detection is increased by adding a black frame around the chessboard, I went on to create a python library based on what I already had.  I created a python/C++ interface and some python routines that use the C++ code.

At the end of all this I ended up with some primitive routines that will classify pictures that contain the chessboard markers into directories.  It currently takes a list of pictures that have been marked with the chessboard markers and moves them to different directories.  The directories depend on the id that is calculated from the chessboard marker in the image.  This means that one can give the algorithm any number of pictures and it will sort them accordingly into separate directories.  Further sorting and accuracy changes can be done with camera data, but that is not implemented yet.

How does it work?

At the moment it requires a bit of work to see it work.  The src code is located here under the “ilac” branch.  To clone the repository from the ilac branch directly you must use the following command:

After you have cloned the repository execute “make” to build the python/C++ part. While you are in the imagelabel directory (the one that was created with the clone command), you can execute the python command line and import the ilac package.  Once you have managed to import the ilac package, you can use the ilac.get_image_id function to automatically sort pictures that contain chessboard markers.  If you don’t have pictures with markers you can go here for some examples.  That link contains 3 pictures of 2 different markers.  When get_image_id is used, the pictures should go to two different directories.

I tested the code in fedora 14.  If you try to build it in some other distro, you might have a problem.  You can always contact me and I will gladly assist you  :)

It’s reasonable to describe the chessboard format.  It’s relevant because not all chessboard dimensions are valid,  The current implementation only detects 6 colors and there needs to be a black margin around the chessboard (among other reasons).

Chessboard dimensions:

We need to accurately detect the position of every square from every direction.  This is not possible with a symmetric chessboard because one corner can be confused with its opposite.  This becomes relevant when extracting the binary information contained in the chessboard.  Instead of 0001, one could extract 1000.  To avoid this we need a chessboard that has one side with an even number of intersections and the other with an uneven number of intersections.  As you can see from the image (10intersections X 7intersections), the checkered characteristic generates a chessboard that has no equal sides.

Counting sides:

The image has a chessboard that is 11 squares by 8 squares.  But this is irrelevant to the algorithm.  It cares about the square intersections as opposed to the squares themselves.  The reasoning behind this is found in the cvFindChessboardCorners function in opencv.  It takes number of intersections instead of corners.  If you are curious go to [1].  This is not as bad as it seems.  The simple way of calculating the -cS and -cs [2] is to subtract one from each square count.  Finally notice that the image in this post contains 10×7 intersections.

Chessboard Colors:

At the moment we are using only 9 colors: red, yellow, green, cyan, blue, magenta, black and white.  White and black are dead colors in the sense that no information is contained in them.  They are there to delimit the other colors.  The contrast that the black squares make against the colored squares allows the algorithm to find the chessboard. The algorithm uses an 8 bit HSV format.  In this format the colors are represented in a cylinder [3]. I chose 6 points in the cylinder as shown in [4] and used those colors to represent bits.

Bit Representation:

Every square that contains a color other than black or white has data.  Each one contains three binary states represented by the three “main” colors (red, green and blue).  If the square is of one of the “main” colors, it will have one bit on and the rest off.  If the square is blue, it will have the blue bit on and the other two off.  On the other hand, if the square is of an “intermediate” (yellow, cyan or magenta), then two bits will be on and the remaining will be off.  If the square is yellow, it will have the red and green bits on and the blue off.  Theoretically if the square is white, all the bits are on.  Though the white square is non functional yet, I don’t see a reason why it shouldn’t work :)

Known squares:

Different lighting situations change the hue value in the HSV format.  This in turn increases error.  To avoid this we have 6 known positions where we put all the used colors [4].  With these known positions we can pre-set the hue ranges and increase the accuracy of the detection.  The known positions start at the top left corner of the image.  Remember that this position can be known even though the chessboard is rotated.

Black margin:

I have already mentioned the reasoning behind the margin in my previous post.  It basically increases the detection accuracy.

Before leaving for vacations I was fighting a very curious issue with my chessboard detection algorithm.  All my tests were done with pictures of chessboards that occupied most of the image.  The tests where very positive even for outside picture and under different lighting.  But when I tried to run the algorithm on a picture taken from a distance;  That is, that contained a “small” (because of perspective) chessboard; it failed to even detect the presence of the chessboard.

Thinking that the problem was in the findChessboardCorners function, I went to debug opencv.  One of the things I notice about the chessboard process is that it initially does a thresholding of the image.  I further noticed that this initial step depended on a histogram equalization.  Since this process depends on the overall values of the image pixels, I thought that the fact that my test images contained mainly snow (white pixels) was the culprit of the problem.  I went to gimp and started to darken the image.  To my surprise the darkening did not work as I expected.  It did increase the amount of sub-squares that were being found, but it did not detect the chessboard.

Finally, after some trial and error, I discovered that if I surround the chessboard with a thick black square (always careful to leave some white between the square and the chessboard), the algorithm would always detect the square (no matter how small it was)

This image is a part of a larger image.  I put it here to show the way that I encapsulated the chessboard image in a black square.  Notice that there is still a bit of white between the chessboard and the black square.

I am now going to create two chessboard images.  They will have different color codes and therefore different numbers. Both will have a black border to enhance detection.  The idea is to try to make an algorithm that will automatically group images depending on their chessboard code.  I will also try to add code that corrects various camera distortions.