Finally some expected results!!!

Past Week(s):

Original picture

After consulting with some very talented computer vision professors, I decided to continue with Image segmentation based on pixel color.  The idea is to segment the image based only on the value of a pixel.  This means that there is no neighborhood information in the calculation.

To accomplish this, I had to re-annotate the pictures from zackenberg with polygons instead of rectangles.  The problem with the rectangles was that they included pixels that were not part of the flower.  Since I was going to do a classification between flower/non-flower pixels, I had to use something that could describe the shape of any flower as an annotation. The impoly command in Matlab seemed like a very good option.  This command lets you interactively create a polygon in an Axis.  Once the polygon is finished, it allows you to change its vertices and its position.

Experiment:

1. Color Space: I considered RGB, HSV and YCbCr.  I chose HSV for the initial experiments because it showed a good separation.  After HSV, I will continue with YCbCr and try to mimic what Son Lam Phung et. al. did in “A Universal and Robust Human Skin color model using Neural Networks“.  I need to reduce the amount of components of the color space (At the moment I am using all of them).  For HSV, the hue and value might be all I need.  For the YCbCr, the Cb and Cr components could be enough.
2. Annotation changes: I increased the number of subtypes for Salix Arctica.  The
   

   Masked Image

   reason for this is that the two different types of females have different color and the three different types of males also have different color.  I now have 8 subtypes.  These include a type that gathers all the low resolution annotations (You can tell it’s a Salix Arctica, but it’s too blurry type).

3. Classification: At the moment I have only tried binary classification.  In other words, I take pixels from a subtype of Salix Arctica and classify it against some None Flower pixels.  I do this for all the subtypes of Salix Arctica. I also want to classify the 8 subtypes (plus the none flower type) in one go and see if it gives better results.
4. Neural Network:  I’m using the newff command from Matlab.  I use a neural network with 10 hidden units and 10 epochs (I will vary these parameters to see if I can get a better result).  I also want to try SVM; given that I am doing a binary classification, an SVM is a logical choice.
5. Negative annotation:  I think this process would improve if I do some negative annotation.  The non-flower pixels are taken randomly from pictures that I know have no flowers.  What I plan to do is to annotate the regions that I know the algorithm is mis-classifying (like the sticks in the figures) and re-run the neural network training to strengthen the model.

Results:

Zoomed Masked image.

In the figures I show the results for Salix Arctica Female type.  The first image shows the original image.  The second image shows a black&white representation of the result of running the Neural Network on the original image.  First thing to notice is that white is non-flower and black is flower.  From the masked image we see that the artificial markers (the sticks stuck in the ground) show as flowers and that the photographers shoe also gets classified as flower.  We can avoid these mis-classifications with negative annotations in these regions.

Another thing to notice is that there is a tendency towards white in the image.  This means that if we use this as an initial filter, we could save ourselves a lot of computation when skipping the non-flower regions. Also notice that there are lots of black regions in the image that are not really flowers.  This means that we still need to do a bit more analyzing to get error rates that we can live with (but this is a great start!!!).

In the last image I zoomed in one region that contained a group of Female Salix Arctica.  Notice that this zoomed image is annotated and that what is contained inside is mostly black pixels.  This means that this method could give lots of false positives but very little false negatives.