3c72 Image Processing Practical Four - Colour Imaging

5%, to be set ~ 8/12/2004, to be completed by 12/1/2005

1. Objective

In the lectures, you have heard about different colour models or representations and about “colour constancy”. The aim of this practical is to illustrate some of these concepts and to exercise some of the techniques used in colour image processing and template matching.

2. Procedure

Carry out each of the steps listed below and write a brief report containing:

Include any code you write yourself (suitably commented) in an appendix.

3. Colour Normalisation Exercises

A set of example images, of a mustard coloured shirt imaged under various lighting conditions, all in ppm format, may be copied as required from the directory:

Select five examples from the set for the exercises in parts 3.1 to 3.5 below. Note that many of the images are rather dark, so it is best to view (but not process) the originals after histogram equalisation, for example using xv. Additional images required for part 4 are stored as

and may also be obtained from the demonstrator, Song Hu.

3.1 Image Data

Use xv, any other convenient tools, or write your own procedure in Matlab to crop or define a region of interest within each of the five mustard coloured shirt images chosen from the set of examples.

 

It is important to ensure, especially if a cropping tool is used to define the regions by hand, that the selected regions within each image set are exactly the same size, P x Q, say. It is thus best to use the crop tool and mouse in xv to define the regions of interest and then to select them using Matlab. If you find it difficult to select regions within each of the objects of interest, the white templates (NB. not the grey which is a region of uncertainty that may or may not be part of the object) in the images in ../Cloth1GT, define the requisite objects in each correspondingly named image. It does not matter if the selected regions correspond to different parts of the mustard coloured shirt, although they should be as representative of the object as possible.

Extract the RGB components of the selected image regions so they can be accessed as matrices for processing as described in sections 3.2-3.5 below. The notes prepared some time ago by Ioannis Douros on image preparation , in particular on how to set up image matrices for each colour channel may be of assistance if you have difficulty with this. It is probably best to perform all of the operations indicated on one of the images and selected image region first in order to be sure that your implementation works and to minimise the number of images and intermediate results you have to store.

3.2 Colour variation in RGB space

Compute the mean colour of the pixels in each chosen image region in RGB space, construct the scatter matrix and use Matlab libraries to compute the eigenvalues and eigenvectors of the scatter matrix. This calculation of the eigensolutions of the scatter matrix is known as principal components analysis or PCA and finds the principal components or axes of variation of the colour distribution of the pixels within each image region. Repeat the calculation of the mean and principal components using data from all the chosen regions within each of the shirt images and by taking the mean over all five regions you have selected from your five chosen images of the shirt.

View the resulting image regions and consider the variations in the results of the means and of the PCA within each set, comment on them in light of what you would expect from Schafer's dichromatic reflectivity model and indicate what the results tell you (if anything) about the lighting conditions under which each of the three sets of images was taken.

In each case, compare the standard deviation of the colour variation along each of the principal axes (defined as the square root of the corresponding eigenvalues of the scatter matrix) with the standard deviation of the colour variation in the red (R), green (G) and blue (B) channels, each defined as the square root of the corresponding diagonal components of the scatter matrix. Be as quantitative as you can.

3.3 Colour (row) normalisation and analysis of chromaticity

Compute the normalised colour representation rgb of each of the chosen image regions, calculate the mean rgb colours or chromaticity and repeat the PCA analysis in the rgb space. If the images are regarded as PQ x 3 rectangular matrices as in the work of Finlayson et al (1998), colour normalisation corresponds to row normalisation of the rectangular matrices.

One of the principal components should now be zero with its axis represented by the corresponding eigenvector oriented along the white direction. Explain why this should be so and check that your results are consistent with it. View the resulting image regions in the rgb representation, compare the results of the PCA obtained in the rgb space with those obtained previously and comment on what you find. Be as quantitative as possible in making this comparison.

3.4 Pixel (column) normalisation

Normalise each original RGB image region within each colour channel by summing over all the pixel values and, as in the work of Finlayson et al (1998), scaling the result by a factor of PQ/3. Calculate the mean colours and principal axes (principal components) of these normalised image regions as in 3.2 above, view the resulting image regions, and comment on the results obtained.

Similarly, normalise and scale the images in rgb representation as obtained in section 3.3 above. View the resulting images and comment on the results obtained both in comparison to those from the unmodified RGB images immediately above (this section, 3.4) and in comparison to those from the rgb (row) normalised images in 3.3 above.

3.5 Comprehensive normalisation

Carry out a comprehensive normalisation of the image regions as described in the work of Finlayson et al (1998) by iteratively repeating the row normalisation described in section 3.3 and the column normalisation in section 3.4 until there is little further change.

Comment on the convergence of this iterative procedure and view the comprehensively normalised image regions. Calculate the mean colour of the pixels in the comprehensively normalised image regions and carry out a similar PCA analysis of their distributions in colour space to those described above. Comment on the results obtained and on the potential significance of each of the three types of normalisation procedure.

4. Colour Template Matching Exercises

In this section, we use colour image processing as an initial step to simplify template matching. We shall use the set of cloth images of the mustard coloured shirt and some additional, “book” images as referenced above or to be supplied by the demonstrator, Song Hu. Since template matching can be computationally demanding, we shall use the colour image processing to restrict the search to parts of the target image likely to contain the image object region of interest. In particular, we will use the region of interest selected in your first image of the mustard coloured shirt to construct models of the mean colour of the shirt and its main components of variation in RGB space as in 3.2 above. Extensions to the column normalised rgb space of chromaticities, the pixel row normalised space, and after Finlayson’s comprehensive normalisation are possible, but regrettably beyond the scope of this course.

4.1       Constructing colour filters in RGB space

Two filters are to be constructed in the RGB space: (i) by setting thresholds on the variation in each of the RGB channels from their respective means relative to their variances, and (ii) by setting a threshold on the square of the Mahanalobis distance of each pixel’s colour from the mean colour. The Mahanalobis distance, d, of a pixel of colour (R,G,B), regarded as a column vector C is defined from the scatter matrix S as:

 

                            ,

 

where the mean colour within the region is denoted by . Note that the thresholds, when defined properly as described above, are all pure numbers.

 

Use the two filters constructed above to select pixels in the four remaining example images that are likely to belong to the mustard coloured shirt. Comment on the ease with which appropriate thresholds may be chosen and on the effectiveness of the two filters you have thus defined in selecting pixels belonging to the shirt.

4.2       Estimating the object location

Set the pixels in each of the four remaining example images that are selected by each of the filters in 4.2 above to one (“on”) and the remaining pixels to zero (“off “). Use the median values of the x and of the y co-ordinates of each “on” pixel to estimate the location of the centre of the region of interest in each image. Select a suitable (ie. not too large, but also not too small) search area around this estimated location and compute the best match of a template constructed from the region of interest selected in the first image with each example image (NB. Not just the regions selected in the later examples!) according to appropriate L1 and L2 metrics. Record the co-ordinates of the location returned by the best match and comment on the accuracy of the results. Discuss why median values of x and y are used to estimate the location of the centre of the region of interest from the output of the colour filters.

4.3       Matching the book images

Repeat the experiment described in 4.1 and 4.2 above for the book images using the bright red and orange “Pattern Classification” book template and the image of the four text-books supplied. Comment on the colour filters obtained, record the co-ordinates of the location returned by the best match and comment on the accuracy of the results.

5.       Reporting

As usual, beware of labouring long and hard to produce a pretty report. There is a rapidly diminishing rate of return for such efforts. Evidence that the exercise has been completed and content of the report is much more important than presentation. Adequate presentation and analysis of the images should however, be included in your report, as part of this evidence.

 

 

Bernard Buxton,
8 December 2004.