WO1996018975A1

WO1996018975A1 - Data recognition system

Info

Publication number: WO1996018975A1
Application number: PCT/AU1995/000813
Authority: WO
Inventors: Charles Tasman Westcott; Leonard George Chadborn Hamey
Original assignee: Arnott's Biscuits Limited
Priority date: 1994-12-13
Filing date: 1995-12-01
Publication date: 1996-06-20
Also published as: JPH10511786A; AUPN002394A0; CN1170469A; DE19581867T1; CA2207326A1; GB9710197D0; NZ296487A; GB2311369A

Abstract

The present invention is directed towards a method of determining the state of a substance (2) having a color property dependent on the state of the substance. The method includes the following steps: forming a pixel image of the substance (3) using a camera (3); projecting the pixel image into a three-dimensional color space using a neural network (7); and comparing the projection (37) with a projection of a second position of the substance (2) to determine the state of the substance (2) using a second neural network (9).

Description

DATA RECOGNITION SYSTEM Held Qf tjhe Invention

The present invention relates to recognition of data characteristics within an image and more particularly to the recognition of characteristics within a baking process.

Background of the Invention

There are several factors which influence the results produced by a baking process. For example, in the production of bread or biscuits or other such foodstuffs, a number of factors including the temperature of the oven, the baking period, and the size, shape, thickness and moisture level of the substances (e.g. dough) utilised influence the final product produced at the end of the baking process. Further, these baked foodstuffs are often produced on a commercial scale, in large volumes, using a mass production process.

At the end of the production process, the goods are normally inspected to ensure quality control. Inspection often takes the form of looking at a particular batch of goods coming off a production line in order to ensure their suitability.

Often the inspection process is undertaken by a human observer. It has been found, in practice, that human observers provide subjective judgements that are prone to both long- and short-term variations. Additionally, human observers are only capable of working at certain fixed speeds and are prone to adverse conditions such as boredom, tiredness and/or sickness.

Summary of the Invention

It is an object of the present invention to provide a method of determining the state of an object having visual data properties that are subject to variation.

In accordance with a first aspect of the present invention, there is provided a method of determining the state of a substance having a color property dependent on said state, said method comprising the steps of: forming a pixel image of said substance; projecting said pixel image into a three dimensional color space; and comparing said projection with a projection of a second portion of said substance, having a predetermined state, so as to determine the state of said substance. In accordance with a second aspect of the present invention there is disclosed an apparatus which is able to operate in accordance with the above method. In accordance with a third aspect of the present invention, there is provided a method of determining the state of a first data sample having collectively a series of characteristics, said method comprising the steps of: projecting said first data sample into a multi-dimensional space; and comparing said projection with a projection of a second data sample having a set of predetermined characteristics, to determine the state of said first data sample characteristics. Brief Description of the Drawings

The preferred embodiment of the present invention is described with reference to the accompanying drawings, in which: Fig. 1 illustrates the steps of the preferred embodiment;

Fig. 2 illustrates the input scanned data format utilised by the preferred embodiment;

Figs. 3-8 illustrate the progression of the color change with increased baking levels; Fig. 9 illustrates a Kohonen self-organising feature map;

Fig. 10 illustrates a first baking curve; and

Fig. 11 illustrates a second baking curve for a different biscuit product. Detailed Description

Although the preferred embodiment will be described with reference to baking, it will be apparent to a person skilled in the art that the invention is not limited thereto and applies to all forms of visualisation where the state of a substance can be determined from its appearance. In the preferred embodiment of the present invention, a detailed analysis of the color characteristics of the baked product is undertaken by a computer system integrated into the production process.

A mass production system 1 for producing baked foodstuffs according the preferred embodiment of the invention is illustrated schematically in Fig. 1. The system 1 comprises a food processing apparatus (e.g. oven) 4, substance characterisation apparatus 3, 6, 7, 8, 9, and control system 11. Baked products 2 are imaged by a camera 3 as the products exit the oven 4. The image taken by camera 3 is subjected to a number of processing steps preferably implemented using a microcomputer, which will be described in more detail hereinafter. The processing includes an image processing or calibration step 6, a first neural network processing step 7, an image processing step 8, and a second neural network step 9 to produce an output indication 10 that indicates the extent of the baking of the baked products 2. The output 10 provides an indication of whether the products 2 are under- or over- baked and can be utilised as an input to the control system 11 to determine if the oven parameters such as heat or time need to be adjusted.

The camera 3 utilised in the preferred embodiment is a three-chip charge coupled device (CCD) camera which produces a 512 X 512 array of red (R), green (G), and blue (B), or RGB, pixel values. This array of pixel values is stored in a "frame grabber" board mounted within the microcomputer. The baking products 2 were imaged using two daylight-color-balanced THORN (Registered Trade Mark) fluorescent lamps for direct illumination and a black curtain to exclude ambient illumination. It should be apparent to a person skilled in the art that other forms of illumination may be used, and/or the background curtain may be eliminated in an industrial production process, although some other form of ambient light shielding may by utilised without departing from the scope and spirit of the invention. In a first sample, the baked products imaged were "SAO" (Registered Trade Mark) biscuits produced in a production line of the Arnott's Biscuits Limited biscuit factory. In order to ensure proper color calibration of the imaged biscuits, the biscuits were imaged with a color calibration chart 12 of the form shown in Fig. 2. The scanned image 12' includes biscuit area 13, background 14, reference white background 15, grey scale 16, and reference colors 17. The scanned image 12' and image 12 were men utilised to color calibrate the pixels of the biscuit 13. The process of color calibrations to ensure color consistency of sampled images is a well known process to those skilled in the art of computer imaging. For a discussion of the color calibration process, reference is made to Novak, C.L. and S.A. Shafer 1990, "Supervised Color Constancy Using a Color Chart," Technical Report CMU-CS-140, Carnegie Mellon University School of Computer Science. The paper by Novak et al. sets out more extensive means of color calibration. However, it should be noted that under appropriate circumstances of illumination and instrumentation control, calibration can be dispensed with.

On completion of the color calibration process (block 6 of Fig. 1), each biscuit image can be stored as an RGB file. The RGB color space is a well known color space, in common use with computer systems. However, it will be apparent to a person skilled in the art that other color systems such as HSN, XYZ or CIE color space coordinates may also be utilised.

Referring now to Figs. 3 to 8, a projection of the pixel values obtained for biscuit 13 (Fig. 2) is shown for different stages of the baking process.

Fig. 3 shows a projection 20 of the pixel image of a "raw" biscuit onto an RGB cube 21.

Figs. 4 to 8 show corresponding projections for biscuits varying from underbaked biscuits (Fig. 4) to overbaked biscuits (Fig. 8). It can be seen from comparing Figs. 4 to 8 that the color characteristics of a biscuit change as that biscuit is subjected to increased levels of baking. The particular progression of pixel data of Figs. 4 to 8 can therefore be utilised to determine the baking state of any particular sample biscuit. In order to utilise the progression from Fig. 4 to Fig. 8 in the baking process, it is necessary to succinctly and compactly describe an approximation to the data samples of Figs. 4 to 8. This can be done through the production of a "baking curve" which is a one-dimensional representation of the important color variations within the three- dimensional data space of Figs. 4 to 8. One method of production of a baking curve is to utilise a Kohonen self-organising feature map which is an unsupervised learning technique that is effective in extracting structure from complex experimental data which bears a highly non-linear relationship. A detailed description of Kohonen 's self- organising feature map is provided in Neural Network Architectures: An Introduction by Judith E. Dayhoff, published 1990 by Nan Nostrand Reinhold at pages 163-191 , the contents of which are hereby incorporated by cross-reference.

Turning now to Fig. 9, there is shown an example of a self-organising map (SOM) as utilised in the preferred embodiment. The SOM 30 has three input nodes 31 which correspond to the red, green and blue color components of pixel values from the digitised color image of the biscuit 13 (Fig. 2). The SOM 30 includes N output nodes 32. Every input node 31 is connected by means of edges e.g. 33, to each of the output nodes 1 to N. The use of a one-dimensional SOM 30 means that, upon training, the SOM network 30 will map the entire set of RGB pixel values 31 from a biscuit image to a one-dimensional array of points or output nodes 32. This will have two main effects: firstly, it will reduce the input data function into a data space of lower dimensionality; and secondly, the interrelationships between the most relevant points in the input data 31 will be retained intact in the output format of the network 30. If the input 31 is denoted E as follows:

E = (Red, Green, Blue), Eqn 1 and if output node i is connected to the input 31 by a series of edges having weights denoted Uj, where Uj takes the following form:

Ui = (Wi,_red , W_{i green} , W_i)blue), Eqn 2 where it is assumed that the weights are initially randomly assigned to edges and range over the same set of data values as the pixel component values (in the preferred embodiment from 0 to 255), then the SOM is trained by finding, for each pixel, the output node 32 having weights Uj which are closest to me input pixel E. The degree of "closeness" is normally measured by means of a Euclidean distance measure. The closest output node, having subscript c, can be denoted mathematically as follows:

The best or winning node c is then altered in conjunction with nodes within the neighbourhood of c (for example, node c - 1 and c + 1). The alteration, for each output node j in the neighbourhood of c (denoted N_c) proceeds by first calculating ΔU; as follows:

ΔUj = α (t) x (E - Uj), Eqn 4 and then deriving a new set of weights Uj^(t+ -⁾ as shown in equation 5: t + 1 t

U - U + A U,-. Eqn 5

J J

Those output nodes that are not in a predetermined neighbourhood around the chosen node c are left unaltered. The function of α(t) in equation 4 is known as the "learning rate" and is a monotonically decreasing function, with an alpha function of the following form being suitable: α (t) = α₀ (1 - VT), Eqn 6 where α₀ takes on values in the range of 0.02 to 0.05, t is the current training iteration, and T is the total number of training iterations to be done. The width d of the neighbourhood N_c can also be chosen to vary in a similar manner as set out in equation 7: d (t) = d,, (1 - VT), Eqn 7 where d₀ can be chosen to initially be, say, a third of the width of the output nodes. As an example of a process carried out in accordance with the preferred embodiment, four biscuit samples of "SAO" from Arnott's Biscuits Limited were scanned to yield 47,967 pixels per biscuit with three separate R, G, and B values for each pixel. The four sets of pixels were then shuffled in a random sequence and used as training input to an SOM of the form of Fig. 9 having ten output nodes for a total of 20 training passes.

Referring now to Fig. 10, there is shown a plot of the ten output node weight values U- within a three-dimensional color cube 35. The ten points (e.g., 36) are shown joined together by curve 37. The curve 37 is hereby denoted to be the final "baking curve" of the input data.

Turning now to Fig. 11 , the process was repeated for a second form of biscuit, comprising Arnott's "MILK COFFEE" (Registered Trade Mark) biscuit for a 15 node output SOM and the results 41 are shown plotted within the color cube 40, with the training data being passed dirough the SOM a total of 50 times. It can be seen that the structure of the baking curve 41 of Fig. 11 is similar to that of the baking curve 37 of Fig. 10, with the two curves 37,41 occupying a different portion of the color cube 35,40 and reflecting the differences in ingredients between the products. The color curve 41 of Fig. 11 is also shorter than that of Fig. 10 as the "MILK COFFEE" form of biscuit exhibits more consistent browning than the "SAO" form of biscuits, which have blisters that can cause uneven browning in color.

Once trained, the SOM 30 of Fig. 9 can be utilised as neural network 7 of Fig. 1 to produce, for each input pixel, an output node indicator having the closest position to the input pixel. The closest matching output node 32 (Fig. 9) for each pixel of an image can be subjected to image processing 8 (Fig. 1) which can take the form of histogramming, thereby producing a histogram profile of the scanned baking product 2.

A second neural network 9, which takes the form of a supervised feed forward neural network, can then be subjected to "training" by imaging a large number of biscuits 2 having known baking characteristics, feeding the images through SOM 7, and forming a histogram 8. The histogram 8 can then form the input data to a supervised back propagation neural network which can be trained, in the normal manner, to classify the color level of the baking product (e.g. the biscuit) 2. The samples can be continuously fed through the steps 6 to 9 until the neural network 9 is properly trained to produce output 10 indicating the level of baking. The output 10 can then be utilised by control system 11 which can take the form of human or automatic process adjustment to adjust the conditions within oven 4 to improve the baking products 2.

The specified system provides automatic segmentation of the biscuit subject from diverse backgrounds. This is accomplished by the histogramming process which "weighs" each image pixel as a reducing function of its distance from the histogram points. Thus, pixels that are significantly distant from the baking curve discovered by the self-organising map are down-weighted to the extent that they make little or no contribution to the overall histogram. In practice, this means that it is not necessary for biscuits to be imaged with a specially prepared background. Any background with colors sufficiently dissimilar to the biscuit colors under consideration will suffice. This is of practical benefit when applying the system to on-line monitoring and oven control, as the imaging background may well be a conveyor belt of inconsistent color.

The computation of the histogram 8 from the map produced by the SOM is performed in detail as follows. In principle, the purpose is to obtain histograms in which each bin represents the weighted count of pixels falling within a fuzzy portion of the baking curve 35,40. The fuzzy portion is defined by a Gaussian weighting function with parameters σ_x and σ_v. The parameter σ_x denotes the spread of the Gaussian weighting function about the baking curve, and it is determined by consideration of the likely color variation around the curve. It is chosen sufficiently small enough to enable the automatic segmentation process previously described. The parameter σ_v denotes the spread of the Gaussian weighting function along the baking curve for a particular histogram bin. Normally, σ_v is greater than σ_x.

A practical implementation of the above technique involves the following steps: a. The SOM nodes are interpolated to obtain a large number of sampling points. The number of sampling points is determined by σ_x so as to limit to an acceptable level the aliasing effect caused by sampling the baking curve at discrete points. The Nyquist result in sampling theory applies in this step. b. The biscuit pixels are histogrammed at the interpolated sampling points. The distance of each pixel (in RGB color space) from each sampling point is computed, and the Gaussian function with spread σ_x is used to compute the weighted contribution of that pixel to the histogram bin at that particular sampling point. c. The histogram produced in (b) is treated as a 1-D signal and filtered with a second Gaussian function with a spread Jσ -σ . In the process, it is subsampled to

the number of input nodes of the feed forward neural network. The value of σ_y is chosen so as to limit to an acceptable level the aliasing caused by this further subsampling. The system may be applied to multi-dimensional data of diverse kinds such as combinations of color, visual texture and/or three-dimensional structure sensing. In some such situations, the set of state points may require a 2-D or 3-D SOM whereas the baking curve requires only a 1-D SOM.

It will be obvious to those skilled in the art that the steps 6 to 9 can be implemented in many different ways, including dedicated neural network hardware and associated computer hardware or in the form of a software simulation of the neural network system. The preferred method of implementation of steps 6 to 9 is in the form of software implementation on a standard microcomputer system as this allows for easy alteration when it is desired to alter the form of baking products. The foregoing describes only one embodiment of the present invention, and modifications obvious to those skilled in the art can be made thereto without departing from the scope of the invention.

Claims

CLAIMS :

1. A method of determining the state of a substance having a color property dependent on said state, said method comprising the steps of: forming a pixel image of said substance; projecting said pixel image into a three dimensional color space; and comparing said projection with a projection of a second portion of said substance, having a predetermined state, so as to determine the state of said substance.

2. The method as claimed in claim 1 , wherein said comparing step further comprises the steps of: deriving a series of state data points from said second portion of said substance; and producing a comparison of said projection with said state data points so as to determine the state of said substance.

3. The method as claimed in claim 2, wherein said deriving step further comprises: deriving said series of state data points by means of a self organising feature map.

4. A method as claimed in claim 2, wherein said producing step further comprises: producing a data histogram from said projected pixel image and said state data points, said histogram indicating the number of projected pixels in the neighbourhood of each of said state data point.

5. The method as claimed in claim 4, wherein said producing step further comprises: deriving a neural network input to a neural network said data histogram being, said neural network using trained to recognise neural network inputs derived from histograms of projected pixel images of substances having a predetermined state and to generate a signal indicative of said predetermined state, and outputting a signal from said neural network indicative of said state of the projected pixel image of said data histogram.

6. A method of determining the state of a first data sample having collectively, a series of characteristics, said method comprising the steps of: projecting said first data sample into a multi-dimensional space; and comparing said projection with a projection of a second data sample having a set of predetermined characteristics, so as to determine the state of said first data sample characteristics.

7. The method as claimed in claim 6, wherein said comparing step further comprises: deriving a series of state data points from said second data sample; and producing a comparison of said projection with said state data points so as to determine the state of said first data sample characteristics.

8. The claim as claimed in claim 7, wherein said deriving step further comprises: deriving said series of state data points by means of a self organising feature map.

9. The claim as claimed in claim 8, wherein said producing step further comprises: producing a data histogram from said first data sample and said state data points, said histogram indicating the number of first data samples in the neighbourhood of each of said state data point.

5 10. The method as claimed in claim 9, wherein said producing step further comprises: deriving a neural network input to a neural network using said data histogram, said neural network being trained to recognise neural network inputs derived from histograms of data samples having a predetermined series of characteristics and to o generate a signal indicative of said predetermined state, and outputting a signal from said neural network indicative of said state of the projected data samples of said data histogram.

11. A method of baking foodstuffs using said determining method in 5 according to any one of claims 1 to 5, said baking method further comprising the step of: controlling an oven used to bake said foodstuffs dependent on the state of said substance.

o 12. A method of baking foodstuffs using said determining method according to any one of claims 6 to 10, said baking method further comprising the step of: controlling an oven used to bake said foodstuffs dependent on the state of said first date sample characteristics. 5

13. A system for determining the state of a substance, comprising: an imaging device for forming said pixel image of said substance; and processing means coupled said imaging device for carrying out the method in accordance with any one of claims 1 to 10. 0

14. An assembly for baking foodstuffs, comprising: an oven; an imaging device for imaging said baked foodstuffs output by said oven; processing means coupled to said imaging device for carrying out the 5 determining method according to any one of claims 1 to 10; and control means coupled to said processing means for receiving an output signal of said processing means indicative of the state of said baked foodstuffs; wherein said control means is coupled to said oven to operate said oven dependent on said state of said baked foodstuffs.

AMENDED CLAIMS

[received by the International Bureau on 15 Apri l 1996 ( 15.04.96) ; original claims 1 ,4-6 and 12 amended ; remaining claims unchanged (3 pages) ]

1. A method of determining the state of a substance having a color property dependent on said state, said method comprising the steps of: forming a pixel image of said substance, wherein said pixel image comprises a plurality of pixels; projecting said pixel image into a three dimensional color space; and comparing said projection with a projection of a second portion of said substance, having a predetermined state, so as to determine the state of said substance.

4. A method as claimed in claim 2, wherein said producing step further comprises: producing a data histogram from said projected pixel image and said state data points, said histogram indicating the number of projected pixels in the neighbourhood of each of said state data points.

5. The method as claimed in claim 4, wherein said producing step further comprises: deriving a neural network input to a neural network using said data histogram, said neural network being trained to recognise neural network inputs derived from histograms of projected pixel images of substances having a predetermined state and to generate a signal indicative of said predetermined state, and outputting a signal from said neural network indicative of said state of the projected pixel image of said data histogram.

_*

6. A method of determining the state of a first data sample having collectively, a series of characteristics, said method comprising the steps of: projecting said first data sample comprising a plurality of data points into a multi-dimensional space; and comparing said projection with a projection of a second data sample having a set of predetermined characteristics, so as to determine the state of said first data sample characteristics.

9. The claim as claimed in claim 8, wherein said producing step further comprises: producing a data histogram from said first data sample and said state data points, said histogram indicating the number of first samples in the neighbourhood of each of said state data point.

10. The method as claimed in claim 9, wherein said producing step further comprises: deriving a neural network input to a neural network using said data histogram, said neural network being trained to recognise neural network inputs derived from histograms of a data samples having a predetermined series of characteristics and to generate a signal indicative of said predetermined state, and outputting a signal from said neural network indicative of said state of the projected data samples of said data histogram.

11. A method of baking foodstuffs using said determining method in according to any one of claims 1 to 5, said baking method further comprising the step of: controlling an oven used to bake said foodstuffs dependent on the state of said substance.

12. A method of baking foodstuffs using said determining method according to any one of claims 6 to 10, said baking method further comprising the step of: controlling an oven used to bake said foodstuffs dependent on the state of said first data sample characteristics.

13. A system for determining the state of a substance, comprising: an imaging device for forming said pixel image of said substance; and processing means coupled said imaging device for carrying out the method in accordance with any one of claims 1 to 10.

14. An assembly for baking foodstuffs, comprising: an oven; an imaging device for imaging said baked foodstuffs output by said oven; processing means coupled to said imaging device for carrying out the determining method according to any one of claims 1 to 10; and control means coupled to said processing means for receiving an output signal of said processing means indicative of the state of said baked foodstuffs;