GB2317487A - Hybrid digital/optical system for pattern recognition - Google Patents

Abstract

An initial Fourier transform on input data is performed by digital electronic means 2 using a specialised fast Fourier transform chip, the resulting spectral phase data being digitally added 4 to pre-stored template data 3 at high speed and loaded onto a phase modulating spatial light modulator 5. A subsequent optical Fourier transform 9 is used to generate a two dimensional correlation between the input scene and template. A non-linear thresholding spatial light modulator 10 is used in the correlation plane to detect a strong correlation peak at a speed commensurate with the template search rate. An area photodiode 11 is employed to convert the optical correlation peak to an electrical signal independent of its location in the output plane. This hybrid correlator arrangement exploits the asymmetry in the processing time requirements between the initial Fourier transform performed on the input image, which needs only to be performed once per input cycle, and that of the stored reference image. This may require several iterations to achieve recognition and so must be performed between 10 and 1000 times faster than the initial Fourier transform to maintain the required system response time.

Description

Hybrid digital/optical system for pattern recognition Introduction to the field of the invention There is a wide and ever expanding need for the rapid identification and fault detection of manufactured products ranging from components in the automotive industry to food and confectionery products. In this way, quality assurance and hence production efficiency can be enhanced. Currently devices to achieve this are commonly designed for a very specific task, often permitting the use of very simple measurement techniques employing, for example, single element photosensors. Present generation, digitally based visual inspection systems capable of a subsecond response time must also be designed for a specific task often requiring a specialised hardware implementation.

More flexible software solutions implemented on general purpose vector processor boards are restricted to the use of relatively simple pattern recognition algorithms such as area and perimeter measurements on constrained overhead views of binary images.

There is, however, a requirement for unconstrained orientation independent object recognition using grey-level imagery. Algorithms capable of this are too numerically intensive for implementation on even specialised hardware when a recognition cycle time of the order of 10 100 msec is demanded.

Thus there is a need for a visual recognition system that is sufficiently programmable to allow its flexible use in a wide range of pattern recognition tasks which also demand a high speed of response. The close integration of current state-of-the-art digital signal processing technology with advanced analogue optical processing techniques offers this possibility.

For many years now it has been appreciated that the inherent Fourier transforming property of a lens, together with the capability of performing a complex multiplication between a holographically recorded function and a coherent wavefront, permits the rapid two dimensional correlation between a reference template and an input scene containing an unknown image [1].

However, for the recognition of a 3-D object unconstrained in scale, rotation and orientation, many reference template searches may be required for recognition. The success of early attempts at the application of optical correlator systems to this problem was greatly hindered by the number of reference templates that could be stored on a thin holographic medium [2,3]. Multiplexing filter techniques have ameliorated this problem to some extent by reducing the sensitivity of a single filter to variations in object orientation [4-6]. However, to realise the full flexibility required, an effective interface must be implemented between the optical processor and a digital computer.

The optical processor is used to implement at very high speed the numerically intensive basic pattern recognition algorithms, the search strategy being under software control from the host processor [7].

The optical-to-digital interface requires the use of an electrically addressed spatial light modulator.

This may be used in the space domain, the reference templates being input to the optical system as images. The frequency domain filter is then realised by the non-linear interaction of the input and reference Fourier transforms in a photorefractive material [8,9]. Alternatively, the reference function Fourier transform may be input to the optical system using a phase modulating spatial light modulator [10].

Recently there has been considerable interest in the multi-level phase modulating capability of commercially available devices designed originally for use as miniature TV displays [13-16].

These devices are limited in their practical use in rapid correlator systems since they rely on a nematic liquid crystal for operation which has a response time of several tens of milliseconds.

However, over the last few years there has been a considerable effort in SLM development [17].

Of particular importance has been the fabrication of high resolution silicon active backplanes addressing a rapid response (100 microsec) chiral smectic C ferroelectric liquid crystal layer [1820], permitting a binary phase modulation. Much research has concentrated on filter realisations implementable with the limited quantisation possible with widely available SLMs [21-24] and shown that effective pattern recognition filters can be realised with these SLMs.

Description of the invention The invention described herein, shown in Figure 1, involves the hybridisation of a digital processor (1-4,12) to an analogue optical processing system (5-10) in order to realise a programmable correlator system. This arrangement exploits the asyrnmetry in the processing time requirements between the initial Fourier transform performed on the input image, which needs only to be performed once per input cycle, and that of the stored reference image. This may require several iterations to achieve recognition and so must be performed between 10 and 100 times faster than the initial Fourier transform to maintain the required system response time. For this reason it is proposed to implement this portion of the correlation operation using analogue optical processing techniques.

The input image data is captured by a video camera (1) which may, or may not, be of a standard CCIR format. The video data is digitised and Fourier transformed at video rates, or faster, by the dedicated digital signal processing (DSP) hardware (2). This means of implementing the initial Fourier transform has the advantage that signal degradations due to an input SLM are avoided and the transformation to frequency space can be effected with the high precision associated with digital techniques. It also allows the phase modulating SLM to be placed in a collimated beam (7) rather than at the focus of a highly convergent beam with consequent improvements in the accuracy with which the phase modulation can be accomplished.

Reference template data, stored in rapid access VRAM memory (3), is read-out at a multi-kilohertz rate. Since it is stored as Fourier transform conjugated phase data, each frame is added to the current input frame Fourier transform phase. Since this requires only a 1 to 5 bit addition, this can be accomplished at very high speed (5-10 kHz) with field programmable gate array (FPGA) based digital hardware (4). The resulting data is down-loaded to the electrically addressed high frame rate (5-10 kHz) phase modulating SLM (5). A coherent optical wavefront from a laser diode (6) addresses the SLM. This wavefront is initially plane in phase and may be of uniform amplitude distribution. Alternatively, it may have a Gaussian or difference of Gaussian amplitude profile (7).

The phase profile of the beam is modified upon reflection from the SLM so as to record the residual phase difference between the input and reference input data displayed on the SLM. In the case of a perfect match between the two, there is no phase difference and the beam is reflected from the SLM as a plane wave. If the input and reference are totally dissimilar, a random phase distribution is generated indicating that there is no match between the input and reference scene.

The complex multiply between the input and reference image Fourier transforms is thus realised with the phase modulating electrically addressed spatial light modulator (5) which provides an efficient coupling between the electronic and optical sub-systems. The optical sub-system can thus be made compact, requiring only a modest power laser diode as the coherent source, and also robust since the phase modulation technique is non-interferometric. The accuracy with which the optical Fourier transform is performed will be ensured by careful Fourier transform lens design (9) which, however, will incorporate only standard lenses available from optical suppliers in order to minimise final production costs.

The phase modulating SLM should ideally be capable of 4 bits of phase modulation allowing near 100% diffraction into the first diffraction order upon Fourier transformation by the lens system (9).

However, 1 bit of phase modulation can be used effectively in pattern recognition applications.

With this level of quantisation, a strong zero order term and higher orders of diffraction are produced. The output window, in the Fourier plane of lens system (9), must be confined to a half plane to ensure the desired first order correlation peak is the only structure detected. Alternatively, and in preference, a phase plate (8) designed according to the methods described in [33,34] may be used to disperse the zero and minus one diffraction order, allowing the output plane to be of the same resolution as the input and Fourier transform planes.

Read-out of the correlation response is accomplished with a non-linear thresholding SLM (10) to detect the correlation peak generated by the matched template. The SLM may incorporate a photosensor and non-linear thresholding circuitry on each pixel. This array may address a liquid crystal layer or deflect a micro-mirror array to realise a non-linear reflectivity. Alternatively, a non-pixellated photosensitive layer may be used to address a ferro-electric liquid crystal layer with an inherent non-linear reflection characteristic. As a futher alternative, a saturable absorber could be used to achieve a similar effect. In each case, a correlation peak that is above a pre-determined threshold will be reflected from the SLM onto an area photodiode. In this way, the correlation peak may be detected, whatever its location in the output plane, at a speed matching the high frame rate of the phase modulating SLM. The template generating the match is determined with a simple timing circuit that measures the time at which the electrical pulse was generated by the photodiode. This corresponds to the interval during which a given template was displayed on the phase modulating SLM. Once the peak is detected, the individual correlation plane can be readout in full with a rapid frame-rate CCD array (12). Thus, with this output detector scheme, the full processing rate of the optical processor may be exploited.

The close integration for the proposed hybrid system allows the rapid 2-D correlation of the input scene with stored reference templates. These are selected under control from software running on the host machine which enables an efficient search strategy to be implemented. This can be readily re-programmed for new pattern recognition tasks with appropriate image templates being downloaded from a mass storage device to the fast access VRAM memory.

The speed of response for orientation independent object recognition can be enhanced by reducing the total number of template searches required. This is to be accomplished with the use of filter multiplexing methods, particularly synthetic discriminant function filters. It could be arranged that these are synthesised automatically, allowing the system to be put into a "teach" mode, during which it is shown representative items of the component to be recognised from which it automatically generates the required multiplexed filters. In appropriate applications this may be achieved directly from a CAD model.

Having described the invention the claims are: (1) The division of a correlator system into a first Fourier transform section which is performed digitally and a second Fourier transform which is performed optically.

(2) A digital/optical hybrid correlator system as described in claim (1) in which the template data is stored as frequency domain phase data of whatever precision and subtracted from input scene Fourier transform phase data so as to form a phase only filter.

(3) The use of laser beam profiling in the hybrid digital/optical correlator system described in claim (1) in such a way as to modify the filter response when the phase modulating spatial light modulator is read-out with this beam.

(4) The use of a non-linear thresholding spatial light modulator (SLM) in an all-optical or hybrid digital/optical correlator system to detect a correlation peak at high rates independent of its location within the 2-D field of view of the correlator. In this way, the output data bottleneck is prevented that would otherwise occur if each entire output data frame had to be detected with a camera. The spatial light modulator may be of a pixellated type in which each individual pixel incorporates a photodiode, non-linear thresholding circuitry and a liquid crystal layer or micromirror array. The reflectivity from this device should be such that above a certain intensity the input optical beam is reflected, but below this the reflectivity is as small as the device implementation permits. Alternatively, the SLM could be optically addressed by means of a photoconductor which in turn activates a ferroelectric liquid crystal layer which has a non-linear optical transmittance or reflectance characteristic to an optical read-out beam (separate from the addressing beam) transmitted by, or reflected from, the device. In each arrangement an area photodiode would detect the correlation peak independent of its location within the 2-D area of the correlation plane and provide an electrical signal to indicate which template had responded by matching its time of occurrence to the template search sequence.

(5) The use of a saturable absorber, rather than a spatial light modulator, to achieve the nonlinear thresholding effect described in claim (4). An area photodiode would be used in the same way as in claim (4) to detect the correlation peak generated by a strongly matched template, independent of its location in the correlation plane.

Claims (1)

1. References

   1. A. Vander Lugt, "Signal detection by complex spatial filtering". IEEE Trans., IT-10, pp 139-145 (1964).

   2. B. Guenther, C. Christensen, J. Upatnieks, "Coherent optical processing: another approach, IEEE, J. Quantum Elect., Vol QE-15. No 12. pp 1349-1362, (1979).

   3. J. Duthie, J. Upatnieks, "Compact real-time coherent optical correlators", Opt Eng., Vol 23, No l,pp 7-12, (1984).

   4. D. Casasent, "Unified synthetic discriminant function computational formulation". Appl.

   Opt., Vol 23, No 10, pp 1620-1627, (1984).

   G. Ravichandran, D. Casasent, 'WMinimum noise and correlation energy optical correlation filter", Appl. Opt., Vol.31, No.11, pp 1823-1833, (1992).

   6 E. Marom, H. Inbar, "New Interpretations of Wiener filters for image recognition", Vol.13, No.7, pp 13251330, (1996).

   7. D. Casasent, "Multiple degree of freedom object recognition using optical relational graph decision nets", Appl. Opt., Vol 27, No 9, pp 1886-1893, (1988).

   8. L. Richon, J.-P. Huignard, "Dynamic joint-Fourier-transform correlator by Bragg diffraction in photorefractive Bil2SiO20 crystals", Opt Comm., Vol 36, No 4, pp 277-2809 (1981).

   9. H. Rajbenbach, S. Barn, Ph. Refregier, P. Joffie, J.-P. Huignard, M.St. Buchkrammer, A. Jensen, E. Rasmussen, K. Brenner, G. Lohman "An optical photorefractive correlator for robotic applications", Report on ESPRIT Project No 2288: New architectures for optical processing in industrial applications (NAOPIA).

   10. D. Psaltis, E. Pack, S. Verikatesh, "Optical image correlation with binary phase modulator", Opt. Eng., pp 698-704, (1984).

   11. M. Flavin, L Horner, "Correlation experiments with a binary phase-only filters", Opt. Eng., Vol 28, pp 470-473, (1983).

   12. J. Davis, L Waas, "Current status ofthe magneto- optic spatial light modulator", SPIE, Vol 1150, (1989).

   13. T. Bames, K. Matsumoto, T. Eiju, K. Matsuda, N. Ooyama, "Phase only liquid crystal light modulator and its application in the Fourier plane of optical correlation systems", SPIE, Vol 1134, pp 204-208, (1989).

   14. K. Lu, B. Saleh, "Theory and design of the liquid crystal TV as an optical phase modulator".

   Opt. Erg., Vol 29. No 3. pp 240-246, (1990).

   15. K. Lu, B. Saleh "Complex amplitude reflectance of the liquid crystal light valve", Appl.

   Opt., Vol 30, No 17, pp 2354-2362 (1991).

   16. J. Amako, T. Sonehara, "Kinoform using an electrically controlled birefringent liquid-crystal spatial light modulator", Appl. Opt. Vol 30, No 32, -pp 4622-4628, (1991).

   17. J. Neff, R. Athale, S. Lee, "Two-dimensional spatial light modulators: a tutorial". Proc.

   IEEE, Vol 78, No 5, pp 826-855, (1990).

   18. N. Collings, W. Crossland, P. Ayliffe, D. Vass, 1. Underwood, "Evolutionary development of advanced liquid crystal spatial light modulators", Appl. Opt., Vol 28. No 52, pp 4740 4747, (1989).

   19. K. Johnson, D. Javed, T. Slagle, K. Wagner, C. Mao, M. Robinson, "Ferroelectric liquid crystal spatial light modulators and their applications", Spatial light modulators and applications, Incline Village, Nevada, Optical Society of America Technical Digest, TuB 1 (1990).

   20. I. Underwood, D. Vases R. Sillitto, G. Bradford, N. Fancey, A. Al-Chalabi. "A high performance spatial light modulator", SPIE, Vol 1562. pp 107-115, (1991).

   21. D. Flannery, "Optimal trade-off distortion-tolerant constrained-modulation correlation filters", JOSA Pt. A, Vol. 12, No.1, pp 66-72, (1995).

   22. B. Kast, M. Giles, S. Lindell, D. Flannery, "Implementation of ternary phase amplitude filters using a magnetooptic spatial light modulator", Appl. Opt., Vol 28, No 6, pp 1044 1046, (1989).

   23. F. Dickey, B. Hansche, "Quad-phase correlation filters for pattern recognition", Appl. Opt., Vol 28, No 9, pp 1611-1613, (1989).

   24. D. Flannery, J. Loomis, M. Milkovich, "Transform-ratio ternary phase-amplitude filter formulation for improved correlation discrimination", Appl. Opt., Vol 27. No 19, pp 4079 40839 (1988).

   25. D. Flamery, J. Horner, "Fourier optical signal processors". Proc. IEEE, Vol 77, No 10. pp 1511-1526.(1989).

   26. B. Vijaya Kumar, "Minimum-variance synthetic discriminant functions". JOSA Pt. A, Vol 3, No 10, pp 1579-1584, (1986).

   27. A. Mahalanobis, B. Vijaya Kumar, D. Casasent, "Minimum average correlation energy filters", Appl. Opt., Vol 26, No 17, pp 3633-3640, (1987).

   28. Ph. Refregier, "Filter design for optical pattern recognition: multicriteria optimisation approach", Opt. Lett., Vol 15, No 15, pp 854-856, (1990).

   29. B. Javidi, D. Painchaud, "Distortion-invariant pattern recognition with Fourier-plane nonlinear filters", Appl. Opt. Vol. 35, No. 2, pp 318-331, (1996).

   30. B. Javidi, J. Wang, "Optimum distortion invariant filter for detecting a noisy distorted target in non-overlapping background noise", JOSA Pt. A, Vol. 12, No. 12, pp 2604-2614, (1995).

   31. A. Mahalanobis, B. Vijaya Kumar, S. Sims, "Distance-classifier correlation filters for multiclass target recognition", Appl. Opt. Vol. 35, No. 17, pp 3127-3133, (1996).

   32. M. Fam, "The making of binary optics". Optics and Photonics News, May (1991).

   33. M. Neil, E. Paige, "Breaking of inversion symmetry in 2-level, binary Fourier holograms", Proc. Holographic Systems, Devices and Applications, Neuchatel, (1993).

   34. T. Wilkinson, D. O'Brien, R. Mears, "Dynamic asymmetric binary holograms using a ferroelectric liquid crystal SLM", Opt. Comm., Vol. 109, pp 222-226, (1994).

   35. H. Damman, K. Gortler, "High efficiency in-line multiple imagery by means of multiple phase holograms", Opt. Comm., Vol 3, p 312, (1971).

   36. J. Turunen, A. Vasara, J. Westerholm, "Kinoform phase relief synthesis: a stochastic method", Opt. Eng., Vol 28, No 11, pp 1162-1167, (1989).