EP1417453A1

EP1417453A1 - Device and method for 3d imaging

Info

Publication number: EP1417453A1
Application number: EP02740754A
Authority: EP
Inventors: M.G. Unilever Research Port Sunlight REED; John E. Unilever Research Port Sunlight WILSON
Original assignee: Unilever PLC; Unilever NV
Current assignee: Unilever PLC; Unilever NV
Priority date: 2001-08-01
Filing date: 2002-06-26
Publication date: 2004-05-12
Also published as: WO2003014665A1

Abstract

A method and device are described for 3D imaging of objects whereby an object is illuminated by a light source and a periodic pattern is generated on the object. The same periodic pattern is generated on the same plane of the object but at an altered position. The images are recorded and analysed to remove the spatial pattern from the images. This is carried out for different views of the object to obtain the full 3D image of the object.

Description

Device and method for 3D imaging

Field of the Invention

The present invention is in the field of 3D scanning and 3D image reconstruction using structured light.

Background of the Invention

Structured light methods have been used to perform depth measurements in microscope images and to obtain 3D mask images of the external surfaces of microscopic and macroscopic objects. These methods are broadly based on trigonometric considerations. Methods based on trigonometry, more specifically triangulation, involve the evaluation of patterned images which are obtained by projecting substantially periodic structures by means of a grating onto an object. By setting the viewing position at an angle to projection, the shape of the object introduces a distortion in the patterned image. The shape of the object is calculated from the extent of 'the distortion of the patterned image. Triangulation methods work best when the patterned image is in focus over the whole of the object.

US-A-4, 657, 394 discloses a technique for determining the three dimensional surface profile of an object. An incident light beam is directed at the object, the beam having a sinusoidally varying intensity pattern. The phase of the sinusoidal intensity pattern is modulated, such as by moving the grating used to obtain said pattern. A deformed grating image of a line profile is detected, preferably at a linear detector; a detector having individual detector elements arranged in a line. For points on the line profile of the object, the distance of each such point from a reference line is then determined, each such distance determination including the step of combining intensity values of the received image at the different modulated phases to obtain an object phase for each point . In this manner the coordinate locations of the points on a line profile of the object can be determined.

Further documents disclosing image composition by triangulation are DE-A-19515949, JP-A-07139922 , DE-A-4416108 , US-A-5 , 085, 502.

The problem with this imaging method for 3D imaging is that it is expensive, takes a long time (at least 1 hour) and is difficult to operate.

There is a need to provide devices that are capable to not only reconstruct the 3D mask of a particular plane of an object but the three dimensional peripheral shape of the object.

There is also a need to provide devices for peripheral view images that are cheap to build and easy and quick to operate and furthermore easy and convenient to use. There is a need for a device that is capable to reconstruct the 3D peripheral view of an object with one single operational step, whereby an operational step is to be understood as placing the object in the scanning device, activate the device and obtain the 3D information of the object without any further intermediate steps to be carried out by the user.

The current invention provides an inexpensive, convenient and safe device and imaging method for reconstructing the 3D peripheral shape of an object using SMI. It has surprisingly been found that a device comprising a component suitable to create mask' type images by generating and recording at least two "periodic patterns" at altered spatial position and a component enabling the creation of masks of at least two different views of an object, enables the construction of a 3D image of an object in a simple and straightforward way. A device according to the invention is cheap to build and easy, quick and convenient to operate whereby the 3D peripheral view of an object can be reconstructed in a single operational step.

Summary of the invention

In one aspect the invention relates to a device for 3D imaging of the peripheral view of an object using structured modulation imaging said device comprising:

-one or more projecting system (SI) generating a periodic pattern on the object said system comprising one or more light source (L) , one or more focussing means (FI) , one or more patterning means (Ml) ,

- one or more image recording system (S2) comprising a focussing means (F2) and one or more means (M2) of recording an image of the periodic pattern generated on the object, wherein SI, S2 and the object are arranged relative to each other such that the periodic patterns can be generated and recorded for at least two different views of the object, said device further comprising means (M3) allowing Ml and the object to be moved relative to each other. In another aspect the invention relates to a method for 3D imaging of the peripheral view of an object using structured modulation imaging comprising a) illuminating the object with a light source, b) projecting a first substantially periodic pattern on the object c) recording an image of the periodic pattern on the object, d) generating at least one more time the periodic pattern at an altered spatial phase of the first pattern, wherein steps a-d are carried out for at least two different views of the object and wherein the data obtained are analysed to reconstruct the 3D peripheral view of the object.

In yet another aspect the invention relates to a kit of parts comprising the device and one or more software package for analysing the recorded information and/or reconstructing the 3D peripheral view of the object.

Description of the figures

Figure 1 shows a representation of a 3D mask image (a) and a 3D peripheral view (b) .

Figure 2 shows a schematic of the principle underlying SMI. L is a light source, M lis a grating, FI is a lens, f is a focal position and d is the distance between focal position and FI. The physical principle that allows the present invention to obtain 3D surface distances from structured illumination requires a light source (L) , a patterning means for generating a substantially periodic pattern of lighter and darker bands (Ml) and a focusing means (FI) . The pattern will be seen with maximum contrast between the dark and light bands at a given focal position (f) . The contrast of the pattern falls off with a known response as a function of distance, d, away from f and the banding of the pattern will eventually become indistinct. The measured modulation depth of the pattern on the surface of the object gives a direct link to the distance that the surface patch is away from the focal position f .

Figure 3 shows schematic representations of preferred embodiments of the projecting system (SI) . A detailed description of the figure is done in the section describing preferred embodiments of SI.

Figures 4-6 are schematic representations of preferred embodiments of the device. A detailed description of the figures is provided in the section describing preferred embodiments of the device .

Detailed description of the invention

The term object view is used to indicate views taken from an object from different points of observation. This is also referred to as different sides of an object.

The method according to the invention involves the evaluation of patterned images, which are obtained by projecting substantially periodic structures by means of a grating onto an object. Commonly to achieve this, gratings are imaged in the object planes and displaced by integral fractions of the grating constant. An image is recorded for each position of the grating and the images are recombined. We choose to describe this physical method Structured Modulation Imaging (SMI) . The claimed method which is based on the frequency transfer of a lens involves the evaluation of patterned images which are obtained by projecting substantially periodic structures by means of a grating onto an object. In this case the extent of defocus of the patterned image is recorded and the shape of the object is calculated. Frequency transfer methods differ from trigonometry methods of the prior art in 2 significant ways:

1. Imaging and viewing from the same position is allowed, i.e. no triangulation is required

2. The patterned image must defocus over the depth of the object (in triangulation no defocus is preferred) .

The invention relies on structured modulation imaging (SMI) . This technique relates the frequency pattern (modulation contrast) of a periodic pattern projected on the surface of an object to its surface structure. This frequency pattern is also referred to as brightness pattern of grey scale pattern. If a periodic grid of given frequency is projected via a lens onto a screen the observed intensity pattern on the screen has a modulation contrast that is at a maximum at the focal position of the lens. When the screen is moved away from the focal plane the observed modulation contrast on the screen diminishes . Reference is made to Figure 2 schematically showing the underlying principle. Figure 2 shows that the modulation contrast of the projected grating varies with distance from the focal plane. The relationship between modulation contrast and distance is a function of the lens properties, the spatial frequency of the periodic grid and wavelength of the light used. If an object is placed within the structured light field, surface features of that object that are in the focal plane show a high contrast of the projected grating while surface features away from the plane show a lower modulation contrast. The relationship between the distance from focal plane and the observed modulation for a lens system is given in W098/45745, Jwfad 4* i-u iiicuipuiάLed 'by r leirsftee. In WO-A-98/45745 and DE 19930816, jk-hgfe- "arc incorporated by- refcroece, it is disclosed that if the grid is displaced by fractions of the grating constant and an image of its projection on the object is recorded for each position of the grating, the algorithms such as those described in WO-A-98/45745 and DE 19930816 Al can be used to mathematically remove the pattern from the surface and hence reconstruct the 3D image of that surface.

WO-A-98/45745 relates to a microscopy imaging apparatus and associated imaging method and describes the use of structured light for generating an optically sectioned image of a specimen. In this method mask images are generated and only the in focus parts of the image generated on the object by the grating are used for the image reconstruction.

In DE 19930816 Al a similar SMI method is described in which surface information is determined by means of a projected structure with a periodic brightness, where this structure is displaced in steps of l/n (where n is a whole number greater than 2) of the grating constant and the projection is captured by means of a 2D CCD camera. An algorithm is presented that also uses the defocused parts of the projected image for the reconstruction of the 3D image.

Both documents have in common that they relate to microscopy and describe methods in which the surface of an object is viewed from a single point of view. The 3D information of the surface of one particular side of an object can then be obtained according to the methods described in the documents cited above . We call these surface structure images obtained from a single point of view ^Λmask' images in that they resemble face masks and do not contain 3D information from behind the front face of the object (reference is made to figure 1 where the difference between peripheral image and mask image is exemplified) .

The device and method according to the invention allow the rapid 3D imaging of the peripheral shape of an object within one or more operational step. An operational step is herewith defined as placing the object on/in the device, activate the device, obtain the 3D surface information (shape) of various different and/or all views of that object.

In a preferred embodiment of the invention the information of the 3D peripheral view is obtained in a single operational step. This means according to a preferred embodiment a user can place the object in the device, activates the device and obtains the information of the peripheral shape and/or colour of the object without having to carry out any intermediate steps.

The functioning of the device according to the invention relies on two principles. The first principle (embodied in steps a-d of the method according to the invention) leads to the generation of a mask. The second principle relies on masks being generated for at least two different views of the object. The information from the different masks is combined to reconstruct the peripheral view of the object.

Preferably the information is determined in terms of brightness values which are also referred to as different grey scales of the mask that is determined. Below, the components of the device according to the invention are listed.

An object according to the invention can be any object of suitable size to be placed in the device. Preferably the object is a macroscopic object, that means it can be seen with the naked-eye. The term "object" as used herein also comprises a multiplicity of objects.

The device according to the invention comprises one or more projecting systems (SI) . Each projecting system comprises one or more of the following: light source (L) , focussing means (FI) , patterning means (Ml) .

The object is illuminated by one or more light source (L) . Any light source can be used, such as for example halogen lamps, electric light bulbs, light sources emitting coherent light, light sources emitting incoherent light, lasers or a collimated light source. Preferably the light source is a fluorescent tube.

A substantially periodic pattern is generated by Ml and projected on the surface of the object by FI . A substantially periodic pattern is one in which there is a clearly dominant frequency component. The frequency components of a given spatial pattern can be estimated using standard computer means such as application of the Fast Fourier Transform algorithm.

The patterning means (Ml) can be for example a grating. Any grating known in the art can be used to work the invention. The gratings may be linear, planar or spiral, one, two or three dimensional. An electronic spatial light modulator, using transmitted or reflected attenuation schemes may also be used. Alternatively, a periodic pattern may also be generated by interference of two or more suitably arranged coherent light beams, e.g. lasers.

Another way of generating a periodic pattern is by electronically modulating the intensity of the light source L, for example by using a collimated light source, in combination with a suitable focusing means (FI) , for example a suitable mirror and a lens.

The periodic pattern generated by Ml can be one dimensional or two dimensional, stationary as well as movable. Preferably, Ml generates a periodic pattern having one dimensional local periodicity.

The position of Ml in relation to L, FI and the object can be anywhere as long as the periodic pattern can be projected on a surface of the object.

The focussing means can be any suitable focussing means known in the art, such as for examples lenses, mirrors, screens or a combination thereof. The lens can for example be spherical or cylindrical. Preferably the lens has a focal depth between 1 and 50 cm. In case FI comprises a mirror, the mirror can for example be curved or parabolic. Preferably the mirror has a focal depth of 1cm to 50 cm.

FI can be positioned anywhere in relation to L, Ml and the object suitably to project the generated periodic pattern on the object . In a preferred embodiment SI projects a one-dimensional (ID) periodic pattern on the object.

According to another preferred embodiment, the visible face of the object is in front or behind the focal point of the focusing means. It is preferred that the visible face of the object is not in the focal point of the focusing means. Even more preferred the object is positioned at the place where the contrast is less than 90% of its maximum value, in a region where the defocus function is at its steepest.

Preferred embodiments of SI are shown in figure 3 to which reference is made.

The preferred embodiment of SI shown in figure 3a comprises a light source (L) illuminating a 2D grating (Ml) and subsequently the object via a focusing means (FI) . The pattern shift means (M3) in this embodiment is a mechanically or electronically driven carriage for moving the grating in the plane of the grating and in a direction perpendicular to the ruled lines on the grating.

The preferred embodiment represented in figure 3b shows the light source ( ) illuminating the object via a 2D grating (Ml) and focusing means (FI) , and a screen D having a slit to produce a ID patterned line of illumination. The angle of the grating with respect to the slit and/or the direction in which the grating is moved provide two means for altering the spatial frequency of the ID illumination pattern, thereby allowing altered modulation contrast characteristics for a fixed focusing means system. The pattern shift means (M3) in this embodiment is a mechanically or electronically driven carriage for moving the grating in the plane of the grating and in a direction that is not parallel to the ruled lines on the grating.

Figure 3 (c) represents another preferred embodiment of SI comprising a spiral grating (Ml) , which is placed around a light source (L) , which preferably is a linear light source, most preferably a fluorescent tube. The focussing means FI comprise a lens, preferably a cylindrical lens. Only a small portion parallel to the long axis of the grating is used for projection of the pattern, so that the spiral grating approximates to a grating of parallel lines. This can, for example, be achieved either by placing a suitable screen (D) between light source and lens, whereby the screen comprises a slit parallel to the long dimension of the grating. Another way to achieve the same is by incorporating the screen in the light source or by placing the screen between light source and grating.

Spatial movement of the pattern is obtained by rotation of Ml carried out by the spatial shift means M3. The embodiment may optionally comprise one or more mirrors.

The preferred embodiment of SI represented by figure 3 (d) comprises a collimated light source (L) imaged via the focusing means (FI) comprising a scanning mirror assembly to produce a ID line of illumination on the object. By the combined operation of the scanning mirror and the synchronised intensity modulation of the light source a range of periodic ID patterns is generated.

The image of the periodic pattern, which is projected on the object, is recorded by one or more recording system S2. S2 comprises at least one recording means M2 and at least one focussing means F2. M2 can be any detector capable of receiving, digitising and storing the information obtained in the image of the projected pattern. M2 can for example be a 1 or 2 D array of optoelectronically active picture elements such as are known in the imaging art, e.g. CCD, CMOS or CIS devices. Preferably, the image-recording system is suitable to receive and record also colour information. This allows not only to reconstruct the peripheral shape of the object but also its colour. Preferably, M2 is a CCD device.

The imaging device can be operated as a continuous acquisition of the patterns such that each image represents an integration of the image over a small time period as the pattern is being moved. Alternatively, the images can be acquired after each movement .

F2 can be any focusing means known in the art as for example lenses, mirrors, screens or a combination thereof. F2 is preferably a lens with a focal depth between 1 and 50 cm. F2 may also be a mirror, preferably a parabolic mirror. Preferably the mirror has a focal depth between 1 and 50 cm. F2 can for example be a lens of a CCD camera.

In order to mathematically remove the projected pattern from the surface and to apply algorithms such as described in WO-A- 98/45745 and DE 19930816 Al the spatial phase of the pattern is altered. For this purpose the device is provided with a spatial shift means M3 moving Ml relative to the object. Movement of Ml relative to the object can be carried out either by moving the object or by moving Ml. M3 may be in the form of a mechanically or electronically driven carriage for moving the object relatively to Ml or vice versa. Movement can be carried out stepwise or continuously.

Movement serves to generate periodic patterns at altered spatial positions within the plane of periodicity of Ml on the object in order to enable removing of the projected pattern from the image and to construct a mask by using the algorithms described in DE 19930816 Al . It is noted that the quoted document refers to displacements by integral fractions of the grating constant. It has been found that in the device according to the invention, the displacements can be carried out at any fractions of the grating constant. It is further noted that periodic patterns at altered spatial positions according to the above in principle only need to be generated twice. If used twice the reconstruction of the image is approximate. In order to obtain an exact reconstruction the patterns are generated at least 3 times .

The device may further comprise a turntable on which the object is placed. The turntable may be moved electronically, mechanically, stepwise or continuously. The turntable may be transparent allowing also the side on which the object rests to be scanned by the device . The turntable may be rotated between 0 and 360 degrees within one or more planes.

In order to obtain the peripheral view of an object, masks from at least two different views of the object need to be constructed. In order to achieve this in a rapid and convenient way SI and S2 are arranged relative to the object so that masks can be constructed of at least two different views of the object. This can be achieved in a number of ways.

The device may for example comprise a turntable on which the object is placed. The turntable may be rotated stepwise or continuously, mechanically or electronically in relation to SI and S2. In this embodiment the device preferably comprises only one SI and one S2, which are at fixed positions. By moving the turntable between 0 and 360 degrees (not including 360 degrees) the object can be imaged peripherally.

Alternatively, SI and S2 may be in a fixed position relative to each other but capable to be moved around and/or over an object either continuously or stepwise, mechanically or electronically. In this embodiment, the object is preferably at a fixed position, although it is also possible to include a turntable in this device.

In another embodiment according to the invention the device comprises a plurality of projecting systems and recording systems at various positions, each combination of SI and S2 facing different sides of the object. The systems may be at fixed positions or movable.

According to another embodiment, the device is configured as to comprise a combination of the above.

In order to image also the side on which the object rests optionally a transparent turntable is used or a second operational step is introduced wherein the object is turned such that it rests on a different side and the scanning process is repeated. In order to reconstruct the 3D information of one view of the object (to generate a mask) algorithms such as described in DE 19930816 Al are applied for which the device preferably comprises suitable software. Preferably, the device is also provided with suitable software to recombine the masks for reconstructing the peripheral shape of the object. Furthermore, the device is optionally provided with means to display the generated peripheral view. The device may also be used as computer peripheral and is then provided with a means to record and store the information such that the information and/or the peripheral view can be processed, generated and displayed on a separate computer. The invention therefore also relates to a kit of parts comprising the device and a software package for analysing processing and displaying the surface information.

In another aspect the invention provides a method of 3D imaging of the peripheral view of an object. Parts of the method have already been illustrated during the description of the device. The method according to the invention comprises a) illuminating the object with a light source, b) projecting a first substantially periodic pattern on the object c) recording an image of the periodic pattern on the object, d) generating at least one more time the periodic pattern at an altered spatial phase of the first pattern, wherein steps a-d are carried out for at least two different views of the object and wherein the data obtained are analysed to reconstruct the 3D peripheral view of the object.

The generation of the patterns can be done in any order suitable for reconstructing the peripheral view of the object. This is further demonstrated by preferred embodiments, which are not to be understood as limiting.

In one embodiment steps a-d are carried out for one particular view of the object. Steps a-d are then repeated for other views of the object and the combined data is analysed for the reconstruction of the peripheral view.

In another embodiment a plurality of systems SI and S2 record images of periodic patterns that have been generated at different sides of the object. After these images have been recorded the systems are moved relative to the object and the periodic patterns are generated and recorded at an altered spatial phase of the first patterns .

Prior to use the device can be calibrated. Calibration can be carried out once and the device is set up or calibration can be carried prior to every single use .

Calibration of the device can for example be provided as follows. A calibration means of known dimension is inserted into the device and the 3D scanning process is carried out. The observed values of modulation ratio for each surface patch are then cross-correlated with the known distances that exist for each surface patch. The results of the correlation are stored in a look up table so that during normal operation an observed modulation ratio value is transformed into distance before further data processing. Alternatively all data processing can be carried out in modulation ratio values and after processing transformed into real distances.

In a preferred embodiment of the invention a calibration figure is inserted on, above or below a turntable and fixed so that on rotation of the turntable the figure rotates . The observed values of modulation ratio for each surface patch are then cross-correlated with the known distances that exist for each surface patch on the calibration figure. The results of the correlation are stored in a look up table so that after a complete revolution of the turntable the observed modulation ratio values can be transformed into distances before further data processing. By this arrangement each full 3D scan is self calibrated without any further calibration steps.

The invention is now further described by a number of preferred embodiments .

Embodiments of the Device

Embodiment I

A schematic of a preferred embodiment of the device is shown in figure 4.

In this embodiment of the invention SI is described in figure 3c, i.e. comprising a fluorescent tube as a light source (L) , being surrounded by a spiral grating (Ml), a screen with a slit parallel to the long axis of the light source (D) and a cylindrical lens (FI) . A spatial shift means (M3) is connected to the grating capable of rotating the grating. The device comprises a turntable on which the object can be placed (indicated by bent arrow) . S2 is a CCD camera.

A preferred way of operating this device is as follows: The object is illuminated and a periodic pattern is generated, projected on the object and recorded. The spiral grating is rotated. A periodic pattern of altered spatial phase is generated, projected and recorded. The spiral grating is rotated again leading to another periodic pattern being generated, projected and recorded. A mask of this view of the object is generated. The turntable is now moved by 90 degrees so that a different side of the objects faces SI and S2. A mask of this view is generated by repeating the procedures described above. The turntable is then turned again by 90 degrees and the procedures described above are repeated. The data of the different masks are combined for reconstruction of the peripheral view of the object.

Another preferred way of operating that device is by scanning a complete peripheral image of the object at one position of the spatial phase of the grating. Then the spatial phase of the grating is shifted by rotation and another peripheral image is scanned. This procedure is repeated once more and the obtained data are processed to reconstruct the peripheral view.

Embodiment II

This embodiment is similar to embodiment I . In this embodiment S2, a CCD camera, is attached to SI which is the preferred embodiment shown in figure 3c, and both SI and S2 are moved around and/or over the object thus obtaining surface information of several views of the object. This embodiment is represented in figure 5.

Embodiment III

This embodiment comprises three projecting and recording systems (SI and S2) arranged at different positions in the device so that each combination of projecting and recording system faces a different side of the object. A representation of this embodiment is shown in figure 6.

Claims

1. A device for 3D imaging of the peripheral view of an object using structured modulation imaging comprising: -one or more projecting system (SI) generating a periodic pattern on the object, said system comprising one or more light source (L) , one or more focussing means (FI) , one or more patterning means (Ml) ,

- one or more image recording system (S2) comprising a focussing means (F2) and one or more means (M2) of recording an image of the periodic pattern generated on the object, wherein SI, S2 and the object are arranged relative to each other such that the periodic patterns can be generated and recorded for at least two different views of the object, said device further comprising means (M3) allowing Ml and the object to be moved relative to each other.

2. A device according to claim 1 wherein the object is a macroscopic object.

3. A device according to any preceding claim comprising software for analysing the recorded information and/or reconstructing the 3D image of the object.

4. A device according to any preceding claim wherein M2 is a CCD camera.

5. A device according to claim 6 wherein F2 is a lens of the CCD camera.

6. A device according to any preceding claim further comprising a turntable on which the object can be placed.

7. A device according to any preceding claim comprising a plurality of SI and S2 in a fixed spatial arrangement each combination of SI and S2 facing different sides of the object .

8. A device according to any of claim 1-6 comprising one SI and one S2 in a fixed spatial position and a turntable which can be turned in relation to SI and S2.

9. A device according to any of claim 1-6 wherein SI and S2 are fixed in relation to each other and wherein SI and S2 can be moved around and/or over the object.

10. A device according to any preceding claim wherein the periodic pattern generated has local one dimensional periodicity.

11. A device according to any preceding claim wherein the periodic pattern is recorded in terms of brightness values .

12. A method for 3D imaging of the peripheral view of an object using structured modulation imaging comprising a) illuminating the object with a light source, b) projecting a first substantially periodic pattern on the object c) recording an image of the periodic pattern on the object, d) generating at least one more time the periodic pattern at an altered spatial phase of the first pattern characterised in that steps a-d are carried out for at least two different views of the object and wherein the data obtained are analysed to reconstruct the 3D peripheral view of the object.

13. A method according to claim 12 wherein the object is a macroscopic object.

14. A method according to claim 12 wherein the periodic pattern is a brightness pattern.

15. A kit of parts comprising a device according to claim 1 and one or more software package for analysing the recorded information and/or reconstructing the 3D peripheral view of the object .