EP0882276A1

EP0882276A1 - Automation methodology for 3d image metrology systems

Info

Publication number: EP0882276A1
Application number: EP97905342A
Authority: EP
Inventors: Horst A. Beyer
Original assignee: Imetric SA
Current assignee: Imetric SA
Priority date: 1996-02-20
Filing date: 1997-02-17
Publication date: 1998-12-09
Also published as: SE9600623D0; WO1997031336A1; AU2226397A; SE9600623L; JP2000505548A; BR9707593A; KR19990082673A; CA2245175A1; CN1212064A; DE882276T1; RU98117236A

Abstract

Method of 3D image metrology using auto-identifiable targets, by determining the exterior orientation of images comprising targets and/or pattern or groups of targets.

Description

Automation methodology for 3 D image metrology systems

TECHNICAL FIELD

The present invention relates to 3D-image metrology with photogrammetric methods.

The invention is applied to 3D Image Metrology but can also be applied to other electro-optical systems. 3D Image Metrology is applicable to all areas where 3D geometric and other information is to be extracted from imagery, imagery being acquired through standard imaging systems or other devices such as imaging radar or sonar. Typical application areas are dimentional measurements to be performed in the industrial process reaching from design, development, production to quality control, in engineering but even in areas such as art and medicine.

BACKGROUND OF THE INVENTION

The background of the invention is methodologies for the automation of so far manual tasks in 3D image metrology systems.

For exemple, robots today are programmed off line which means a teaching of robots using a computer simulation program. Due to technical limits in the mechanical movements of a robot arm, there is a difference between the programmed position of a robot arm and its real position. This differecne can be measured with the robot calibration system and correction polynoms for the different ankles can be derived. Using such a correction for the simulation, the accuracy of the real robot movement can be increased. Three or four CCD-cameras are mounted in the vinicity of the robot to observe the measurement volume. The cameras detect auto-recognizable targets mounted on and in a specific relation to the tool center point. According to these detections a calibration process is activated and corrections for the simulation is established.

Targets were designed to faciliate automatic detection and identification. For exemple, the targets could be designed as a white circle with a concentric ring on a black base. Around the concentric ring there is another concentric ring as a label area in which a ten bit target identification is coded. The central circle has a black dot in the middle to faciliate manual measurement. Black and white are colors chosen to get the maximum contrast.

Therefore, part of the invention, i.e. auto-recognizable targets, is already used.

A large number of attemts to arrive at this goal of automation have been undertaken. All attemts so far have only achieved partial results and/or are not suited for widespread use.

One approach of automatically identifying individual targets was relying on different forms and shapes of the targets. Targets which were made up of various geometric shapes such as circles, rectangles and crosses were used to differentiate between 2D patterns of targets. This approach need relatively large targets and does not allow for a large number of different patterns.

Another approach used targets with an additional linear pattern area in the vicinity of circular targets. The pattern was similar to typical bar codes. The code was such not integrated with the target per se and needed relatively much space. These targets were so called "coded" targets and were developed by several groups at that time.

Another approach with "coded" targets uses a bit pattern around the targe . The pattern was a binary bit pattern which was designed to be independent of the rotation of the taget. Two major disadvantages were common to these targets . They had too small distance between target and code when used with CCD-cameras and they could only use the labels provided by the code, i.e. typically numbers like 1,2,3 but no alphanumeric labels. The insufficient amount of space between target and code degraded the measurement accuracy of the target. Groups with several hundred targets were discernible with that approach. All codes provided only to use the same label for the target as the label which was encoded in the labeling area of the target.

Furthermore, groups of targets were used by some systems to identify one particular group and through that group individual targets of the group. These techniques rely on the particular geometric relationship of all targets in a group which can either be identified in an image and/or in 3D space.

Other approaches to identify an element was to use groups of targets which lie in a particular geometric relationship to eachother and which discern the group from other targets within an image or even within the 3D space. Disadvantages of these techniques ly in the fact that one always needs several targets and that the technique does not allow for a large number of different groups with resonable economic effort and space requirements.

The above approches were furthermore combined with the use of colours in order to increase the number of targets which are discernible.

Another aspect of automation is the establishment of correspondence of targets which are not discernible through their particular individual characteristics such as a label, a colour, a form. Two basic techniques were developed to overcome this problem. One uses the geometric relation between images from which geometric conditions can be derived, i.e. the so called epipolar line. Using this technique the correspondence between large numbers of targets could be established automatically. A similar approach where the geometric relation between images and images and the object space is used is the Multiphoto Geometrically Constrained Matching. In another approach the 3D relation of points was used directly. Both techniques only adressed part of the problem as they required that the exterior orientation and/or the relative orientation of images had to be known a priori .

The establishment of the correspondence of homologous points in images can be performed by the use of the epipolar geometry, as well as using the 3D relation of points.

These approaches of the known solutions to automatically identify individual targets were rather limited and shows disadvantages in the following ways:

1. The targets could not be measured with sufficient accuracy (not for all techniques) .

2. Only numeric labels were associated with the targets although in many applications alphanumeric labels are required.

3. Groups of targets require much more space than a target with an associated label area.

4. Groups of targets are not sufficiently discernible from other targets in applications with large numbers of targets.

The automatically indentifiable targets have priviously been designed with film based 3D image metrology systems in mind. They did not take care of the imaging characteristics of CCD-sensor or other digital imaging systems. The establishment of the correspondence did not use auto- identifiable targets to establish the initial orientation of the images. With respect to a complete system they did not allow to: Automatically identify the object.

1. Provide for a mechanism to automatically load all the relevant data for a particular application.

2. Provide for highest measurement accuracy, immunity to occlusions, and alphanumeric labels. 3. Provide for automatic identification of adapters independent of knowledge on their spatial location.

Overall, the existing methods so far were only addressing part of the problem.

OBJECT OF THE INVENTION

The object of the invention, in parts or total, is to provide to partial and/or complete automation for dimentional measurements with image metrology. Most important are systems where 3D image metrology is to perform fully automatically, such as in automated production systems where robots or other manufacturing systems are to be guided, e.g. for machining, fitting, moving.

An other object of the invention is to provide for a higher degree or even total automation in 3D image metrology and/or other applications where such systems are used.

SUMMARY OF THE INVENTION

The invention applies to the following subtasks:

- Finding targets for the purpose of automatically determining the exterior orientation of images. Finding targets in several images for the purpose of establishing homologous point in these images.

Finding targets from different views in an electro- optical system in order to establish their relation.

Identifying targets which serve as references for the definition of a coordinate system.

- Identifying measurement adapters in order to identify the particular adapter and its geometric and/or other characteristics. Examples for such adapters are:

- Individual targets for which some offsets and other geometric parameters are known.

- Two or more targets forming a straight line with another spatial location which they refer to.

- Three or more targets which are in a spatial relation to one or more other spatial locations. Examples thereof are:

Adapters consisting of three or more visible targets and three or more half spheres which lie on an object surface. The visible targets and the other locations are in a known relative spatial relation.

- Adapters consisting of three or more visible targets which are in relation to one or more locations. The visible targets and the other locations are in a known relative spatial relation. Physical examples are the DigiPen, which is a touchprobe of the Imetrics systems, or a spindle and the targets on the spindle holder in a TI² system.

- LUT

The invention will be part of a future improvement of a 3D image metrology system. It is applicable to other systems. It will also be used in an industrial manufacturing system.

DESCRIPTION OF THE INVENTION

The invention uses the following steps in the current implementation either on an individual basis or as in any combination of the individual techniques.

1. Measurement Accuracy: The design of the auto- identifiable targets is such that the distance between the central target and the label area, which is used to identify the target, i.e. the area with a binary code or other code, is sufficiently large in order not to degrade the accuracy with which the central target can be located. This distance was determined to be at least equal to the diameter of the central target when the central target was designed to provide for an optimum measurement accuracy, i.e. to be imaged on typically 6 pixels in diameter.

2. Immunity to partial occlusions of label area: In many applications, part of the area containing the label can be occluded from other objects lying between the imaging system and the label area. Techniques using a redundancy in the code, additional areas such as a border around the label, and a-priori knowledge on which auto- identifiable targets were used, can be used individually or in combinations to circumvent this problem.

3. Labeling Freedom: To overcome the problem of having to apply particular targets on particular points which one would like to label differently, a translation between the label of the target and its physical label is used. Such an auto-identifiable target with the label "11" can be used as a target with a label "A". This can for example be implemented using a look up table where a "user" label is connected to each physical label of a target. This look up table can be user definable or fixed. Furthermore a double look up table can be used, i.e. a table where the label provided by the label area is first translated into another label and later is finally related to a label to be used in the further processing. For example, the label on the target contains the label "11". On the target, the name "A" can be used to identify it to a person, but in the particular application the target is to be used as target "X" . This helps to reduce the number of auto- identifiable targets physically required in order to fullfill a large number of different applications.

4. Identification of the Measurement Object: Here, one or more labels of auto-identifiable targets are used to identify a particular object. For example, the hood of model A of a car will have auto-identifiable targets with label "K" and the hood of model B of a car will have auto-identifiable targets with label "Y" . One or more labels can be used. Based on this information, data related to the particular object can be automatically selected. Use of such an object identification as a safe guard (for example in manufacturing, but also when performing quality inspection) and/or to load data related to the particular object.

5. Default Project Strategy: A methodology used in connection with the auto-identifiable targets where the data pertaining to a particular task (e.g. coordinates of reference locations, compensation for adapters, measurements to be performed) is collected in a "default project". This can be a collection of files containing the pertinent data or a particular access to a data base or some other technique to store and access the relevant data.

6. Automated Orientation: A procedure where in the setup of a measurement system auto-identifiable targets are used to automatically compute the exterior orientation and/or relative orientation of one or more cameras/images.

A procedure where during measurements auto-identifiable targets are used to automatically compute the exterior orientation of one or more cameras/images in order to account for eventual changes in the relation between the object (s) on which the auto-identifiable targets are fixed and the camera (s) and/or to perform other measurements.

Any procedure where three or more auto-identifiable targets are used to establish a particular coordinate system.

In all these procedures, the 3D coordinates of at least three auto-identifiable targets must be known.

7. Automated Measurement:

Procedures where auto-identifiable targets are used to establish the geometric relation of images with respect to a coordinate system related to the auto-identifiable targets (exterior orientation) or to some arbitrary coordinate system, for example related to one or more of the cameras/images (relative orientation) in order to be able to establish the correspondence between standard targets in two or more images.

Determining the exterior orientation of individual images using for at least three auto-identifiable targets of which the 3D spatial coordinates in some 3D coordinate system are known, either prior to start of measurements or which were determined in the course of the measurements. Determining homologous points in images. This can for example be used to compute the relative orientation of these images.

Combination of auto-identifiable targets and standard targets. This technique allows to establish the spatial geometric relation of the images, i.e. either their exterior orientations or their relative orientations, through auto-identifiable targets and eventual other targets which were already located in the images but not identified (i.e. the relation between a particular target in an image and its origin in object space and/or its corresponding image in another image is not known) , and to establish the correspondence between homologous targets imaged in different images using the geometric relation of these images (i.e. their exterior orientations or their relative orientations) and/or the 3D geometry of the targets to be identified which becomes computable using the geometric relation of the images in which they were imaged.

Targets can be retro-reflective targets but also any other feature of an object which can be located in an image or a group of images.

8. Scale Bar: The use of one or more auto-identifiable targets to identify a particular scale bar in order to obtain its calibrated distance or distances (if more than two points are located on the scale bar) . The following elements are part of the technique:

1) Identify the particular scale bar

2) Help in the search for others targets on the scale bar

3) Use several known distances on one scale bar for the purpose of improved accuracy and reliability. They can be applied individually or in combinations.

9. Particular adapters or object are identified by using either one or more of the labels. Using a single or a group of adapters is then in turn used.

One or more labels are used to identify a particular object. Such an object can be:

- An individal point with or without some given characteristics such as XYZ-coordinates for the purpose of orientation. Using the label, the 3D spatial coordinates of a point can be taken from a data base in order to compute exterior orientations, to compute spatial position of an object with respect to some coordinate system through a spatial similarity transformation.

- An individual label serving to identify an adapter with its characteristics. These can for example be the offset of a button target, the vector distances for reductions of double vector targets or hidden point bars.

- A multi-point adapter where the label is used to find the geometric relation between visible targets and another point or points which are in particular geometric relation to the targets.

- A particular scale bar.

- A particular touch probe or other device such as a spindle or other manufacturing device and other objects which are to be identified.

Adapters are any technique where one or more auto- identifiable targets on any sort of adapter is to: - Identify the particular adapter to derive geometric parameters of the adapter.

- Identify the particular adapter to retrieve its spatial coordinates from a data base or other storage mechanism. - Identify the particular adapter to control the software such that it performs a particular measurement sequence or other process. Examples: a) The system finds a double vector target and computes automatically the mechanical point related to the two optical targets. b) The system finds a particulat target group and commands a robot to go and grab the piece.

The following first defines a number of adapters and then defines the use of the auto-identifiable targets with these adapters.

Single Point Adapter: A "Single Point Adapter" is a device consisting of one optically visible target, e.g. a retro-reflective target or LED, and one or more mechanical points, e.g. the location of a shank, the center of a sphere where the relation of the optical and mechanical targets are known and usually fixed.

Typical examples are: - A "Button Target" which consists of one optical target sitting on a mechanical piece with a shank adapted to tooling holes. The target is in the centre of the shank axis but at a certain distance from a mechanical interface. - Target integrated into a partial sphere as for example targets integrated into a Taylor Hobson sphere.

The auto-identifiable target is used to identify an individual "Single Point Adapter" and to extract some generally geometrical characteristics from a data base.

Such information can be the offset of the optical target and the one or more mechanical points. This allows to use adapters with various offsets in one measurement task without requiring the user to identify the adapter and without relying on applying one particular adapter with a particular offset to specific locations. The auto-identifiable target is used to find the 3D reference coordinates of the "Single Point Adapter", i.e. either of the optical target or the reference location.

A "Two Point Adapter" is a device consisting of two or more optically visible targets and one or more mechanical or virtual points. The differentiation between "Two Point Adapters" and "Multi-Point Adapters" is the fact that the first one does not allow to establish a 3D relation between the optically visible targets and the mechanical or virtual points without additional knowledge on the spatial location of the "Two Point Adapter". In geometric terms, the points of the "Two Points Adapter" cannot be used to determine a 3D spatial similarity transformation with six parameters. Usually the point (s) to be computed using a "Two Point Adapter" are on a 3D line with two optical targets.

Typical examples of "Two Point Adapters" are:

"Double Vector Targets". This adapter consists typically of two optically visible targets. The mechanical or virtual point of interest is on a spatial line with the two optically visible targets and its (spatial) distance from one or both targets is known or it may be in the geometric centre of the two optically visible targets.

One or more auto-identifiable targets are used to identify a particular "Two Point Adapter" in order to: - Identify the adapter and obtain its geometric parameters and/or the naming convention for the one or more points to be computed.

- Identify the adapter to retrieve the spatial coordinates of one or more optical targets and/or one or more reference locations of the adapter.

- Identify the adapter to control particular software functions or to indicate certain actions by the system.

A "Multi-Point Adapter" is a device consisting of three or more optically visible targets which are in a geometric relation to an object and/or one or more mechanical or virtual points. The "Multi-Point Adapter" can for example be used to determine the spatial position and orientation of the adapter or an object physically connected to the adapter. Furthermore, it can be used to derive the spatial location of one or more points on the adapter or another object physically connected with the adapter.

Examples of "Multi-Point Adapters" are: - V-Plates: A device consisting of three optical targets and three half spheres.

- Robot Calibration Fixtures: Devices fixed to robots for the purpose of calibrating robots.

- Touch Probes such as the DigiPen and other devices consisting of three or more optical targets connected to a CMM touch probe for the purpose of determining the 3D spatial coordinates of the probe.

- Tooling Fixtures: Any device on a tool which serves to position a piece in a manufacturing production process. Adapters can be built to position tooling fixtures .

- Tool holders or Tools (drills, grippers, cutters and other end effectors) : These can be on manufacturing systems such as robots or robot arms, CNC machines, but also on hand-held devices.

The use of auto-identifiable target (s) for:

- Identifying the particular adapter to derieve its geometric characteristics and/or to retrieve its spatial reference coordinates.

- Identifying the particular adapter to initiate particular actions or computations to be performed such as to automatically compute the invisible targets.

Notes : - Targets to be mounted in tooling holes or other places with a target that can automatically be identified, labeled and measured by a computer program.

Automatically measuring targets for the purpose of determining the orientation of one or more cameras automatically for the purpose of orienting cameras.

Update orientation.

Tool with three or four targets to automatically orient cameras. - Devices with three or more points that are automatically identifiable and are used to derive

1. the position and/or attitude of the device that they are attached to in space

2. the position of one or more other points physically connected with these points in space.

Tool to perform an automatic orientation and to automatically compute approximations for object points.

Method where points are labeled by indicating their label in one image only.

Coded targets to position other devices such as robots, CMMs, CNC machines.

Claims

Claims :

1. Method of 3D image metrology using auto-identifiable targets, characterized by determining the exterior orientation of images comprising targets and/or pattern or groups of targets.

2. A method as claimed in claim 1, characterized by finding auto-identifiable targets in several images and establishing homologous points in these images.

3. A method as claimed in any of the claims 1-2, characterized by finding auto-identifiable targets from different views and establishing their relative relation.

4. A method as claimed in any of the claims 1-3, characterized by identifying auto-identifiable targets which are defined as a reference for the definition of a coordinate system.

5. A method as claimed in any of the claims 1-4, characterized by identifying measurement adapters for identification of a particular adapter with its characteristics.

6. A method as claimed in any of the claims 1-5, comprising a central target and a label area characterized by applying the label area at distance from the central target beeing sufficiently large in order not to degrade the accuracy with which the central target can be located.

7. A method as claimed in claim 6, characterized in that the distance is beeing set at a value least equal to the diameter of the central target when designed for an optimum measurement accuracy.

8. A method as claimed in claim 7, characterized in that the distance is determined as a result of the central target to be imaged on typically 6 pixels in diameter.

9. A method as claimed in any of the claims 1-8, characterized by establishing immunity to partial occlusions of label area, for exemple techniques using a redundancy in the code, additional areas such as a border around the label, or a-priori knowledge on which auto- identifiable targets, can be used individually or in combinations.

10. A method as claimed in any of the claims 1-9, characterized by that a translation between the label of the target and its physical label is made in order to establish a labeling freedom to apply particular targets on particular points which one would like to label differently.

11. A method as claimed in any of the claims 1-10, characterized by using one or more labels of auto- identifiable targets to identify a particular object.

12. A method as claimed in any of the claims 1-11, characterized in that data, e.g. coordinates of reference locations, compensation for adapters, measurements to be performed, pertaining to a particular task, is collected in a default project.

13. A method as claimed in any of the claims 1-12, were the 3D coordinates of at least tree auto-identifiable targets is known, characterized by establishing an automated orientation by a procedure where in the setup of a measurement system auto-identifiable targets are used to automatically compute the exterior orientation and/or relative orientation of one or more cameras/images.

14. A method as claimed in any of the claims 1-13, characterized by establishing an automated orientation by a procedure where during measurements auto-identifiable targets are used to automatically compute the exterior orientation of one or more cameras/images in order to account for eventual changes in the relation between the object (s) on which the auto-identifiable targets are fixed and the camera (s) and/or to perform other measurements.

15. A method as claimed in any of the claims 1-14, characterized by establishing an automated orientation by a procedure where three or more auto-identifiable targets are used to establish a particular coordinate system.

16. A method as claimed in any of the claims 1-15, characterized in that an automated measurement is achieved by procedures where auto-identifiable targets are used to establish the geometric relation of images with respect to a coordinate system related to the auto-identifiable targets or to some arbitrary coordinate system, for example related to one or more of the cameras/images in order to be able to establish the correspondence between standard targets in two or more images.

17. A method as claimed in any of the claims 1-16, characterized in that an automated measurement is achieved by determining the exterior orientation of individual images using at least tree auto-identifiable targets of which the 3D spatial coordinates in some 3D coordinate system are known, either prior to start of measurements or which were determined in the course of the measurements.

18. A method as claimed in any of the claims 1-17, characterized in that an automated measurement is achieved by determining homologous points in images.

19. A method as claimed in any of the claims 1-18, characterized in that an automated measurement is achieved 19 by establishing the spatial geometric relation of the images, through auto-identifiable targets and eventual other targets which were already located in the images but not identified and to establish the correspondence between homologous targets imaged in different images using the geometric relation of these images and/or the 3D geometry of the targets to be identified which becomes computable using the geometric relation of the images in which they were imaged.

20. A method as claimed in any of the claims 1-19, characterized by identifing a particular scale bar in order to obtain its calibrated distance or distances.

21. A method as claimed in any of the claims 1-20, characterized by applying one or more auto-identifiable targets on any sort of adapter.

22. A method as claimed in any of the claims 1-21, characterized by identifying the particular adapter to derive geometric parameters of the adapter.

23. A method as claimed in any of the claims 1-22, characterized by identifying the particular adapter to retrieve its spatial coordinates from a data base or other storage mechanism.

24. A method as claimed in any of the claims 1-23, characterized by identifying the particular adapter to control the software such that it performs a particular measurement sequence or some other process .

25. A device using 3D image metrology auto-identifiable targets, characterized by a determination apparatus for the exterior orientation of images comprising targets and/or pattern or groups of targets.

26. A device as claimed in claim 25, characterized by comprising any features or combination of features claimed in any of the claim 1-24.