JP7150105B2 - 3D image processing device and 3D image processing method - Google Patents

Description

The present invention relates to a 3D image processing apparatus and a 3D image processing method.

Many production sites such as factories have introduced image processing devices that automate and speed up inspections that used to rely on human visual inspection. The image processing device uses a camera to capture an image of a workpiece that is flowing through a production line such as a belt conveyor, and uses the obtained image data to perform measurement processing such as edge detection and area calculation of a predetermined area. Then, based on the processing result of the measurement processing, inspections such as workpiece chipping detection and alignment mark position detection are performed, and a determination signal for determining the presence or absence of workpiece chipping and positional deviation is output. Thus, the image processing device may be used as one of the FA sensors.

An image targeted for measurement processing by an image processing apparatus used as an FA sensor is mainly a luminance image that does not contain height information. For this reason, when it comes to detecting chipping of workpieces as described above, we are good at stably detecting the two-dimensional shape of chipped portions, but we are good at stably detecting three-dimensional shapes that are difficult to appear in luminance images, such as the degree of dents in scratches. is difficult to detect. For example, it is conceivable to indirectly detect the three-dimensional shape by detecting shadows caused by dents in scratches by devising the type and direction of lighting that illuminates the workpiece during inspection. Clear shadows are not always detected. In order to prevent erroneous judgment of defective products as non-defective products when unclear shadows are detected, for example, if the judgment threshold is biased toward the safe side, a large number of non-defective products will be judged as defective, resulting in a decrease in yield. There is a risk of inviting

Therefore, in addition to the luminance image in which the pixel value is the grayscale value corresponding to the amount of light received by the camera, the height is expressed two-dimensionally by using the grayscale value as the pixel value according to the distance between the camera and the workpiece. Appearance inspection using a distance image (see, for example, Patent Document 1) is conceivable.

An example of a three-dimensional image processing apparatus is shown in the schematic diagram of FIG. This three-dimensional image processing apparatus includes a head unit having an imaging means such as a light receiving element, and a controller that is connected to the head unit, receives image data captured by the head unit, and generates a distance image from the acquired image data. consists of

Here, the principle of triangulation will be explained with reference to FIG. In the head portion, an angle α between the optical axis of the incident light emitted from the light projecting portion 110 and the optical axis of the reflected light entering the light receiving portion 120 (the optical axis of the light receiving portion 120) is set in advance. . Here, when no work is placed on the work, the incident light emitted from the light projecting section 110 is reflected by the point O on the placement surface of the work and enters the light receiving section 120 . On the other hand, when a work is placed on the work, the incident light emitted from the light projecting section 110 is reflected by a point A on the surface of the work and enters the light receiving section 120 as reflected light. Then, the distance d in the X direction between the points O and A is measured, and the height h of the point A on the surface of the workpiece is calculated based on this distance d.

The three-dimensional shape of the workpiece is measured by calculating the heights of all points on the surface of the workpiece by applying the measurement principle of triangulation described above. In the pattern projection method, all points on the surface of the workpiece are irradiated with incident light. Therefore, the incident light is emitted from the light projection unit 110 according to a predetermined structured pattern, and the reflected light reflected on the surface of the workpiece is received. To efficiently measure the three-dimensional shape of a workpiece based on a plurality of pattern images.

As such pattern projection methods, a phase shift method, a spatial encoding method, a multi-slit method, and the like are known. By three-dimensional measurement processing using a pattern projection method, the projection pattern is changed, and the head section repeats imaging a plurality of times, and sends it to the controller section. In the controller section, calculation is performed based on the pattern projection image sent from the head section, and a distance image having height information of the workpiece can be obtained.

On the other hand, existing image processing apparatuses mainly use a luminance image in which luminance is used as a pixel value. For example, there is a system that inspects a workpiece conveyed on a line by image processing by capturing a brightness image that expresses the appearance of the workpiece with optical shading using a monochrome camera. Under such circumstances, if a new three-dimensional measuring device and a processing device for processing three-dimensional data (point cloud data) output therefrom are introduced, a considerable cost is incurred.

Therefore, if the height information is represented as a distance image that expresses the gradation of the image, the image data itself to be handled is equivalent to the existing luminance image. It can handle images.

In this case, in the conventional luminance image, an image with a relatively low number of gradations is used. For example, images in which information per pixel is expressed in eight gradations are often used. On the other hand, image data having height information requires a large number of gradations (for example, 16 gradations) to express the height information. Therefore, in order to process the distance image with an existing image processing device that handles low-tone images, it is necessary to convert the multi-tone distance image into a comparatively low-tone distance image. . (Hereinafter, an image obtained by converting a multi-tone distance image into a low-tone distance image is called a "low-tone distance image", and a conventional luminance image having luminance information is called a "luminance image".)

In this way, in addition to the luminance image that has been used conventionally, by using the low-gradation-converted low-gradation-distance image, an image such as the conventional shape extraction can be used when inspecting whether a workpiece is normal or abnormal. It is possible to use not only processing but also processing such as height measurement using height information in the three-dimensional image processing apparatus.

In this case, gradation conversion for converting height information of high gradation (for example, 16 gradations) into the same number of gradations as the existing luminance image (for example, 8 gradations) is essential. At this time, if the gradation conversion parameters are not appropriately set, height information necessary for inspection may be lost, and an image suitable for inspection may not be obtained.

Therefore, it is conceivable to use the height information of the workpiece to be inspected to appropriately set the gradation conversion parameters and determine the reference height for gradation conversion.

However, when there is variation in the height direction of the work to be inspected by the user, it may not be possible to obtain an image suitable for inspection simply by performing gradation conversion using the initially specified gradation conversion parameters.

Japanese Patent No. 4969478

The present invention has been made to solve such conventional problems. A main object of the present invention is to provide a 3D image processing apparatus and a 3D image processing method that suppress the loss of height information and suppress the deterioration of accuracy when converting a distance image into a low gradation distance image. to do.

Means for solving the problem and effects of the invention

In order to achieve the above object, according to the three-dimensional image processing device according to the first aspect of the present invention,
A three-dimensional image processing apparatus capable of acquiring a distance image including height information of an inspection object and performing image processing based on the distance image, comprising imaging means for taking an image of the inspection object. a distance image generating means capable of generating a distance image based on the image captured by said imaging means; a reference plane setting means for setting a reference plane to be used as a reference for gradation conversion into a number of distance images; gradation conversion means for performing gradation conversion to a low gradation distance image in which the height information of the distance image is replaced by the grayscale value of the image, and which has a gradation number lower than that of the distance image ; Extraction direction input means for designating a direction for extracting a local shape change at the time of gradation conversion by the gradation conversion means; inspection executing means for executing a predetermined inspection process, wherein the reference plane setting means calculates a curved surface from which high-frequency components are removed from the distance image generated by the distance image generating means; A curved surface is set as a reference surface , and the extraction direction input means can specify any of the X direction, Y direction, or XY direction of the range image as the direction for extracting the local shape change . With the above configuration, it is possible to extract changes in the inspection object in the direction specified by the extraction direction input means. shape extraction becomes possible.

Further, according to the three-dimensional image processing apparatus according to the second aspect, in addition to the above configuration, the distance image generating means may generate a Sequentially generating range images of the inspection object based on the input image of the captured inspection target, and the reference plane setting means resetting the reference plane based on height information of the sequentially generated range images. Therefore, the reference plane can be changed according to the height of the inspection object.

Further , according to the three-dimensional image processing apparatus according to the third aspect, in addition to any one of the above configurations, the tone conversion means once reduces the distance image and performs filter processing upon tone conversion. A free-form surface image is created by post-expansion, and a difference between the free-form surface image and the distance image is taken to extract the shape change of the distance image. You can only shrink in the direction specified by . With the above configuration, by reducing the distance image only in the specified direction without reducing the direction other than the direction specified by the extraction direction input means, the uniform image in the specified direction of the free-form surface image By leaving continuous height information and taking a difference from this, it is possible to eliminate uniformly continuous gradual shape changes in that direction and extract only the necessary shape.

Furthermore, according to the three-dimensional image processing apparatus according to the fourth aspect, in addition to any one of the above configurations, it further includes a light projecting means for projecting light onto the inspection object by the light section method. The image capturing means may capture an image projected by the light projecting means, and the distance image generating means may generate a distance image by synthesizing profiles obtained by a light section method.

Furthermore, according to the three-dimensional image processing apparatus according to the fifth aspect, in addition to any one of the above configurations, incident light is projected obliquely to the optical axis of the imaging means for structured illumination in a predetermined projection pattern. wherein the imaging means captures a plurality of pattern projection images by acquiring reflected light projected by the light projecting means and reflected by the inspection object, The distance image generating means can be configured to generate a distance image based on a plurality of pattern projection images captured by the imaging means. With the above configuration, inspection processing such as height measurement using a distance image using pattern projection can be realized.
Furthermore, according to the 3D image processing apparatus according to the sixth aspect, in addition to any one of the above configurations, a height to be extracted based on the reference plane when performing gradation conversion by the gradation conversion means and extracting height setting means for specifying height information, and the extracting height setting means selects the height information to be extracted, which is higher or lower than the reference plane, or both higher and lower than the reference plane. When the higher side is specified, the gradation conversion means performs gradation conversion so that the reference plane becomes the lower limit of the distance range of the low gradation distance image, and the lower side is specified. In this case, the gradation conversion means performs gradation conversion so that the reference plane becomes the upper limit of the distance range of the low gradation distance image. The gradation conversion can be performed so that the reference plane is in the middle of the distance range of the low gradation distance image.

Furthermore, according to the three-dimensional image processing method according to the seventh aspect, a three-dimensional image processing method for acquiring a distance image including height information of an inspection object and performing image processing based on the distance image a step of projecting incident light from an oblique direction with respect to an optical axis of an imaging means as structured illumination of a predetermined projection pattern by a projection means; a step of acquiring the reflected light and capturing a plurality of pattern projection images by an imaging means; and a step of generating a distance image by a distance image generation means based on the plurality of pattern projection images captured by the imaging means. , as a gradation conversion parameter constituting a gradation conversion condition for gradation conversion of the distance image generated by the distance image generating means to a distance image having a lower gradation number than the distance image, setting a reference plane that serves as a reference for performing the tone conversion; and using the height of the set reference plane as a reference, converting the distance image to a number of tones lower than that of the distance image; a step of tone-converting the height information of the distance image into a low-tone distance image in which the grayscale value of the image is replaced ; prompting to specify one of the X direction, the Y direction, or the XY direction; At the time of conversion, a curved surface from which high-frequency components are removed from the distance image is calculated, and the calculated curved surface can be used as a reference surface for gradation conversion into a low-gradation distance image.

Furthermore, according to the three-dimensional image processing method according to the eighth aspect, in addition to any of the above, when performing the tone conversion, the distance image is once reduced, filtered, and enlarged to form a free-form surface image. By taking the difference between the free-form surface image and the distance image, the shape change of the distance image is extracted. When the distance image is reduced, the local shape change is extracted in the gradation conversion. Reduction can be performed only in the direction designated by the extraction direction input means for designating the direction of extraction. As a result, the distance image is reduced in a direction other than the designated direction without reducing in the direction designated by the extraction direction input means, leaving the height information in the designated direction in the free-form surface image. By taking the difference from this, it is possible to eliminate the gradual shape change in the direction and extract only the necessary shape.
Furthermore, according to the three-dimensional image processing method according to the ninth aspect, in addition to any of the above, the height information to be extracted based on the reference plane at the time of the gradation conversion is the reference plane a step of prompting the user to specify one of a higher side, a lower side, or both higher and lower than, and when the higher side is specified, the reference plane is the lower limit of the distance range of the low gradation distance image. When the lower side is designated, the gradation conversion is performed so that the reference plane becomes the upper limit of the distance range of the low gradation distance image, and when both the high and low are designated, The gradation conversion can be performed so that the reference plane is in the middle of the distance range of the low gradation distance image.
Furthermore, according to the three-dimensional image processing method according to the tenth aspect, a distance image including height information of an inspection object is acquired, and image processing can be performed based on the distance image. An image processing apparatus comprising imaging means for capturing an image of an object to be inspected, distance image generating means capable of generating a distance image based on the image captured by the imaging means, and generating by the distance image generating means. a reference plane setting means for setting a reference plane that serves as a reference when tone-converting the obtained distance image into a distance image having a number of tones lower than the number of tones of the distance image; Using the height of the reference plane set in 1 as a reference, the distance image is obtained by replacing the height information of the distance image with the grayscale value of the image, with the number of gradations lower than the number of gradations of the distance image. gradation conversion means for converting gradation into a gradation distance image; and extraction height setting means for designating height information to be extracted based on the reference plane when the gradation conversion is performed by the gradation conversion means. and inspection execution means for executing a predetermined inspection process on the low-gradation distance image that has been tone-converted by the tone conversion means, wherein the reference plane setting means corresponds to the distance image generation means. A curved surface excluding high-frequency components is calculated from the distance image generated by and the calculated curved surface is set as a reference plane. Either a higher side, a lower side, or both higher and lower than the reference plane can be specified, and when the higher side is specified, the extraction height setting means determines that the reference plane is the distance When tone conversion is performed so as to be the lower limit of the range and the lower side is specified, the extraction height setting means performs tone conversion so that the reference plane is the upper limit of the distance range of the low tone distance image. When both the height and the low are specified, the extraction height setting means can carry out tone conversion so that the reference plane is in the middle of the distance range of the low tone distance image.
Furthermore, according to the three-dimensional image processing method according to the eleventh aspect, the three-dimensional image processing method acquires a distance image including height information of an inspection object and performs image processing based on the distance image. a step of projecting incident light from an oblique direction with respect to an optical axis of an imaging means as structured illumination of a predetermined projection pattern by a projection means; a step of acquiring the reflected light and capturing a plurality of pattern projection images by an imaging means; and a step of generating a distance image by a distance image generation means based on the plurality of pattern projection images captured by the imaging means. , as a gradation conversion parameter constituting a gradation conversion condition for gradation conversion of the distance image generated by the distance image generating means to a distance image having a lower gradation number than the distance image, setting a reference plane that serves as a reference for performing the tone conversion; and using the height of the set reference plane as a reference, converting the distance image to a number of tones lower than that of the distance image; a step of gradation conversion into a low gradation distance image in which the height information of the distance image is replaced with the grayscale value of the image;
a step of prompting the user to specify any one of a higher side, a lower side, or both higher and lower than the reference plane as height information to be extracted based on the reference plane when performing the gradation conversion; performing a predetermined inspection process on the obtained low-gradation distance image, calculating a curved surface from which high-frequency components are removed from the distance image in the tone conversion, and using the calculated curved surface as a reference surface. When tone conversion is performed to a low tone distance image and the high side is specified, tone conversion is performed such that the reference plane is the lower limit of the distance range of the low tone distance image, and the low side is specified. gradation conversion is performed so that the reference plane is the upper limit of the distance range of the low gradation distance image, and when both the high and low are specified, the reference plane is the distance range of the low gradation distance image It is possible to perform gradation conversion so as to be in the middle of .

1 is a diagram showing a system configuration example of a 3D image processing system including an image processing device according to an embodiment of the present invention; FIG. FIG. 10 is a diagram showing a system configuration example of a 3D image processing system according to a modification of the present invention; FIG. 5 is a schematic diagram showing the hardware configuration of a 3D image processing apparatus according to Embodiment 2 of the present invention; 4A is a schematic diagram showing a head portion of a three-dimensional image processing apparatus according to Embodiment 3 of the present invention, and FIG. 4B is a schematic diagram showing a head portion of a three-dimensional image processing apparatus according to Embodiment 4 of the present invention. FIG. 10 is a block diagram showing a three-dimensional image processing device according to Embodiment 3 of the present invention; FIG. 6 is a block diagram showing a controller unit of FIG. 5; 4 is a flowchart showing processing operations of the image processing apparatus according to the embodiment; 10 is a flow chart showing a procedure of static conversion at the time of setting; FIG. 10 is an image diagram showing an initial screen to which "imaging" is added in the three-dimensional image processing program; FIG. 11 is an image diagram showing a state in which image registration is selected in the imaging setting menu; FIG. 10 is an image diagram showing an example of an image registration screen; FIG. 10 is an image diagram showing an example of a screen during registration of a distance image; FIG. 10 is an image diagram showing an example of a screen during registration of a luminance image; FIG. 11 is an image diagram showing an imaging setting screen; FIG. 11 is an image diagram showing an imaging enable setting screen; FIG. 10 is an image diagram showing a three-dimensional measurement setting screen; FIG. 4 is an image diagram showing selectable types of preprocessing. FIG. 10 is an image diagram showing options that can be set in the non-measurable standard setting field; FIG. 11 is an image diagram showing an example in which "none" is selected in the non-measurable standard setting column; FIG. 11 is an image diagram showing an example in which "low" is selected in the non-measurable standard setting column; FIG. 11 is an image diagram showing an example in which "medium" is selected in the non-measurable standard setting column; FIG. 11 is an image diagram showing an example in which "high" is selected in the non-measurable standard setting column; FIG. 10 is an image diagram showing options that can be set in the equal interval processing setting field; FIG. 11 is an image diagram showing an example in which equal interval processing is turned on and the 'display image' selection field is set to 'height image'; FIG. 10 is an image diagram showing an example in which equal interval processing is turned on and the “display image” selection field is set to “grayscale image”; FIG. 10 is an image diagram showing an example of displaying a distance image with equal interval processing turned off; FIG. 10 is an image diagram showing an example of displaying a luminance image with equal interval processing turned off; FIG. 10 is an image diagram showing options that can be set in the space code setting field; FIG. 10 is an image diagram showing a state in which a "height image" is selected in the "display image" selection field and a distance image is displayed in the second image display area. FIG. 11 is an image diagram showing a state in which a “grayscale image” is selected in the “display image” selection field and a luminance image is displayed in the second image display area; FIG. 10 is an image diagram showing an example in which OFF is selected in the space code setting field; It shows a state in which a "grayscale image" is displayed in the "display image" selection column. FIG. 5 is an image diagram showing options that can be set in a projector selection setting field; FIG. 11 is an image diagram showing an example in which "1" is selected in the projector selection setting field; FIG. 11 is an image diagram showing an example in which "2" is selected in the projector selection setting field; FIG. 11 is an image diagram showing an example in which "1+2" is selected in the projector selection setting field; FIG. 10 is an image diagram showing options that can be set in the “display image” selection field; FIG. 10 is an image diagram showing an example of displaying a pattern projection image of the first projector in the second image display area by selecting "stripe projection-projector 1" in the "display image" selection column. FIG. 10 is an image diagram showing an example of displaying a pattern projection image of the second projector in the second image display area by selecting "stripe projection-projector 2" in the "display image" selection field. FIG. 10 is an image diagram showing a three-dimensional measurement setting screen in which the shutter speed setting field is set to "1/15"; FIG. 10 is an image diagram showing a three-dimensional measurement setting screen in which the shutter speed setting field is set to "1/30"; FIG. 11 is an image diagram showing a three-dimensional measurement setting screen in which the grayscale range setting field is set to "normal (0)"; FIG. 10 is an image diagram showing a three-dimensional measurement setting screen in which the grayscale range setting field is set to “high (1)”; An image showing how the "height measurement" processing unit is added from the state in Fig. 9 45 is an image diagram showing an initial screen to which a "height measurement" processing unit has been added through FIG. 44; FIG. It is an image figure which shows a height measurement setting screen. FIG. 11 is an image diagram showing an inspection target area setting screen; FIG. 48 is an image diagram showing a state in which the drop-down menu of the "measurement area" setting column in FIG. 47 is displayed; FIG. 11 is an image diagram showing a state in which "rotated rectangle" is selected in the "measurement area" setting field; FIG. 10 is an image diagram showing a measurement area editing screen; FIG. 10 is an image diagram showing a state in which "circumference" is selected in the "measurement area" setting field of the measurement area edit screen; FIG. 11 is an image diagram showing an initial screen to which a second “height measurement” processing unit is added; FIG. 11 is an image diagram showing a state in which a “rotated rectangle” is set as a measurement area; FIG. 10 is an image diagram showing a state in which a “rotated rectangle” is set as a measurement area; FIG. 10 is an image diagram showing a state in which a “rotated rectangle” is set as a measurement area; FIG. 10 is an image diagram showing a state in which a "numerical operation" processing unit is to be added; FIG. 10 is an image diagram showing an initial screen to which a “numerical calculation” processing unit has been added; FIG. 10 is an image diagram showing a state in which a numerical calculation edit screen is displayed; FIG. 59 is an image diagram showing a state in which an arithmetic expression is input to the numerical calculation edit screen of FIG. 58; FIG. 10 is an image diagram showing an initial screen on which a “numerical calculation” processing unit is set; FIG. 11 is an image diagram showing an initial screen to which an "area" processing unit has been added; FIG. 11 is an image diagram showing an area setting screen; FIG. 11 is an image diagram showing an area setting screen for setting details of a rotated rectangle; FIG. 11 is an image diagram showing an area setting screen on which a rotated rectangle is set; FIG. 11 is an image diagram showing a height extraction selection screen; FIG. 11 is an image diagram showing a GUI of a one-point designation screen; FIG. 11 is an image diagram showing a GUI of a one-point designation screen; FIG. 11 is an image diagram showing a GUI of a one-point designation screen; FIG. 69A is an image diagram showing the profile of the input image, and FIG. 69B is an image diagram showing the profile of the low tone distance image obtained by tone-converting the input image of FIG. 69A. FIG. 69 is an image diagram showing a state in which the gain is increased from the state of FIG. 68; FIG. 71 is an image diagram showing a state in which the gain is lowered from the state of FIG. 70; FIG. 72 is an image diagram showing a state in which detailed setting is selected in FIG. 71; FIG. 11 is an image diagram showing a GUI of an emphasis method detail setting screen for specifying one point; FIG. 74 is an image diagram showing a state in which a drop-down list of "extraction height" setting fields is displayed from the state of FIG. 73; 75A is an image diagram showing the profile of the input image, FIG. 75B is an image diagram showing the input image of FIG. 75A with reference to the reference plane, and FIG. 75C is an image diagram showing the profile of the low gradation distance image obtained by gradation conversion of FIG. 75B is. 76A is a luminance image, FIG. 76B is a high gradation distance image, FIG. 76C is a low gradation distance image obtained by gradation conversion of FIG. 76B, FIG. 76D is a low gradation distance image with a higher gain than FIG. 76C, and FIG. 76E 76F is an image diagram showing a low gradation distance image in which noise removal is higher than that in FIG. 76D, and a low gradation distance image in which the "extraction height" is set on the higher side in FIG. 76E. 77A is an image diagram showing the profile of the input image, and FIG. 77B is an image diagram showing the profile of the low gradation distance image obtained by gradation conversion of the input image of FIG. 77A by setting the "extraction height" on the high side. FIG. 10 is an image diagram showing a state in which a gradation-converted image is displayed; FIG. 79A is a perspective view showing an example of a workpiece for which the method of setting a reference plane by specifying one point is effective, and FIG. 79B is an image diagram of a low gradation distance image obtained by converting the gradation of the distance image captured in FIG. 79A. FIG. 11 is an image diagram showing a GUI of a height extraction selection screen; FIG. 11 is an image diagram showing a GUI of a three-point designation screen; FIG. 82 is an image diagram showing a state in which the first point is designated by the height extracting means from the state of FIG. 81; FIG. 83 is an image diagram showing a state in which a second point is further specified from the state in FIG. 82; FIG. 84 is an image diagram showing a state in which a third point is further specified from the state in FIG. 83; FIG. 11 is an image diagram showing a GUI of a detailed setting screen for specifying three points; FIG. 86A is a perspective view showing an example of a workpiece for which the method of setting a reference plane by specifying three points is effective; FIG. 86B is an image diagram of a low-gradation distance image obtained by converting the gradation of the distance image captured in FIG. 86A; FIG. 86B is an image diagram of a binarized image, and FIG. 86D is an image diagram of a binarized image obtained by specifying one point when the workpiece in FIG. 86A has an inclination. FIG. 87A is a perspective view showing an example of another workpiece for which the method of setting a reference plane by specifying three points is effective, and FIG. 87B is an image diagram of a low gradation distance image obtained by converting the gradation of the distance image captured in FIG. 87A. 87C is an image diagram of a binarized image of FIG. 87B, and FIG. 87D is an image diagram of a binarized image obtained by specifying one point when the workpiece in FIG. 87A has an inclination. FIG. 11 is an image diagram showing a GUI of a height dynamic extraction setting screen; FIG. 11 is an image diagram showing a drop-down box in a “calculation method” selection field; FIG. 11 is an image diagram showing an average height reference setting screen; FIG. 11 is an image diagram showing a mask area setting screen; FIG. 11 is an image diagram showing a plane reference detail setting screen; FIG. 93A is a perspective view showing an example of a workpiece for which the method of setting the reference plane based on the average height is effective, and FIG. 93B is an image diagram of a low gradation distance image obtained by converting the gradation of the distance image captured in FIG. 93A. FIG. 11 is an image diagram showing a plane reference detail setting screen; 95 is an image diagram showing details of an invalid pixel designation field on the screen of FIG. 94; FIG. FIG. 10 is an image diagram showing a free-form surface reference setting screen; FIG. 97 is an image diagram showing a state in which the numerical value in the "extraction size" designation field in FIG. 96 is increased. FIG. 98A is a perspective view showing an example of a workpiece for which the method of setting a reference plane based on a free-form surface reference is effective, and FIG. 98B is an image diagram of a low-gradation distance image obtained by converting the gradation of the distance image captured in FIG. 98A. FIG. 91 is an image diagram showing a state in which an extraction area setting dialog is displayed from the state of FIG. 90; 99. FIG. 99 is an image diagram showing a state in which "rectangle" is selected in the extraction area selection column of FIG. FIG. 101 is an image diagram showing a state in which an extraction area edit dialog is displayed from the state in FIG. 100; 99. FIG. 99 is an image diagram showing a state in which "circle" is selected in the mask area selection column of FIG. FIG. 103 is an image diagram showing a state in which the mask area edit dialog is displayed from the state in FIG. 102; FIG. 10 is an image diagram showing a state in which height extraction is set in the "area" processing unit; FIG. 11 is an image diagram showing a filter processing setting screen; FIG. 11 is an image diagram showing a binarization level setting screen; FIG. 10 is an image diagram showing a state in which filter processing is set; FIG. 11 is an image diagram showing a determination condition setting screen; FIG. 10 is an image diagram showing a state in which determination conditions are set; FIG. 10 is an image diagram showing an initial screen with the addition of a “blob” processing unit; FIG. 10 is an image diagram showing how filtering is set in the “blob” processing unit; FIG. 10 is an image diagram showing how detection conditions are set in a “blob” processing unit; FIG. 10 is an image diagram showing how a determination condition is set in a “blob” processing unit; FIG. 11 is an image diagram showing a state in which a "color inspection" processing unit is to be added to the initial screen; FIG. 10 is an image diagram showing how a circular area is set in the “color inspection” processing unit; FIG. 11 is an image diagram showing a state in which a circle area is set in the "color inspection" processing unit; FIG. 10 is an image diagram showing a state in which density average is set in the "color inspection" processing unit; FIG. 11 is an image diagram showing an initial screen on which a "color inspection" processing unit is set; 10 is a flow chart showing the flow of processing during operation in the head unit of the 3D image processing apparatus according to Embodiment 3; 14 is a flow chart showing the flow of processing during operation in the head unit of the 3D image processing apparatus according to Embodiment 4; 4 is a flowchart showing a tone conversion method according to Embodiment 1; 10 is a flow chart showing the flow of processing during operation in the controller section of the 3D image processing apparatus according to Embodiment 3; FIG. 4 is an image diagram showing an initial screen of a three-dimensional image processing program; FIG. 4 is an image diagram showing a state in which a search target area is set on a luminance image; FIG. 4 is an image diagram showing a state in which a plurality of inspection target areas are set on a distance image; FIG. 126 is an image diagram showing a state in which the distance image in FIG. 125 is enlarged; FIG. 4 is an image diagram showing a state in which height measurement is executed by a three-dimensional image processing program; FIG. 4 is a data flow diagram for generating a range image by combining the phase shift method and the spatial encoding method; FIG. 4 is a data flow diagram for generating a range image using only the phase shift method without using the spatial encoding method; FIG. 5 is a data flow diagram showing a procedure for obtaining a Z image by turning off XY equal pitch conversion; It is a figure which shows the example which outputs point cloud data. FIG. 10 is a flow chart showing a procedure of static conversion during operation; FIG. 10 is a flow chart showing a procedure of dynamic conversion during operation; 1 is a perspective view showing a workpiece to be inspected; FIG. FIG. 4 is a schematic diagram showing a method of preparing a plurality of gradation conversion parameter sets in advance; Figure 136 is a flow chart showing the steps of the method of Figure 135; FIG. 10 is a perspective view showing an example of a workpiece with almost no height variation; FIG. 138A is a perspective view showing an example of a work whose height variation is desired to be detected, and FIG. 138B is an image diagram of a low gradation distance image obtained by gradation conversion of the distance image captured in FIG. 138A. FIG. 10 is a perspective view showing an example of a workpiece for which specification of a reference plane by dynamic conversion (average height reference) is effective; FIG. 10 is a perspective view showing an example of a workpiece for which specification of a reference plane by dynamic transformation (plane reference) is effective; FIG. 11 is a perspective view showing an example of a workpiece for which designation of a reference plane by dynamic conversion (free-form surface reference) is effective; 4 is a flow chart showing a procedure for repeating adjustment of tone conversion parameters until an appropriate image is obtained; FIG. 143 is a flow chart showing the procedure in FIG. 142 omitting the judgment as to whether or not the image is appropriate; 5 is a flowchart showing a specific procedure for tone conversion parameter adjustment; FIG. 145A shows the appearance of the work, FIG. 145B shows the image of the distance image obtained from the work of FIG. 145A, and FIG. 145D is an image showing a state in which an inspection target area is set for image inspection processing in the distance image of FIG. 145B, and FIG. 145E is a low gradation distance image obtained by performing gradation conversion on the distance image of FIG. 145D. 1 is a diagram showing an image of each. 4 is a flowchart showing a procedure for setting; 145B is an image diagram showing a screen for setting an "area" processing unit for the work of FIG. 145A; FIG. 148 is an image diagram showing a screen for setting conditions for height extraction in FIG. 147; FIG. 4 is a flow chart showing a procedure during operation; FIG. 150 is a flow chart showing a procedure when tone conversion is performed in the inspection process of FIG. 149; FIG. FIG. 150 is a flow chart showing a procedure when gradation conversion is not performed in the inspection processing of FIG. 149; FIG. 4 is a flowchart showing a procedure for setting inspection processing conditions; FIG. 11 is an image diagram showing an image setting screen; FIG. 10 is an image diagram showing a state in which an image variable selection screen is called from which a luminance image or a distance image can be selected; FIG. 10 is an image diagram showing a state in which an image variable selection screen that allows selection of only a distance image is called. 5 is a flow chart showing a procedure for selecting an image, selecting an inspection process, and setting inspection process conditions. FIG. 4 is a schematic diagram showing a state in which a luminance image and a range image are acquired; FIG. 158 is a schematic diagram showing inspection processing tools that can be set when a luminance image is selected in FIG. 157; FIG. 158 is a schematic diagram showing inspection processing tools that can be set when a distance image is selected in FIG. 157; It is a schematic diagram which shows a mode that a range image is imaged by a triangulation ranging method. 4 is a flowchart showing a procedure for fitting height information distributed within a free-form surface target area; FIG. 3 is an image diagram showing a distance image of an inspection object having a uniform shape in the vertical direction; FIG. 163 is an image diagram showing a low gradation distance image extracted in the XY direction from the distance image of FIG. 162 and subjected to gradation conversion. FIG. 11 is an image diagram showing another example of the free-form surface reference setting screen; 165 is an image diagram showing a detailed setting screen of the free-form surface reference setting screen of FIG. 164; FIG. FIG. 163 is an image diagram showing a low gradation distance image extracted in the Y direction from the distance image of FIG. 162 and subjected to gradation conversion; FIG. 11 is an image diagram showing a distance image of another inspection object having a uniform shape in the vertical direction; FIG. 168 is an image diagram showing a low gradation distance image obtained by extracting in the XY directions from the distance image of FIG. 167 and converting the gradation. FIG. 168 is an image diagram showing a low gradation distance image extracted in the Y direction from the distance image of FIG. 167 and subjected to gradation conversion;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiments shown below are examples of a three-dimensional image processing apparatus and a three-dimensional image processing method for embodying the technical idea of the present invention. The original image processing method is not specified as follows. In addition, this specification does not specify the members shown in the claims as the members of the embodiment. Unless otherwise specified, the dimensions, materials, shapes, relative arrangements, etc. of components described in the embodiments are not intended to limit the scope of the present invention, but are merely explanations. Just an example. Note that the sizes and positional relationships of members shown in each drawing may be exaggerated for clarity of explanation. Furthermore, in the following description, the same names and symbols indicate the same or homogeneous members, and detailed description thereof will be omitted as appropriate. Furthermore, each of the elements constituting the present invention may be configured with the same member so that a single member may serve as a plurality of elements, or conversely, the function of one member may be performed by a plurality of members. It can also be realized by sharing.

In this specification, the term "distance image (height image)" is used to mean an image containing height information. The synthetic image is also used in the sense that it is included in the range image. Further, in this specification, the display form of the distance image is not limited to two-dimensional display, and includes three-dimensional display.
(Embodiment 1)

FIG. 1 shows the configuration of a three-dimensional image processing apparatus according to Embodiment 1 of the present invention. This three-dimensional image processing apparatus 100 includes a head section 1 and a controller section 2 . The head unit 1 includes a light projecting unit 20 for illuminating an inspection object (work) W, an imaging unit 10 for capturing an image of the work W, and a head-side communication unit 36 for connecting with the controller unit 2 .

On the other hand, the controller unit 2 executes measurement processing such as edge detection and area calculation based on the captured image. Further, to the controller unit 2, display means 4 such as a liquid crystal panel, input means 3 such as a console for the user to perform various operations on the display means 4, PLC (Programmable Logic Controller), etc. can be detachably connected.

In the three-dimensional image processing apparatus 100 described above, the light projecting means 20 of the head unit 1 projects measurement light onto the work W, and the measurement light is incident on the work W and the reflected light is captured by the imaging means 10 in a pattern. Capture as a projection image. Also, a distance image is generated based on the pattern projection image, and further this distance image is converted into a low gradation distance image in which the height information of each pixel is replaced with luminance. The controller unit 2 executes measurement processing such as edge detection and area calculation based on the converted low gradation distance image.

The work W, which is an object to be inspected, is, for example, an article that is sequentially conveyed on a line, and is moving or stationary. The moving work includes parallel movement by a conveyor or the like as well as rotating work.
(Light projecting means 20)

The light projecting means 20 is used as illumination for illuminating the work W to generate a distance image. Therefore, the light projecting means 20 can be, for example, a light projector that projects linear laser light onto the work, or a sinusoidal striped pattern onto the work, depending on the light section method or pattern projection method for acquiring the distance image. can be a pattern projector or the like for projecting the . In addition to the light projecting means, a general illumination device for performing bright field illumination or dark field illumination may be separately provided. Alternatively, the light projecting means 20 can also function as a general illumination device.

The controller unit 2 executes image processing using the distance image data acquired from the head unit 1, and outputs a determination signal as a signal indicating the quality of the workpiece to a control device such as a PLC 70 connected externally. do.

The imaging means 10 images the workpiece based on a control signal input from the PLC 70, for example, an imaging trigger signal that defines the timing of capturing image data from the imaging means 10. FIG.

The display means 4 is a display device for displaying image data obtained by imaging a workpiece and results of measurement processing using the image data. In general, the user can confirm the operating state of the controller section 2 by viewing the display means 4 . The input means 3 is an input device for moving the focus position on the display means 4 and selecting menu items. When a touch panel is used as the display means 4, the display means and the input means can be used together.

Also, the controller section 2 can be connected to a personal computer PC for generating a control program for the controller section 2 . A three-dimensional image processing program for setting three-dimensional image processing can be installed in the personal computer PC, and various settings for processing performed by the controller unit 2 can be performed. Alternatively, software running on this personal computer PC can generate a processing order program that defines the processing order of image processing. In the controller unit 2, each image processing is sequentially executed along the processing order. The personal computer PC and the controller unit 2 are connected via a communication network. 2. Conversely, it is also possible to import the processing order program, layout information, etc. from the controller unit 2 and edit them on the personal computer PC. Note that this processing order program may be generated not only by the personal computer PC but also by the controller unit 2 .
(Modification)

Although dedicated hardware is constructed as the controller unit 2 in the above example, the present invention is not limited to this configuration. For example, like the three-dimensional image processing apparatus 100' according to the modification shown in FIG. It can also be used by connecting to the head unit 1. This three-dimensional image processing apparatus performs image processing and other necessary settings using a three-dimensional image processing program, and then performs image processing on the low-gradation distance image according to the pattern projection image captured by the head unit 1. inspection.
(Head side communication means 36)

Accordingly, an interface that can be connected to either the dedicated controller section 2 or a personal computer functioning as the controller section 2 can be provided as the head-side communication means 36 on the head section 1 side. For example, the head unit 1 is provided with a controller connection interface 36A for connecting with the controller unit 2 as shown in FIG. A PC connection interface 36B is provided. Further, by making such an interface replaceable as a unit, other configurations of the head section can be shared to some extent, and the common head section can be connected to either the controller section or the personal computer. Alternatively, one head-side communication means having an interface connectable to any of a dedicated controller section 2 and a personal computer may be provided. For such an interface, existing communication standards such as Ethernet (trade name), USB, RS-232C, etc. can be used. Also, a dedicated communication method may be used instead of a standardized or general-purpose communication method.
(PC connection mode)

Furthermore, the three-dimensional image processing program can be provided with a PC connection mode for setting when a personal computer is used as the controller unit 2 ′ connected to the head unit 1 . That is, by changing the items that can be set and the setting contents depending on whether the controller unit is dedicated hardware or a personal computer, it is possible to appropriately perform settings related to 3D image processing in any case. It becomes possible. Furthermore, by installing a viewer program for checking the operation of the head unit 1 and having a simple measurement function in the personal computer functioning as the controller unit 2', the operation and functions of the connected head unit can be checked. You may do so.

The "distance image" obtained by using the imaging means 10 and the light projecting means 20 shown in FIG. refers to an image in which changes In other words, it can be said to be an image whose grayscale value is determined based on the distance from the imaging means 10 to the work W, or a multivalued image having grayscale values corresponding to the distance to the work W. It can also be said to be a multi-valued image having grayscale values corresponding to heights. Furthermore, it can also be said that the luminance image is a multivalued image obtained by converting the distance from the imaging means 10 into a grayscale value for each pixel of the luminance image.

There are roughly two methods for generating a depth image. One is a passive method (passive measurement method) that generates a depth image using an image captured under lighting conditions for obtaining a normal image. , and the other is an active method (active measurement method) in which a distance image is generated by actively irradiating light for measuring in the height direction. A typical passive method is the stereo measurement method. Since a range image can be generated simply by preparing two imaging means 10 and arranging these two cameras in a predetermined positional relationship, a general image processing system for generating a luminance image can be used. The distance image can be generated by using the system, and the system construction cost can be suppressed. However, in the stereo measurement method, it is necessary to determine to which point in the image obtained by the other camera a point in the image obtained by one camera corresponds. There is a problem that it takes time to In addition, since the measurement positions are only corresponding points and not all pixels, this point is also not suitable for speeding up the appearance inspection.

On the other hand, typical active methods are the light section method and the pattern projection method. The light section method is a method of optically measuring the shape and roughness of the surface, and projects a thin slit image at an angle of about 45° to the surface of the inspection object, A method of observing from the side and a method of observing the shape of the cutting line generated on the surface by irradiating the object to be inspected with a thin slit-like beam of light so as to cut it are known. In the light-section method, only one line of profile (cut plane) can be obtained. Can also be configured.

The light-section method here is the above-mentioned stereo measurement method, in which one of the cameras is replaced with a light projector, a line-shaped laser beam is projected onto the workpiece, and a line-shaped laser beam is emitted according to the shape of the object surface. The three-dimensional shape of the workpiece is restored from the degree of distortion of the image. Since the light section method does not require determination of corresponding points, stable measurement is possible. However, since only one line can be measured in one measurement, the object or camera must be scanned to obtain measurement values for all pixels. On the other hand, in the pattern projection method, multiple images are captured by shifting the shape and phase of a predetermined pattern projected onto the workpiece, and the captured multiple images are analyzed to restore the three-dimensional shape of the workpiece. It is something to do. There are several types of pattern projection methods. Multiple images (at least 3 images) are taken by shifting the phase of the sinusoidal striped pattern, and the phase of the sinusoidal wave is obtained for each pixel from the multiple images. Using the obtained phase to find the 3D coordinates on the workpiece surface, and using a kind of spatial frequency undulation phenomenon that occurs when two regular patterns are synthesized, the 3D shape is restored. Moire photography method, the pattern itself projected on the work is changed for each shooting, for example, the stripe width becomes half the screen, 1/4, 1/8, etc. at a black-and-white duty ratio of 50%. A spatial encoding method in which patterns are sequentially projected, pattern projection images are taken for each pattern, and the absolute phase of the height of the workpiece is obtained. A typical example is the multi-slit method, in which the pattern is moved at a pitch narrower than the slit cycle and the images are taken multiple times.

The 3D image processing apparatus 100 according to this embodiment generates a range image by combining the phase shift method and the spatial encoding method described above. Thereby, a distance image can be generated without relatively moving the workpiece or the head. The present invention is not limited to generating the range image by the phase shift method and the spatial encoding method, and the range image may be generated by other methods. In addition, any conceivable method for generating a range image may be adopted, such as a method other than the above-described methods, such as an optical radar method (time of flight), a focusing method, a confocal method, and a white light interferometry. do not have.

The arrangement layout of the imaging means 10 and the light projecting means 20 shown in FIG. is obliquely arranged so that the imaging means 10 is held in a vertical posture. In this way, by tilting the light projection direction and the imaging direction instead of matching them, it is possible to capture a pattern projection image that captures shadows caused by the unevenness of the surface shape of the workpiece W.
(Embodiment 2)

However, the present invention is not limited to this arrangement example. Alternatively, the light projecting means 20 side may be held in a vertical position. With the head unit 1B arranged in this manner, a pattern projection image capturing the shadow of the workpiece W can be captured by similarly tilting the light projection direction and the imaging direction.
(Embodiment 3)

Furthermore, one or both of the light projecting means and the imaging means can be arranged in plurality. For example, as in the three-dimensional image processing apparatus 300 shown in FIG. 4A as Embodiment 3, while the imaging means 10 is held in a vertical posture with respect to the work W, two light projecting means 20 are arranged around the imaging means 10. It can also be configured as a head portion 1C that is arranged on both sides and projects light from the left and right respectively. By capturing the pattern projection images with different light projection directions in this way, a state where the pattern projection images cannot be partially captured occurs, for example, the shadow pattern is hidden by the workpiece W itself when the light is projected from one direction. Inaccurate or impossible height measurements can be reduced. In particular, by arranging the light projecting means 20 so as to project light from a direction opposite to the work (for example, right and left or front and back), the possibility that the work itself is blocked and cannot be imaged can be greatly reduced.
(Embodiment 4)

In addition, in the above example, the configuration in which one image capturing means and two light projecting means are provided has been described. Such an example is shown in FIG. 4B as a three-dimensional image processing apparatus 400 according to the fourth embodiment. In the head unit 1D shown in this example, the light projecting means 20 is held in a vertical posture with respect to the work W, and the imaging means 10 on the left and right sides thereof are arranged in a tilted posture with respect to the work W in the drawing. Even with this configuration, the workpiece W can be imaged from different angles of inclination, so that it is possible to prevent the pattern projection image from becoming partially difficult to be imaged, as in the case of the third embodiment. Also, with this method, two pattern projection images can be captured simultaneously by one light projection, so there is an advantage that the processing time can be shortened.

On the other hand, even if the same workpiece is imaged from different angles with two imaging means, the parts and fields of view being imaged are different, so it is necessary to match the position of each pixel, and there is a possibility that errors will occur. be. On the other hand, according to the above-described third embodiment, since the image capturing means is shared, the image of the same field of view can be captured regardless of which light projecting means projects the measurement light. It is possible to eliminate the need for such integration work, avoid the occurrence of errors associated with the integration work, and obtain the advantage of simplifying the processing.

In the above example, an example in which the imaging means 10 and the light projecting means 20 are integrally configured in each head portion has been described, but the present invention is not limited to this configuration. For example, the imaging means 10 and the light projecting means 20 can be made into a head section composed of separate members. Also, it is possible to provide three or more image capturing means and light projecting means.
(Block Diagram)

Next, FIG. 5 shows a block diagram showing the configuration of a three-dimensional image processing apparatus 300 according to Embodiment 3 of the present invention. The three-dimensional image processing apparatus 300 includes a head section 1 and a controller section 2, as shown in FIG.
(Head part 1)

The head unit 1 includes a light projecting unit 20, an imaging unit 10, a head-side control unit 30, a head-side computing unit 31, a storage unit 38, a head-side communication unit 36, and the like. The light projecting means 20 includes a measurement light source 21 , a pattern generator 22 and a plurality of lenses 23 , 24 and 25 . The imaging means 10 includes a camera and a plurality of lenses (not shown).
(Light projecting means 20)

The light projecting means 20 is a member for projecting incident light from an oblique direction with respect to the optical axis of the imaging means as structured illumination of a predetermined projection pattern. A projector can be used as the light projecting means 20, and includes a lens, a pattern generation unit 22, and the like, which are optical members. The light projecting means 20 is arranged obliquely above the position of the stationary or moving workpiece. Note that the head section 1 can also include a plurality of light projecting means 20 . In the example of FIG. 5, the head section 1 includes two light projecting means 20. In the example of FIG. Here, a first projector 20A (on the right side in FIG. 5) capable of irradiating the workpiece with illumination light for measurement from a first direction and a projector for measurement from a second direction different from the first direction. A second projector 20B (left side in FIG. 5) capable of emitting illumination light is arranged. The first projector 20A and the second projector 20B are arranged symmetrically with the optical axis of the imaging means 10 interposed therebetween. Measurement light is projected alternately from the first projector 20A and the second projector 20B onto the workpiece, and the pattern of each reflected light is imaged by the imaging means 10 .

A halogen lamp that emits white light, a white LED (light emitting diode) that emits white light, or the like can be used as the measurement light source 21 of each of the first projector 20A and the second projector 20B. The measurement light emitted from the measurement light source 21 is incident on the pattern generator 22 after being appropriately condensed by the lens.

Furthermore, in addition to the projection means for emitting the measurement light for acquiring the pattern projection image for generating the distance image, an observation illumination light source for capturing a normal optical image (brightness image) can be provided. . As an illumination light source for observation, a semiconductor laser (LD), a halogen light, an HID, or the like can be used in addition to an LED. In particular, when a device capable of color imaging is used as the imaging device, a white light source can be used as the illumination light source for observation.

The measurement light emitted from the measurement light source 21 is appropriately condensed by the lens 113 and then enters the pattern generator 112 . The pattern generation unit 22 can realize arbitrary pattern illumination. For example, by inverting the pattern according to the color of the workpiece or the background, such as black on white or white on black, an appropriate pattern that is easy to see or measure can be expressed. A DMD (digital micromirror device), for example, can be used for such a pattern generator 22 . A DMD can express an arbitrary pattern by turning on/off a minute mirror for each pixel. This makes it possible to easily irradiate a pattern in which white and black are reversed. By using a DMD for the pattern generation unit 22, arbitrary patterns can be easily generated, and mechanical pattern mask preparation and replacement work are not required, so there is an advantage that the device can be made compact and measurements can be made quickly. . In addition, the pattern generation unit 112 using the DMD can be used in the same manner as normal illumination by irradiating all illumination patterns with all pixels turned on, so it can also be used to capture a luminance image. The pattern generator 22 can also be an LCD (liquid crystal display), an LCOS (liquid crystal on silicon: reflective liquid crystal element), or a mask. The measurement light incident on the pattern generator 22 is converted into a preset pattern and a preset intensity (brightness) and emitted. The measurement light emitted by the pattern generator 22 is converted by a plurality of lenses into light having a diameter larger than the observable/measurable field of view of the imaging means 10, and then irradiated onto the workpiece.
(Imaging means 10)

The imaging means 10 is provided with a camera for capturing a plurality of pattern projection images by acquiring reflected light projected by the light projecting means 20 and reflected by the workpiece WK. A CCD, a CMOS, or the like can be used for such a camera. In this example, a monochrome CCD camera capable of obtaining high resolution is used. Needless to say, it is also possible to use a camera capable of capturing images in color. In addition to the pattern projection image, the image capturing means can also capture a normal luminance image.

The head-side control section 30 is a member for controlling the first projector 20A and the second projector 20B, which are the imaging means 10 and the light projecting means 20 . For example, the head unit side control unit 30 creates a light projection pattern for obtaining a pattern projection image by projecting the measurement light onto the work by the light projection unit 20 . As a result, the imaging means 10 projects a projection pattern for phase shift from the light projecting means 20 to capture a phase shift image, and projects a projection pattern for spatial encoding from the light projecting means 20 to produce a spatial code image. Take an image. In this manner, the head-side control unit 30 functions as light projection control means for controlling the light projection means so that the image pickup means 10 picks up the phase shift image and the space code image.

The head-side calculator 31 includes a filter processor 34 and a distance image generator 32 . A distance image generating means 32 generates a distance image based on the plurality of pattern projection images captured by the imaging means 10 .

The head-side storage means 38 is a member for holding various settings, images, etc., and a storage element such as a semiconductor memory or a hard disk can be used. For example, it includes a luminance image storage unit 38b for holding the pattern projection image captured by the imaging unit 10 and a distance image storage unit 38a for holding the distance image generated by the distance image generation unit 32.

The head-side communication means 36 is a member for communicating with the controller section 2 . Here, it is connected to the controller side communication means 42 of the controller section 2 to perform data communication. For example, the distance image generated by the distance image generating means 32 is sent to the controller section 2 .
(Distance image generating means 32)

The distance image generation means 32 is a means for generating a distance image in which the grayscale value of each pixel changes according to the distance from the imaging means 10 for imaging the work WK to the work WK. For example, when the distance image is generated by the phase shift method, the head-side control unit 30 controls the light projecting unit 20 so as to project a sinusoidal fringe pattern onto the workpiece with a phase shift. The head-side control section 30 controls the imaging means 10 so as to pick up a plurality of images in which the phases of the wavy fringe pattern are shifted. Then, the head-side control unit 30 obtains the phase of the sine wave for each pixel from the plurality of images, and uses the obtained phase to generate the distance image.

When a range image is generated using the spatial encoding method, the space irradiated with light is divided into a large number of small spaces having a substantially fan-shaped cross section, and these small spaces are given a series of space code numbers. Therefore, even if the height of the work is high, in other words, even if the height difference is large, the height can be calculated from the space code number as long as the work is in the space irradiated with light. Therefore, it is possible to measure the shape of a tall workpiece over its entirety.

By generating the distance image on the head section side and sending it to the controller side in this way, the amount of data to be sent from the head section to the controller side can be reduced, and the delay in processing that may occur due to the transfer of a large amount of data can be reduced. can be avoided.

In this embodiment, the distance image generation processing is performed on the head unit 1 side, but the distance image generation processing may be performed on the controller unit 2 side, for example. In addition, the gradation conversion from the distance image to the low gradation distance image can be performed by the head unit as well as by the controller unit. In this case, the head-side computing section 31 implements the function of the gradation converting means.
(Controller section 2)

The controller section 2 also includes controller-side communication means 42 , controller-side control section, controller-side calculation section, controller-side storage means, inspection execution means 50 , and controller-side setting means 41 . The controller-side communication means 42 is connected to the head-side communication means 36 of the head section 1 to perform data communication. The controller-side control section is a member for controlling each member. The controller-side calculation unit implements the functions of the image processing unit 60 . The image processing unit 60 implements functions such as the image search means 64 and the gradation conversion means 46 .
(gradation conversion means)

The gradation conversion means 46 gradation-converts the high gradation distance image into a low gradation low gradation distance image based on the distance image (detailed procedure will be described later). As a result, a distance image having height information generated by the head unit can be expressed as a two-dimensional grayscale image that can be handled by existing equipment, thereby contributing to measurement processing and inspection processing. In addition, the distance image generation processing and the gradation conversion processing can be divided between the head unit and the controller unit, thereby providing the advantage of distributing the load. In addition to the generation of the distance image, the head section may also generate the low-gradation distance image. Such processing can be performed by the head-side computing unit. As a result, the load on the controller side is further reduced, and efficient operation becomes possible.

Further, the gradation conversion means preferably selects only a necessary portion of the distance image and performs gradation conversion instead of performing gradation conversion on the entire distance image. Specifically, only a portion corresponding to an inspection target area set in advance by an inspection target area setting means (details will be described later) is tone-converted. By doing so, the load required for tone conversion can be reduced by limiting the process of converting a multi-tone distance image into a low-tone distance image only for the inspection target area. This also contributes to shortening the processing time. That is, by shortening the processing time, it can be suitably used even in applications with limited processing time, such as FA inspection, and real-time processing is realized.

The controller-side storage means is a member for holding various settings and images, and a semiconductor storage element, hard disk, or the like can be used.

The controller-side setting means 41 is a member for performing various settings for the controller section, receives operations from the user via the input means 3 such as a console connected to the controller section, and sends necessary conditions and the like to the controller side. instruct. For example, the functions of a gradation conversion condition setting means 43, a reference plane setting means 44, a space encoding switching means 45, an interval equalization processing setting means 47, a light projection switching means 48, a shutter speed setting means 49 and the like are realized.

The reference plane setting means 44 sets the distance image received by the controller-side communication means 42 as a gradation conversion parameter constituting a gradation conversion condition when performing gradation conversion for converting the distance image into a two-dimensional low gradation distance image. , the reference plane for this gradation conversion is set based on the distance image. Using the reference plane set by the reference plane setting means 44 as a reference, the gradation conversion means 46 converts the distance image into a gradation number lower than the gradation number of the distance image, and converts the height information into the gradation value of the image. Gradation conversion is performed to the replaced low-gradation distance image.

The inspection execution means 50 performs a predetermined inspection process on the low tone distance image that has been tone-converted by the tone conversion means 46 .
(Hardware configuration)

Next, an example of hardware configuration of the controller section 2 is shown in the block diagram of FIG. The controller unit 2 shown in this figure performs numerical calculations and information processing based on various programs, and has a main control unit 51 that controls each hardware unit. The main control unit 51 stores, for example, a CPU as a central processing unit, a work memory such as a RAM that functions as a work area when the main control unit 51 executes various programs, a startup program, an initialization program, and the like. and a program memory such as a hardened ROM, flash ROM, or EEPROM.

Further, the controller unit 2 is connected to the head unit 1 including the imaging means 10 and the light projecting means 20, etc., and controls the light projecting means 20 to project the sinusoidal striped pattern onto the workpiece with a phase shift. , a controller-side connection unit 52 for capturing image data obtained by imaging with the imaging unit 10, an operation input unit 53 for inputting an operation signal from the input unit 3, and a display unit 4 such as a liquid crystal panel. A display control unit 54 composed of a display DSP or the like for displaying an image by using the display control unit 54, a communication unit 55 communicably connected to an external PLC 70, a personal computer PC or the like, a RAM 56 for holding temporary data, a setting A controller-side storage means 57 for storing contents, an auxiliary storage means 58 for holding data set by a three-dimensional image processing program installed in a personal computer PC, and executing measurement processing such as edge detection and area calculation. and an output unit 59 for outputting the results of predetermined inspections based on the processing results of the image processing unit 60 and the like. These pieces of hardware are communicably connected via an electrical communication path (wiring) such as a bus.

In the program memory in the main control unit 51, control for controlling each unit of the controller side connection unit 52, the operation input unit 53, the display control unit 54, the communication unit 55, and the image processing unit 60 by commands of the CPU, etc. program is stored. Further, the processing order program described above, that is, the processing order program generated in the personal computer PC and transferred from the personal computer PC is stored in the program memory.

The communication unit 55 functions as an interface (I/F) that receives an imaging trigger signal from the PLC 70 when a sensor (such as a photoelectric sensor) connected to the external PLC 70 receives a trigger input. It also functions as an interface (I/F) for receiving a three-dimensional image processing program transferred from the personal computer PC, layout information specifying the display mode of the display means 4, and the like.

Upon receiving the imaging trigger signal from the PLC 70 via the communication section 55 , the CPU of the main control section 51 sends an imaging instruction (command) to the controller-side connection section 52 . Also, based on the processing order program, it transmits a command for instructing the image processing to be executed to the image processing section 60 . As a device for generating the imaging trigger signal, instead of the PLC 70 , a trigger input sensor such as a photoelectric sensor may be directly connected to the communication unit 55 .

The operation input unit 53 functions as an interface (I/F) that receives operation signals from the input means 3 based on user's operations. The contents of the user's operation using the input means 3 are displayed on the display means 4 . For example, when a console is used as the input means 3, each part such as a cross key for moving the cursor displayed on the display means 4 up, down, left, or right, an enter button, or a cancel button can be arranged. By manipulating these components, the user can create a flow chart defining the processing order of image processing, edit parameter values for each image processing, and set a reference area on the display means 4. , the reference registration image can be edited.

The controller-side connection unit 52 takes in image data. Specifically, for example, when an imaging command for the imaging means 10 is received from the CPU, an image data capture signal is transmitted to the imaging means 10 . After the imaging is performed by the imaging means 10, the image data obtained by the imaging is captured. The captured image data is temporarily buffered (cached) and assigned to an image variable prepared in advance. The term "image variable" refers to a variable that is assigned as an input image to the corresponding image processing unit and serves as a reference for measurement processing and image display, unlike normal variables that handle numerical values.

The image processing unit 60 executes measurement processing on image data. Specifically, first, the controller-side connection unit 52 reads the image data from the frame buffer while referring to the image variables described above, and internally transfers the read image data to the memory in the image processing unit 60 . The image processing unit 60 then reads out the image data stored in the memory and executes measurement processing. The image processing section 60 also includes a gradation converting means 46, an abnormal point highlighting means 62, an image searching means 64, and the like.

The display control unit 54 transmits a control signal for displaying a predetermined image (video) to the display means 4 based on a display command (display command) sent from the CPU. For example, a control signal is transmitted to the display means 4 in order to display image data before or after measurement processing. The display control unit 54 also transmits a control signal for causing the display unit 4 to display the content of the user's operation using the input unit 3 .

The head section 1 and the controller section 2, which are configured by hardware as described above, are configured so that each means and function in FIG. 5 can be implemented by software using various programs. In this example, a three-dimensional image processing program is installed in the computer shown in FIG. 1, and settings necessary for three-dimensional image processing are performed.
(gradation conversion)

The three-dimensional image processing apparatus described above acquires a distance image of a workpiece, performs image processing on this distance image, and inspects the result. The three-dimensional image processing apparatus according to the present embodiment performs height inspection processing in which height information, which is the pixel value of a distance image, is used as it is to perform calculations. Two types of inspection can be performed: an image inspection process that performs calculations using . Here, in order to maintain the accuracy of height inspection processing, it is necessary to generate a multi-gradation distance image. On the other hand, existing hardware cannot perform image inspection processing on such a multi-gradation range image. Therefore, in order to perform image inspection processing using existing hardware, a multi-gradation distance image is subjected to gradation conversion to generate a low-gradation distance image.

However, if the height information of the multi-gradation distance image is directly converted into the low-gradation distance image, there is a problem that the accuracy of the height information is impaired. Many of the general images used for FA applications are monochrome images in which the grayscale value of each pixel is expressed in eight gradations. On the other hand, the distance image uses a high-gradation image such as a 16-gradation image. For this reason, when converting a multi-tone distance image into a low-tone distance image, a considerable amount of height information is lost, which affects the accuracy of inspection. On the other hand, if the number of gradations of an image handled by existing image processing is increased in order to improve accuracy, the introduction cost rises, the processing load increases, and the hurdles to use increase.

Therefore, upon such tone conversion, it is necessary to set conditions for tone conversion such that necessary height information is maintained. The method and procedure will be described in detail below.
(height inspection or image inspection)

First, processing operations for performing height inspection processing using the three-dimensional image processing apparatus will be described based on the flowchart of FIG. This three-dimensional image processing device includes a height inspection processing tool for performing height inspection on distance images and various image inspection processing for performing image inspection on existing luminance images as tools for performing calculation processing. with tools. Here, height inspection processing will be described.

First, a distance image is generated (step S71). Specifically, the distance image generating means 32 generates a distance image using the imaging means 10 and the light projecting means 20 . Next, a desired calculation process is selected (step S72). Here you select the tools you need for your calculations.

If the image inspection processing tool is selected, the process proceeds to step S73, and the high-gradation range image obtained in step S71 is subjected to gradation conversion processing to be converted into a low-gradation range image. As a result, even inspection processing tools provided in existing image processing apparatuses can handle low-gradation distance images. Note that the gradation conversion process is preferably performed only within the inspection target area set for the image inspection process, instead of being performed on the entire range image of high gradation.

On the other hand, if the height inspection tool is selected, the height information of the multi-tone distance image is used as it is, so the process proceeds to step S74 without tone conversion.

Furthermore, the inspection executing means 50 performs various calculation processes (step S74), and then determines whether or not the work is a non-defective product based on the calculation results (step S75). If the inspection execution means 50 determines that the workpiece is non-defective (step S75: YES), the determination signal output means 160 outputs an OK signal to the PLC 70 as a determination signal (step S76). If it is determined that the workpiece is not good, that is, is defective (step S75: NO), an NG signal is output to the PLC 70 as a determination signal (step S77).
(setting mode)

Next, an example of procedures in a setting mode for performing various settings for the three-dimensional image processing apparatus prior to execution of height inspection and image inspection will be described with reference to the flowchart of FIG. First, in step S81, an image for setting (image for setting) is selected. Here, an image to be inspected can be input in advance and stored as a registered image can be called up, or a new input image can be obtained and settings can be made for it. Here, an input image obtained by picking up a workpiece is registered as a registered image as an alternative to input images that are sequentially input during operation. Alternatively, a registered image that has been registered in advance may be called. Next, in step S82, the gradation conversion method is selected. Here, the user is prompted to select either static or dynamic conversion. Next, in step S83, the gradation conversion parameters are adjusted. Here, if static conversion is selected in step S82, tone conversion parameters are adjusted for the image acquired in step S81. A method of adjusting the tone conversion parameters will be described later. Note that the procedure described above is an example, and a different order is also possible. For example, acquisition of an image may be performed after selection of a gradation conversion method.
(Details of setting procedure)

Next, the details of the procedure for setting will be described. In the three-dimensional image processing apparatus, necessary settings are performed in the setting mode prior to the operation mode. Various setting means for performing such settings can be provided on the controller section 2 side, for example. For example, in the example of FIG. 1, a console, which is one form of the input means 3 connected to the controller section 2, can be used. Alternatively, or in addition to this, the three-dimensional image processing program installed in the personal computer connected to the controller unit 2 as described above can be made to realize the function of such setting means. Hereafter, using the three-dimensional image processing program installed in the personal computer shown in FIG. 1, the details of the procedure for making each setting will be explained using the user interface (GUI) screens of the three-dimensional image processing program shown in FIGS. will be explained based on In these GUI examples, the distance image is displayed as a "height image" and the luminance image is displayed as a "grayscale image".
(Registration process of distance image and luminance image)

First, a range image and a luminance image are registered. Here, the setting of the "imaging" processing unit 263 is performed from the initial screen 260 of the three-dimensional image processing program shown in FIG. Specifically, the button of the button 263 of the "imaging" processing unit is pressed. This switches to the imaging setting menu 269 of FIG.
(three-dimensional image processing program)

In the GUI screen example of FIG. 10, a first image display area 111 for displaying an image is provided on the right side of the screen, and a setting item button area 112 in which a plurality of buttons representing a plurality of setting items are arranged is provided on the left side of the screen. The setting item button area 112 includes an "image registration" button 113, an "imaging setting" button 284, a "camera setting" button, a "trigger setting" button, a "flash setting" button, a "lighting volume" button, a "lighting extension unit ” button, a “Save” button, and the like are provided. The user can select desired setting item buttons from the setting item button area 112 to set necessary setting items.

When the "image registration" button 113 provided in the setting item button area 112 is pressed in the imaging setting menu 269 of FIG. 10, the screen is switched to the image registration screen 270 of FIG. From this screen, various settings such as registration target, camera selection, registration destination, etc. can be performed. Here, after adjusting to a desired image using various buttons provided in the operation area, registration, that is, saving of the image data is performed. Here, the distance image is displayed in the second image display area 121 and the image variables assigned to this image are displayed in the operation area 122 . In the example of FIG. 11, "camera 1" is selected in the "camera selection" column 271 for selecting the imaging means, and the image variable "&Cam1Img" is set to "camera selection" as the distance image captured by this "camera 1". ” column 271 is displayed.

When the "Register" button 272 provided at the bottom of the operation area is pressed after the setting is completed, registration of the distance image currently displayed in the second image display area 121 is started as shown in FIG. 12, and the progress is displayed graphically. Is displayed. In addition, as shown in FIG. 13, luminance image registration is also performed. In this example, the distance image is first stored in the distance image storage unit 38a, and then the luminance image is stored in the luminance image storage unit 38b. The image variable “&Cam1Img” of the distance image and the image variable “&Cam1GrayImg” of the luminance image are also recorded. These image variables are assigned unique variables to each image, and can be used as indexes when calling registered images. However, this example is only an example, and the order of registration of each image may be reversed, or the images may be registered simultaneously. In this way, by simultaneously storing the distance image and the luminance image as registered images, the user can save labor in registering each image. However, it is also possible to adopt a configuration in which the distance image and the luminance image are individually registered as registered images.
(Phase shift method)

Here, a phase shift method will be described as one of methods for non-contact measurement of the displacement and three-dimensional shape of a workpiece. The phase shift method is also called a lattice pattern projection method, a fringe scanning method, or the like. In this method, a light beam having a grid pattern in which the illuminance distribution is varied sinusoidally is projected onto the workpiece. Moreover, three or more lattice patterns with different phases of the sine wave are projected, and each brightness value of the height measurement point is imaged for each pattern from an angle different from the projection direction of the light beam, and the lattice pattern is obtained from each brightness value. Calculate the phase value of . Depending on the height of the measurement point, the phase of the grating pattern projected onto the measurement point changes, and a light ray with a phase different from the phase observed by the light ray reflected at the reference position is observed. Therefore, by calculating the phase of the light ray at the measurement point and substituting it into the geometric relational expression of the optical device using the principle of triangulation, the height of the measurement point (and thus the object) is measured, and the three-dimensional shape is obtained. The method. According to the phase shift method, the height of the workpiece can be measured with high resolution by reducing the grating pattern period. Only objects with a small height difference can be measured.
(Spatial encoding method)

Therefore, the spatial encoding method is also used. According to the space coding method, a space to be irradiated with light is divided into a large number of small spaces having substantially fan-shaped cross sections, and a series of space code numbers are assigned to the small spaces. Therefore, even if the height of the work is high, that is, even if the height difference is large, the height can be calculated from the space code number as long as the work is in the space irradiated with light. Therefore, it is possible to measure the shape of a tall workpiece over its entirety. In this way, according to the spatial encoding method, the allowable height range (dynamic range) is wide, and the shape of the entire workpiece can be measured even for a tall workpiece.

Next, setting examples for capturing a distance image by the imaging means will be described with reference to FIGS. 14 to 42. FIG. 14. When the "imaging setting" button 284 is pressed from the imaging setting menu 269 of FIG. 10 prior to registering the image described above, the imaging setting screen 280 of FIG. On the imaging setting screen 280 . You can make basic settings related to imaging. For example, when the "imaging effective setting" button is pressed, the imaging effective setting screen of FIG. 15 is displayed, and the imaging means, that is, the camera connected to the three-dimensional image processing apparatus can be selected. For example, by connecting a camera capable of acquiring height information in place of or in addition to a monochrome CCD camera or a color CCD camera that captures a normal luminance image as an imaging means, the distance image is subjected to three-dimensional image processing. It becomes possible to take it into the device. Also, when a plurality of imaging means are connected to the three-dimensional image processing apparatus, one or more imaging means can be selected.

14, pressing a "detailed setting" button 282 provided in the operation area brings up a three-dimensional measurement setting screen 290 shown in FIG. In addition, in FIG. 16, different works are displayed for convenience of explanation. The three-dimensional measurement setting screen 290 of FIG. 16 includes a "continuous update display" field 292 corresponding to real-time updating means, a shutter speed setting field 294 corresponding to shutter speed setting means 49, a grayscale range setting field 296, a preprocessing setting field 310, a measurement It has an impossibility criterion setting column 312, an equal interval processing setting column 314, a space code setting column 316, a projector selection setting column 318, a "display image" selection column 322, and the like.
(Real-time update means)

Here, real-time updating means is provided for updating the image being displayed on the second image display area 121 to the changed setting when the setting is changed in the operation area. The real-time updating means can be switched ON/OFF. In the screen example of FIG. 16, the real-time update function can be operated by turning ON the "display by continuous update" column 292 provided in the upper part of the operation area as one form of the real-time update means.

In the example of FIG. 16, items that can be set in the operation area include shutter speed, grayscale range, pre-processing, non-measurable criteria, uniform interval processing, space code, projector selection, display image, and the like. They will be described in order below.
(Shutter speed setting means 49)

In the example of FIG. 16, a shutter speed setting field 294 is provided as one mode of the shutter speed setting means 49 for adjusting the shutter speed at the time of imaging by the imaging means. The user can specify the shutter speed from the shutter speed setting field 294 . Here, select a preset shutter speed from the drop-down box, e.g. to select. The number of seconds corresponding to the selected numerical value is displayed in the numerical display field 295 on the right side. You can also specify the desired shutter speed directly with a numerical value. For example, when "numerical value input" is selected as an option in the drop-down box, graying out of the numerical value display column 295 is canceled, and a numerical value can be directly input. Based on the numerical value specified in the shutter speed setting field 294 in this way, the exposure time of the camera (imaging device), which is the imaging means, is adjusted. When adjusting the shutter speed, it is easier to check if the grayscale image of the luminance image is displayed in the second image display area 121 rather than the distance image. Furthermore, the real-time update function allows the user to visually check whether the current settings are appropriate by quickly reflecting the image for which the shutter speed has been changed in the shutter speed setting field 294 in the second image display area 121. This makes it possible to confirm and easily perform the adjustment work.
(Darkness range setting field 296)

In the grayscale range setting field 296, the dynamic range of the luminance image, which is a grayscale image, is adjusted. Here, the dynamic range is increased or decreased by selecting one of "Low (-1)", "Normal (0)", and "High (1)" from the drop-down box.
(Pre-processing setting field 310)

In the preprocessing setting field 310, a common filtering process to be performed before the distance image is generated by the head unit is specified. Examples of common filtering include an averaging filter, a median filter, and a Gaussian filter. In the example of FIG. 17, one of "none", "median", "gaussian", and "average" is selected as the filtering process for the pattern projection image. In addition to the preprocessing performed before generating the distance image, the filtering process can also be performed on the distance image obtained by the head unit 1 side.
(Measurable standard setting field 312)

In the non-measurable standard setting field 312, a level for cutting noise components is set. That is, height measurement is not performed for the amount set in the non-measurable standard setting column 312 . In measuring three-dimensional height information using a pattern projection image, accurate height information cannot be measured unless there is a certain amount of light. On the other hand, when there is multiple reflection, the amount of light must be reduced because it is too bright. In this manner, the noise component cut amount is selected according to the captured pattern projection image. Specifically, a threshold is determined for the data to be considered as invalid data due to noise in order to calculate the height information of each pixel.

Here, as shown in FIG. 18, one of "high", "medium", "low" and "none" is selected from the drop-down box. If "none" is selected, height measurement is performed for all pixels without cutting noise components. For example, in the example shown in FIG. 19, "none" is selected in the non-measurable reference setting column 312, and height data is calculated at all points including noise data. Although it is difficult to distinguish from this screen, an incorrect height is measured due to noise data at the corner of the workpiece.

On the other hand, in the example shown in FIG. 20, "low" is selected in the non-measurable standard setting column 312, and the number of incorrect height information based on noise data is reduced. Furthermore, in the example shown in FIG. 21, "medium" is selected in the non-measurable standard setting column 312, and the number of points with incorrect height information is reduced.

On the other hand, it can also be confirmed from FIG. 21 that the number of black dots indicating that measurement is impossible has increased particularly in the lower left area of the workpiece, and that the number of positions where height measurement is impossible due to being regarded as noise is increasing. Furthermore, the example shown in FIG. 22 shows a state in which "high" is selected in the non-measurable standard setting field 312, and as a result of excessive noise component removal, even data that should be retained is lost. can be confirmed. Thus, if the setting of the unmeasurable standard indicating the noise removal threshold is too low, the height will be calculated based on the noise. On the other hand, if it is too high, the part that should be left will be regarded as invalid. Therefore, by using the real-time update function, the user can adjust the setting of the non-measurable standard, check the adjusted image in the second image display area, and directly refer to the setting result on the image. Can be adjusted to a suitable value.
(Equal interval processing setting field 314)

In the equal interval processing setting field 314, error correction due to the angle of view is performed. The equal interval processing setting column 314 functions as the interval equalizing processing setting means 47 . In the equal interval processing setting field 314, ON and OFF can be selected as shown in FIG. By turning on the equal interval processing, distance images arranged at equal pitches in the xy direction are obtained. In this case, the second image display area 121 is caused to display equipitch images in which the positions in the XY directions are at equal intervals regardless of the height (Z coordinate). For example, in applications such as inspecting dimensions on the XY plane, it is necessary to turn on the equal spacing process. In addition, the part where the data is lost due to the correction is treated as invalid. 24 and 25 show a state in which the equal spacing process is turned ON. In the example of FIG. 24, the "display image" selection field 322 is displayed in the second image display area 121 as the "height image", that is, the distance image. Displayed examples are shown respectively.

On the other hand, if the equal interval processing is turned off, the image (Z image) as seen by the eye is obtained as shown in FIG. 26, and distortion occurs in the XY directions toward the edges of the screen. However, since the equal interval processing is not performed, the time required until the image is displayed can be shortened. Note that FIG. 27 shows an example in which a “grayscale image” is selected in the display image field and a luminance image is displayed in the second image display area 121 .
(Spatial code setting field 316)

In the space code setting column 316, use or non-use of the space encoding method is selected. That is, the space code setting column 316 functions as the space code switching means 45 . In this three-dimensional image processing apparatus, the phase shift method is essential for generating the range image, and in addition to the phase shift method, whether or not to apply the spatial encoding method can be selected in the space code setting field 316 . In the space code setting column 316, ON and OFF can be selected as shown in FIG. When the space code setting field 316 is turned ON, height measurement is performed by a combination of the space code method and the phase shift method. This example is shown in FIGS. 29 and 30. FIG. In these figures, FIG. 29 shows a state in which the distance image is selected as the image to be displayed in the second image display area 121. FIG. Specifically, the “height image” is selected in the “display image” selection field 322 . On the other hand, FIG. 30 shows a state in which a luminance image is displayed in the second image display area 121, and a “grayscale image” is selected in the “display image” selection field 322. FIG. Appropriate range images can be obtained by using a spatial encoding method in addition to the phase shift method. Specifically, since the spatial encoding method can correct the phase jump (phase unwrap) by the phase shift method, it is possible to measure with high resolution while ensuring a wide dynamic range of height. However, the imaging time is approximately doubled compared to the case of OFF.

On the other hand, as shown in FIGS. 31 and 32, when OFF is selected in the space code setting field 316, height measurement is performed only by the phase shift method. In this case, since the height measurement dynamic range is narrowed, in the case of a work having a large difference in height, if the phase shifts by one period or more, the height cannot be measured correctly. Conversely, in the case of a workpiece with little change in height, the striped image is not picked up or synthesized by the spatial encoding method, so the processing speed can be increased accordingly, and the imaging time can be reduced by about half. For example, when measuring workpieces with small differences in the height direction, it is not necessary to take a large dynamic range, so even with the phase shift method alone, it is possible to shorten the processing time while maintaining high-precision height measurement performance. can be done. In this case, since the height measurement dynamic range is narrowed, in the case of a work having a large difference in height, if the phase shifts by one period or more, the height cannot be measured correctly. Conversely, in the case of a workpiece with little change in height, turning off the spatial code has the advantage of halving the imaging time.

In the example of FIG. 31, the "display image" selection field 322 shows a state in which a "height image", that is, a distance image is displayed. ” is displayed.

Although the phase shift method is essential in this example, it may be possible to select ON/OFF of the phase shift method.
(Projector selection setting field 318)

The projector selection setting field 318 functions as the light projection switching means 48 for switching ON/OFF of the first projector and the second projector. Here, in the projector selection setting field 318, the light projecting means (projector) to be used is selected from among the two light projecting means, the first projector and the second projector. In this example of the projector selection setting field 318, as shown in FIG. 33, "1" (first projector), "2" (second projector), "1+2" (first projector and second projector).

When "1" or "2" is selected in the projector selection setting field 318, i.e., in the case of single projection, which is projection of light from either the first projector or the second projector, the portion shaded by the projection height measurement is not performed. The example in FIG. 34 shows an example in which "1" is selected in the projector selection setting field 318, and the example in FIG. 35 shows an example in which "2" is selected. On each screen, the data of the shadowed portion is displayed in black, and it can be confirmed from each screen that there are areas on the workpiece where the height cannot be measured. Moreover, as is clear from these figures, it can be seen that the non-measurable region differs depending on the light projecting means. In other words, even if one of the light projecting means cannot be measured, the other light projecting means can project light, and therefore there are many areas where the height can be measured. Therefore, by combining these, the unmeasurable region can be reduced. In particular, by arranging the first projector and the second projector so as to sandwich the workpiece from both sides, the first light from the first projector and the second light from the second projector are in opposite directions. Since the workpiece is irradiated with the light, even if an area is shadowed by one of the light projections, the risk of being shadowed by the light projected from the opposite direction can be reduced.

Specifically, as shown in FIG. 36, when "1+2" is selected in the projector selection setting field 318, both the first projector and the second projector are switched to both projection. In this state, even if a portion is shaded by the projection of light from one of the projectors, it can be interpolated if the other projector can project light. However, in this case, it takes about twice as much imaging time as compared to the one-way projection. The user selects which projection light to use according to the unevenness of the workpiece to be inspected, the allowable imaging time, and the like.
(“Display image” selection field 322)

In the “display image” selection field 322, an image displayed in the second image display area 121 is selected. For example, by selecting a display target according to the purpose of inspection, the validity of each setting can be visually confirmed from the actually displayed image. In particular, by turning on the above-described real-time updating means, it is possible to sequentially update the setting change and compare before and after the change, so that the setting can be adjusted based on the image so that the intended image is obtained according to the application. . Also, even a beginner who is not familiar with the meaning of each setting parameter can obtain the advantage of being able to set while viewing the image. In this example, as shown in FIG. 37, from the "display image" selection field 322, "height image", "grayscale image", "brightness/blackout image", "stripe projection - projector 1", "stripe image" Select either “projection-projector 2”. A "height image" is a distance image, and a colored image is displayed by color-coding contour lines for each height. A "grayscale image" is a luminance image. In this example, an image obtained by synthesizing a plurality of pattern projection images captured based on the phase shift method is used as the luminance image. However, it is also possible to illuminate the workpiece, capture an optical image with the imaging means, and use it as a brightness image.
(Abnormal point highlighting means 62)

Further, the three-dimensional image processing apparatus is provided with abnormal point highlighting means 62 as shown in FIG. For example, the "brightness/blackout image" that can be selected in the above-described "display image" selection field 322 partially replaces the luminance image with saturated whiteout pixels and insufficient light amount blackout pixels. It is a colored image. In this way, areas in the image where accurate values are not obtained and where the reliability of measurement accuracy is thought to be low are highlighted by coloring processing, so that the user can visually see areas where measurement accuracy is low. to make it easier to check whether an image suitable for the desired inspection application has been obtained. In this example, overexposed pixels are colored yellow, and overexposed pixels are colored blue. With this, the user can visually confirm how the blocked-out areas are distributed in the image, using the color as a clue. Also, in the lower part of the second image display area 121, the numbers of overexposed pixels and overexposed pixels are displayed. While referring to this, the user adjusts each setting item so that the number of pixels approaches zero.

Note that the color and manner of coloring are not limited to these, and various known manners such as displaying in other colors or blinking can be used as appropriate. In addition, by changing the color of the overexposed pixels and overexposed pixels, the user can be notified of the reason why the reliability of the measurement is low, thereby making it easier to take countermeasures. However, the same color or highlight may be given to overexposed pixels and overexposed pixels.

“Stripe projection-projector 1” is a pattern projection image represented by gradation obtained by pattern projection with only the first projector. "Stripe projection-projector 2" is a pattern projection image obtained only by the second projector. FIG. 38 shows an example in which "stripe projection - projector 1" is selected in the "display image" selection field 322 and the pattern projection image of the first projector is displayed in the second image display area 121. FIG. An example in which the pattern projection image of the second projector is displayed in the second image display area 121 by selecting "stripe projection-projector 2" is shown. By checking the stripe image on this screen, it is possible to identify the cause when the height of the workpiece cannot be measured. It can be confirmed from the original pattern projection image before the distance image is generated that it is a difficult work.

While referring to the image displayed in the second image display area 121, the user confirms whether the shutter speed and the grayscale range are appropriate, and adjusts them to appropriate values. Specifically, in a state in which a blown-out/black-out image is displayed in the second image display area 121, adjustment is performed while confirming that over-exposed pixels and blocked-up black pixels are reduced. For example, by adjusting the shutter speed in the shutter speed setting field 294, blackout pixels, that is, portions that are too dark due to insufficient light amount are eliminated. Also, by adjusting the grayscale range, overexposed pixels, ie, overly bright portions, are eliminated. In the example of FIG. 16, there are many blackened pixels in the overexposed/blackened image because the image is too dark. Therefore, the shutter speed is adjusted.

For example, if the state of FIG. 16 in which the shutter speed setting field 294 is set to "1/120" is switched to "1/15" as shown in FIG. I know what happened. However, the number of overexposed pixels has increased. Therefore, when the shutter speed is switched to "1/30" as shown in FIG. 41, it can be seen that the number of blown-up pixels remains zero while the number of blown-up pixels also decreases.

41, in which the density range setting field 296 is set to "low (-1)", is switched to "normal (0)" as shown in FIG. I know. Furthermore, as shown in FIG. 43, when switching to "high (1)", it can be seen that overexposed pixels are reduced to zero. As a result, both the number of overexposed pixels and the number of overexposed pixels become 0, and the operation of adjusting the shutter speed and the grayscale range is completed.

After setting the desired imaging conditions as described above, the distance image and the luminance image are registered as shown in FIGS.
(Height measurement setting procedure)

Next, a procedure for setting an area for height measurement in an input image of a work to be inspected that is sequentially input during operation will be described with reference to FIGS. 44 to 55. FIG. Here, a typical work representative of a plurality of input works is registered in advance as a registered image by the above-described procedure, and a region whose height is to be measured is specified for this registered image.

Specifically, a "height measurement" processing unit 266 is added from the initial screen 260 in FIG. 9 as shown in FIGS. In the example of FIG. 44, select "Add" from the first submenu 370 displayed by right-clicking or the like below the "imaging" processing unit 263 in the flow display area 261, and select "Measurement" in the second submenu 372. is selected to add a "height measurement" processing unit 266 that performs "height measurement" among the inspection processes listed in the "measurement" menu 373 displayed. As a result, as shown in FIG. 45, a new “height measurement” processing unit 266 is added below the “imaging” processing unit 263 in the flow display area 261 . Thus, the "measurement" menu 373 functions as inspection process selection means for selecting inspection processes to be executed by the inspection execution means.
(Inspection target area setting screen 120)

Next, a procedure for setting an area as a setting to be performed by the "height measurement" processing unit 266 will be described with reference to FIGS. 46 to 49. FIG. First, when the edit screen of the "height measurement" processing unit is called from the screen of FIG. 45, the screen shifts to a height measurement setting screen 460 shown in FIG. In the GUI screen example of FIG. 46 as well, as in FIG. 10, a first image display area 111 for displaying an image is provided on the right side of the screen, and a setting item button area 112 in which a plurality of setting item buttons are arranged is provided on the left side of the screen. The setting item button area 112 includes an "image registration" button 113, an "image setting" button 114, an "area setting" button 115, a "preprocessing" button 117, a "detection condition" button 118, a "detailed setting" button 119, A "judgment condition" button, a "display setting" button, a "save" button, etc. are provided. From this screen, when the user presses a "set area" button 115 corresponding to inspection target area setting means, the screen transitions to an inspection target area setting screen 120 shown in FIG. On the inspection area setting screen 120, an area to be inspected can be specified. In the example of FIG. 47, a second image display area 121 is provided on the left side of the screen, and an operation area 122 for performing various operations is arranged on the right side of the screen. A “display image” selection field 124 for selecting an image to be displayed in the second image display area 121 is provided in the upper part of the operation area 122 . In the example of FIG. 47, a registered image is selected in the “display image” selection field 124 . Furthermore, below it, a "measurement area" setting column 126 is provided as an inspection target area setting means for specifying an area to be inspected.

A predetermined area can be selected in the “measurement area” setting field 126 . Here, when the "measurement area" setting field 126 is selected, a drop-down box is displayed as shown in FIG. 48, and the desired shape of the measurement area can be selected. In this example, the selectable measurement area shape candidates are "none", "rectangle", "rotated rectangle", "circle", "ellipse", "circumference", "arc", "polygon", "Complex area" or the like is displayed. If "none" is selected, the entire image displayed in the second image display area 121 is used as the inspection target area.

Further, according to the shape selected in the "measurement area" setting field 126, it is possible to set detailed dimensions and the like. In the example of FIG. 48, an eraser is used for the workpiece, and in the "measurement area" setting column 126, "rotated rectangle" is selected as shown in FIG. When the "edit" button 128 is pressed in this state, a measurement area edit screen 130 shown in FIG. 50 is displayed. In the example of FIG. 50, in the second image display area 121, a rotated rectangle is displayed superimposed on the workpiece. Here, a rectangular measurement area is drawn on the eraser case and displayed superimposed on the distance image. In addition, the basic vector of the rotated rectangle is displayed as an arrow within the frame of the rotated rectangle, and the width and height of the rotated rectangle, the XY coordinates of the center, the inclination angle of the basic vector, etc. are displayed on the screen of the measurement area editing screen 130. Is displayed. The user can arbitrarily adjust the shape, position, etc. of the rotated rectangle by directly inputting numerical values on the measurement area edit screen 130 or by operating the handles displayed on the rotated rectangle with a mouse or the like.

Items that can be set on the measurement area edit screen 130 change according to the shape selected in the “measurement area” setting field 126 . For example, when "circumference" is selected, as shown in FIG. 51, it is possible to set parameters related to the circumference, such as specification of the outer diameter and inner diameter of the circumference. Furthermore, it is also possible to designate a mask area from the screen of FIG. As the mask area, a circular shape, a doughnut shape, a rectangular shape, other polygonal shapes, a free curve, and the like can be specified. In this way, it is possible to appropriately set the inspection target area according to the shape of the workpiece to be inspected, and eliminate unnecessary areas for inspection such as perforated portions and backgrounds, thereby improving the processing efficiency. .
(Second measurement display area)

When the measurement area is set in this manner, the set measurement area is superimposed on the work as shown in FIG. Subsequently, when specifying another measurement area, the same operation is repeated. That is, as shown in FIG. 52 , another second “height measurement” processing unit 266B is added below the “height measurement” processing unit 266 in the flow display area 261 . Then, as shown in FIGS. 53 and 54, a rotated rectangle is set as a new measurement area on the area of the eraser, which is the work here, not covered by the case. As a result, as shown in FIG. 55, the newly set second measurement area is superimposed on the work and displayed.

In the above example, one height measurement process is performed by one "height measurement" processing unit. That is, in order to perform a plurality of height measurement processes, it is necessary to add a plurality of "height measurement" processing units. However, it goes without saying that it is also possible to configure a single "height measurement" processing unit to perform a plurality of height measurement processes.
(measurement processing)

When the setting of the measurement area is completed in this way, next, a process of actually performing measurement is added. Here, as shown in FIG. 56, in the flow display area 261, a "numerical calculation" processing unit that performs "calculation" is added below the second "height measurement" processing unit 266B. Numerical calculations, image calculations, calibration, image concatenation, etc. can be selected as the content of calculations to be executed by the "numerical calculation" processing unit. Here, as shown in FIG. 57, a "numerical calculation" processing unit is added in which numerical calculation is selected.
(“numerical operation” processing unit)

In the "numerical operation" processing unit, a concrete arithmetic expression can be input. For example, as shown in FIG. 58, a numerical calculation edit screen is displayed on which a mathematical expression can be directly input, and the user specifies the arithmetic expression. Here, a calculator-like input pad is prepared, and edit buttons such as copy, cut, and paste are also prepared to facilitate creation of arithmetic expressions. The user inputs a desired arithmetic expression from this screen. FIG. 59 shows an example of an input arithmetic expression.

When the content of the numerical calculation process is defined in this way, the calculation formula is displayed in the third image display area 262 on the initial screen 260 as shown in FIG.
("area" processing unit)

Furthermore, in the example of FIG. 61, an "area" processing unit is added below the "numerical calculation" processing unit. In the "area" processing unit, the conditions for actually making pass/fail judgments are defined. Specifically, in order to appropriately change the gradation conversion conditions for gradation conversion of the distance image to the low gradation distance image according to the registered image and the input image, the gradation conversion parameter (details will be described later) A region for acquiring reference information, a condition for extracting height from this region, a condition for filtering when generating a distance image, and the like are set. That is, in the "area" processing unit, area setting, height extraction, preprocessing, judgment, etc. are set. First, the procedure for setting the area is the same as for the registered image described above. That is, when the "area setting" button 115 arranged in the setting item button area 112 is pressed on the area setting screen 620 as shown in FIG. 62, the area setting screen shown in FIG. 63 is displayed, and the target area is specified. Again, select a rotated rectangle, and specify detailed coordinates and the like as necessary. In this way, the area in the "area" processing unit is determined, and the rotated rectangle is displayed superimposed on the workpiece in the second image display area as shown in FIG.
(Height extraction setting screen)

Next, set height extraction. The setting of height extraction is to set the gradation conversion parameter when performing gradation conversion. That is, when the "height extraction" button 116 is pressed in the setting item button area 112 in FIG. 62, the height extraction selection screen 140 shown in FIG. 65 is displayed, and the display image, extraction method, etc. can be selected. The height extraction selection screen 140 also has a second image display area 121 on the left side of the screen and an operation area 122 for performing various operations on the right side of the screen, as in FIG. 47 and the like. A “display image” selection field 124 for selecting an image to be displayed in the second image display area 121 is provided in the upper part of the operation area 122 . In the example of FIG. 65, a registered image is selected in the “display image” selection field 124 . Furthermore, an extraction method selection means 142 for selecting an extraction method for the height extraction function is provided below it.

Here, the "extract height" button 116 functions as the gradation conversion condition setting means 43 for setting gradation conversion parameters for performing gradation conversion of the distance image by the gradation conversion means. In particular, the gradation conversion condition setting means 43 is displayed when a process that does not require image height information is selected by the inspection process selecting means. Conversely, when a process requiring image height information is selected by the inspection process selection means, the gradation conversion condition setting means is not displayed. Specifically, when the “height measurement” processing unit 266 is selected as the inspection processing tool, the “height extraction” button is not displayed in the flow display area 261 . For other inspection processing tools, such as the "area" processing unit, the "blob" processing unit 267, the "color inspection" processing unit 267B, the "Shapetrax2" processing unit 264, the "position correction" processing unit 265, etc., the "height An "extract" button 116 is displayed to enable setting of tone conversion conditions. By doing so, the gradation conversion condition setting means 43 is displayed when gradation conversion is necessary, and the user is urged to make necessary settings. By hiding the means for setting the conditions, the user is prevented from being confused by unnecessary settings, and a user-friendly environment is realized.
(Extraction method selection means 142)

The extraction method selection means 142 designates a gradation conversion method. For example, let the user select either static conversion or dynamic conversion. In the example of FIG. 65, one of "single-point designation" or "three-point designation (plane)" corresponding to static transformation, or "real-time extraction" corresponding to dynamic transformation is selected from the drop-down box. Let
(One point designation screen 150)

If "single point designation" is selected from the extraction method selection means 142 on the screen of FIG. 65, the screen shifts to the single point designation screen 150 of FIG. 66 to 96 show an example using a 50-yen coin as the workpiece for the sake of explanation. On the one-point designation screen 150 in FIG. 66, the height of the part designated on the second image display area 121 is set as the reference height (reference height). In the example of FIG. 66, when the "Extract" button 144 provided in the operation area 122 on the right side of the screen is selected, the screen shown in FIG. 67 appears, and any position on the second image display area 121 on the left side of the screen can be specified become. Here, the height extracting means is used to specify the center position of the height in the work displayed in the second image display area 121 . In the example of FIG. 67, the height extracting means consists of an "extract" button 144 displaying a dropper-shaped icon SI. A pointer 146 with a shape is displayed. The position designated by this pointer 146 is registered as the middle height of the distance range.

Also, the range for obtaining the height around the point designated by the pointer 146 can be designated in the “extraction region” designation field 145 . In the "extraction area" specification column 145, one side of the area for which the average height is to be calculated is specified by the number of pixels. In the example of FIG. 67, "16" is specified in the "extraction area" specification column 145, and the average height within the area of 16 pixels×16 pixels centered on the point specified by the pointer 146 is extracted. , be the height extracted by the pointer 146 . The size of the area specified by the pointer 146 on the second image display area 121 can be changed in conjunction with the numerical value specified in the “extraction area” specification field 145 .

Also, in the "Z height" display field 152, the height information of the designated part is displayed as a numerical value (in the example of FIG. 68, 1.253 is displayed in the "Z height" display field 152). For example, when expressing the distance range with 2 ⁸ =256 gradations (0 to 255), the gain (density value/mm; details will be described later) is 128 with the height specified by the height extraction means as the central value. is set to be With this configuration, the user can specify the height to be inspected directly on the screen, and the range around the specified height will be converted into a low-gradation distance image. Avoid loss of information.
(simple display function)

When the distance range and span are determined as tone conversion parameters necessary for tone conversion in the manner described above, it becomes possible to tone-convert a high-tone distance image into a low-tone distance image. Also, on the second image display area 121, as shown in FIG. 68, a low gradation distance image that has undergone gradation conversion under the gradation conversion conditions currently set in the operation area 122 is simply displayed. . Further, when the gradation conversion conditions are changed in the operation area 122, the simple display of the low gradation distance image after the gradation conversion on the second image display area 121 is also performed according to the changed gradation conversion conditions. updated. As a result, the user can quickly visually confirm the change in the gradation conversion condition after the adjustment, and can easily perform the adjustment work by trial and error. As described above, the image displayed in the second image display area 121 includes a mode for displaying a distance image before tone conversion, a mode for displaying a low tone distance image after tone conversion, and a mode for displaying a normal luminance image. It can be changed by switching the mode to make it work.
(Gain adjusting means)

Furthermore, the user can use the gain adjustment means to perform gain adjustment, which is one of the gradation conversion parameters. In the example of FIG. 68, an emphasis method setting field 154 is provided in the middle of the operation area 122, and a gain adjustment field 156 is arranged here as gain adjustment means. The current gain is displayed numerically in the gain adjustment field 156 . Here, the gain [gradation/mm] is a parameter corresponding to the span when performing gradation conversion. For example, when converting a distance image of 16 gradations into 8 gradations, the number of gradations out of 8 gradations per mm is set. Increasing the gain value results in gradation conversion with clear contrast. For example, if the gain value is set to 100 [gradation/mm], the gradation conversion is set such that 0.010 mm per gradation. If the height information of the distance image before conversion has a resolution of 0.00025 mm per gradation, the difference between the obtained reference plane and the input height data is N gradations before conversion. , after conversion, can be calculated as N [gradation]×0.00025 [mm/gradation]×100 [gradation/mm]=N×0.025 gradation.

Here, the reference plane is a plane determined by a method such as one-point specification, average height standard, three-point specification, plane standard, or free-form surface standard, and serves as a reference for tone conversion. For example, as shown in FIG. 69A, when the cross-sectional profile of the range image (input image) before conversion to 16 gradations has a shape indicated by solid lines, the reference plane is indicated by broken lines. FIG. 69B shows the profile of the low-gradation distance image obtained by converting such an input image from 16-gradation to 8-gradation. is applied.

In addition, the height per gradation (the reciprocal of the gain value) can be automatically calculated according to the gain value described above and displayed together. In the example of FIG. 68, 250 [gradation/mm] is displayed as the gain value, and 0.0040 mm is displayed as the height per gradation. The user can adjust the gain value by changing the gain value. For example, if the gain value is increased, as shown in the screens of FIGS. 68 to 70, the height information can be inspected in detail by emphasizing the density difference, but the inspectable height range is narrowed. Conversely, if the gain value is lowered, a wide height range can be inspected as shown in FIG. 71, but fine changes are impaired. In this way, by adjusting the gain with the gain adjusting means, the gradation-converted image obtained under the gradation conversion conditions can be confirmed in the second image display area 121 . As a result, the user can adjust the gain value appropriately according to the inspection purpose or the like while confirming the tone-converted image updated in real time.
(Setting extraction height)

Further, setting items for the enhancement method can include setting of extraction height in addition to gain value. For example, in the screen of FIG. 72, when the "details setting" button 158 provided at the lower right of the operation area 122 is pressed, the screen is switched to the emphasis method detail setting screen 160 of FIG. In addition to 156, an "extraction height" setting field 162 is displayed. In the "extraction height" setting field 162, the height information to be extracted by the height extraction means is any of high height information, low height information, and both high and low height information included in the area. You can choose Here, as shown in FIG. 74, any one of “high side”, “low side”, and “both high and low” can be selected from a drop-down list provided in the “extraction height” setting field 162 . For example, if the "higher side" is selected, gradation conversion is performed so that the height of the position specified by the pointer 146 becomes the lower limit of the distance range. As a result, a low gradation distance image is generated in which only the side higher than the designated height is extracted. Similarly, if the "lower side" is selected, tone conversion is performed so that the height of the position designated by the pointer 146 becomes the upper limit of the distance range. As a result, a low gradation distance image is generated in which only the side lower than the specified height is extracted. Furthermore, in the case of "both high and low", gradation conversion is performed so that the height of the position specified by the pointer 146 is the middle of the distance range described above. Pixels outside the range after gradation conversion are clipped to black on the lower side (pixel value 0 in the case of 8 gradations) and to white on the higher side (pixel value 255).

Further, in the enhancement method detail setting screen 160 of FIG. 73, a noise removal setting column 164 for removing noise and an invalid pixel specifying column 166 for specifying a value to be given to invalid pixels are also provided.
(Noise removal setting field 164)

In the noise removal setting field 164, as one of the gradation conversion parameters, how many mm difference from the reference plane is to be removed as noise is specified. For example, if the noise removal parameter is set to 0.080 mm, the difference of 0.080 mm from the reference plane is removed. Here, if the height information before conversion has a resolution of 0.00025 mm per gradation, the difference of 0.080 [mm] ÷ 0.00025 [mm] = 320 [gradation] is ignored. becomes. This state will be described with reference to FIGS. 75A to 75C. 75A, similar to FIG. 69A, the cross-sectional profile of the distance image before conversion to 16 gradation is indicated by a solid line, its reference plane is indicated by a wavy line, and the range to be noise-removed is indicated by a dashed-dotted line. there is A profile shown in FIG. 75B is obtained as a result of performing noise removal on such an input image with reference to the reference plane. Furthermore, the profile of the low gradation distance image obtained by performing gradation conversion from 16 gradation to 8 gradation for the distance image in FIG. 75B is as shown in FIG. is applied.

Further, the effects of gain adjustment and noise removal will be described with reference to FIGS. 76A to 76F. First, assume that a luminance image as shown in FIG. 76A and a distance image with high gradation (16 gradations) as shown in FIG. 76B are obtained. Here, the distance image in FIG. 76B is set to low gradation (8 gradations) with the initial settings (here, the gain is 100 [gradation/mm] and the noise reduction is 0.000 [mm]). FIG. 76C shows a low gradation distance image that has undergone gradation conversion to . This low tone distance image has relatively low contrast. Therefore, when the gain is increased from this state, as shown in FIG. 76D, a low gradation distance image with increased contrast is newly gradation-converted from FIG. 76B and displayed. However, the noise component is also large in this image. In the example of FIG. 76D, the gain is set to 1000 [gradation/mm] and the noise reduction is set to 0.000 [mm]. Therefore, FIG. 76E shows a low gradation distance image obtained by increasing the amount of noise removal from FIG. 76D. Here, the gain is set to 1000 [gradation/mm] and the noise reduction is set to 0.080 [mm]. Although the noise component was reduced by this, it can be confirmed that noise having a height lower than that of the reference plane exists in the upper right of the upper left "E". Therefore, if the "extraction height" setting field 182 shown in FIG. As a result of ignoring and extracting only the side higher than the reference plane, a low gradation distance image as shown in FIG. 76F is obtained. For example, if the "extraction height" is set to the "high side" for the profile distance image (16 gradations) shown in FIG. 77A, the result is that only the high side from the reference plane is extracted. A low gradation distance image (8 gradations) as shown in FIG. 77B is obtained. In this way, it is possible to obtain a low gradation distance image with higher contrast and less noise components than those in FIG. 76C or the like. In this example, the final gradation conversion parameters are set to 1000 [gradation/mm] for gain, 0.080 [mm] for noise reduction, and ``high side'' for "extraction height". The distance image of 76B is tone-converted into the low-tone distance image of FIG. 76F.

When the conditions necessary for executing one-point designation are set in this way, the input image is scaled from a high-gradation range image to a low-gradation range image according to the specified gradation conversion conditions, that is, the reference height and the like. It is tone-converted and displayed in the first image display area 111 as shown in FIG.

Here, an example of a workpiece for which the method of designating a reference surface by specifying one point is effective will be described with reference to FIGS. 79A and 79B. FIG. 79A shows a workpiece WK7 that does not have a planar inclination on the measurement surface of the workpiece, or that has a slight inclination that does not affect the inspection process. Here, an inspection process of reading whether or not the character string is appropriate is performed on the workpiece WK7 in which numbers and character strings are three-dimensionally formed on the surface of the casting by OCR. In such a use, the dropper-shaped icon SI is displayed on the second image display area 121 by pressing the "extract" button 144 shown in FIG. Then, with the pointer 146, one point on the upper surface of the work WK7 on which no character string is formed (background surface) is designated as shown in FIG. 79A. As a result, gradation conversion is performed using the height of the extraction region (16 pixels in the example of FIG. 67) designated by the pointer 146 as a reference plane, and the low gradation distance image shown in FIG. 79B is obtained. In this low gradation distance image, the character string portion protruding from the plane of the work WK7 is clearly extracted from the plane of the work WK7 as a background, so that accurate OCR can be easily performed. In this way, one-point designation can be effectively used in cases where the inspection process is not affected even if the workpiece is tilted to some extent. One point specification also has the advantage of low load and high speed processing.
(three points designation)

The method for setting tone conversion conditions by specifying one point has been described above. Next, a method of setting tone conversion conditions by specifying three points will be described with reference to the GUI screens of FIGS. 80 to 85. FIG. The three-point designation is a method of tone-converting a distance image into a low-tone distance image using a plane obtained from three points designated by the user as a reference plane. Similarly to the one-point designation reference height described above, the reference plane is, for example, the middle height of the height range (distance range) in which the height information of the distance image is gradation-converted into the low-gradation distance image. Alternatively, the upper limit (highest position for tone conversion) or lower limit (lowest position for tone conversion) of the distance range can be used.

In the state where the "height extraction" button 116 is pressed on the GUI screen of the three-dimensional image processing program in FIG. 62 and the height extraction selection screen 140 shown in FIG. , select "Three points (plane)". As a result, a three-point designation screen 170 for performing height extraction setting shown in FIG. 81 is displayed.
(Three-point designation screen 170)

On the three-point designation screen 170 , three points on the second image display area 121 are designated and set as a reference plane that serves as a reference for gradation conversion. For this reason, the three-point designation screen 170 in FIG. 81 is provided with height extracting means. Specifically, by selecting the "Extract" button 144 provided in the operation area 122 on the right side of the screen, the screen shown in FIG. You will be able to specify points. Here, a point-like pointer 146 is displayed as height extracting means in the same manner as in FIG. 67, and the user sequentially designates a desired position with a mouse, trackball, or a pointing device such as a touch panel. First, when the first point is specified on the second image display area 121, the rectangular shape changes to a cross shape at the specified position as shown in FIG. Similarly, the pointer 146 can be used to specify. At this point, the color distance image displayed in FIG. is displayed in the second image display area. When the second point is further specified, the position of the second point changes from a rectangular shape to a cross shape as shown in FIG. 84, and the third point can be specified. At this point, the gradation conversion is performed again based on the inclined surface including the height of the two designated points, and the low gradation distance image is updated. Then, when the third point is specified, the reference plane is set by a plane containing these three specified points. Further, when each point is specified by the height extraction means, the height of the point currently specified on the height extraction screen display area may be displayed in the “Z height” display field 152 .

Furthermore, the inclination angle can be displayed as the information of the reference plane. In the example of FIG. 84, the height extraction display field 172 provided in the operation area 122 displays the X-direction tilt, the Y-direction tilt, and the Z-direction height of the third point.

Also, as with single point designation, the emphasis method can also be designated as necessary. For example, use the gain adjustment means to adjust the gain, or press the three-point designation "detailed setting" button 174 to call the three-point designation detailed setting screen 180 as shown in FIG. In addition to the gain adjustment field 156, an "extraction height" setting field 182 is displayed, and height information to be extracted by the height extraction means is selected from a drop-down list of "high side", "low side", and "both high and low". You can choose either

In this way, it is possible to perform gradation conversion of a distance image using an arbitrary plane defined by three specified points as a reference plane. As a result, not only the gradation conversion based on the horizontal plane, such as the one-point specification described above, but also the gradation conversion using an inclined plane as the reference plane becomes possible. For example, when inspecting for scratches or foreign matter on an inclined surface of a workpiece, the distance range would be narrower if the inclined surface were left as it was. Scratches and foreign objects can be detected. In this way, it is possible to realize flexible gradation conversion using height information according to the workpiece and inspection purpose.

Here, an example of a workpiece for which the method of specifying the reference surface by three-point specification is effective will be described with reference to FIGS. 86A to 86D. FIG. 86A shows a work WK8 in which the result of the inspection process is affected if there is a planar tilt on the measurement surface of the work or if there is a minute planar tilt. Here, an inspection process for detecting a ball grid array (BGA) formed on the substrate is performed. In such a use, the dropper-shaped icon SI is displayed on the second image display area 121 by pressing the "extract" button 144 shown in FIG. Then, with the pointer 146, as shown in FIG. 86A, three points on the upper surface of the workpiece WK8 where no BGA is formed are specified. As a result, a plane containing the specified three points is extracted as a reference plane, and gradation conversion is performed to convert it into the low gradation distance image shown in FIG. 86B. In this low gradation distance image, the BGA projecting from the plane of the workpiece WK8 is clearly extracted against the background, so that the shape of the BGA can be confirmed by binarizing it, for example, as shown in FIG. 86C. This method has the advantage of being able to accurately detect even if the plane of the work WK8 is tilted. If the workpiece WK8 shown in FIG. 86A is tilted, for example, one-point specification results in a binarized image as shown in FIG. 86D, which cannot be detected correctly. On the other hand, in three-point designation, the inclination is corrected as described above, and an accurate detection result is obtained. In this way, three-point designation is effective in cases where the inclination of the plane affects the result of inspection processing.

Further, consider an inspection process for detecting a recess when the upper surface of a planar work WK9 as shown in FIG. 87A has a gentle recess. Here, as shown in FIG. 87A, the measurement area ROI is set to an area including a depression. As a result, gradation conversion is performed using the plane obtained from the height data of the entire measurement region ROI including the dent as the reference plane, and a low gradation distance image as shown in FIG. 87B is obtained. In this example, the reference plane is estimated by the method of least squares. Further, the obtained low-tone distance image is binarized to obtain the binarized image shown in FIG. 87C. This makes it possible to correct the tilt and stably extract only the recessed portion. If a binarized image is obtained by specifying one point in a state where there is an inclination, the result shown in FIG. 87D is obtained, and it can be seen that the inclined surface makes it difficult to detect depressions. In this way, three-point designation is effective for estimating a reference plane with high accuracy. In terms of processing time, although the processing time is longer than that of specifying one point, the processing can be performed at a relatively high speed.

In the above description, the static conversion is described in which the gradation conversion conditions are specified in advance at the setting stage and the gradation conversion is performed under the specified conditions during operation. In other words, in static conversion, the gradation conversion parameter is a constant value regardless of the input image. Next, a specific example of dynamic conversion for adjusting tone conversion conditions according to an input image to be inspected will be described. First, for dynamic conversion, as a specific method for correcting the standard of height information to be left when converting a distance image to a low-gradation distance image,
(B1) Average height reference for gradation conversion using the average height (average distance) in the average extraction area specified for the input image as the average reference height,
(B2) a plane reference for generating an estimated plane within a specified area of the input image and using this as a reference plane for gradation conversion;
(B3) A free-form surface reference is included in which a free-form surface is generated by removing high-frequency components from the input image and used as a reference surface for gradation conversion. Each method will be described below in order.
(B1: average height standard)

The average height reference is a method of calculating the average height within a designated average extraction area for each input image and using this as the average reference height for gradation conversion. The average extraction area for defining the average reference height is preset prior to operation (step S83 in FIG. 8 described above). An example of the procedure for specifying the average extraction area in step S83 of FIG. 8 will be described below with reference to the GUIs of FIGS. 62, 65, and 88 to 92. First, select the "height extraction" button 116 on the GUI screen of FIG. 62 to proceed to the height extraction selection screen 140 of FIG. Then, the height dynamic extraction setting screen 190 of FIG. 88 is displayed. In this example, an example using an eraser as the workpiece is shown again. Next, in the "calculation method" selection column 192 provided below the extraction method selection means 142, the standard of dynamic conversion is specified. Here, any one of "average height standard", "plane standard", and "free curved surface standard" is selected from a drop-down box as shown in FIG. Here, select "Average height standard". As a result, the screen shifts to the average height reference setting screen 210 in FIG. For convenience of explanation, FIGS. 90 and 91 show an example in which a 50-yen coin is used as the workpiece.

It should be noted that, when making settings related to dynamic conversion, it is necessary to set an average reference height and the like for an image different from the distance image input during actual operation. For this reason, an image corresponding to the workpiece during operation is captured in advance and saved as a registered image, and when setting dynamic conversion, the registered image is read and used as a substitute for the workpiece image during operation. Make various settings in the form. Therefore, the corresponding "registered image" is specified in the "display image" selection field 124 on the screen of FIG. 65 or the like.

On the average height reference setting screen 210, a separately set inspection target area is used as it is, or an arbitrary average extraction area is designated as necessary. Any method can be used for specifying the average extraction area, such as a rectangular shape, four corners, a circle by specifying the center and radius, or a free curve. It is also possible to designate only one point of the workpiece, or conversely, the entire workpiece or the entire image displayed in the second image display area 121 to be the average extraction area. Alternatively, as described above, the separately specified inspection target area can be used as the average extraction area. In these cases, the task of designating the average extraction area by the height extraction means may be omitted.
(mask area)

Moreover, it is possible to specify a mask area from which the average height is not extracted for the average extraction area. For example, when the "extraction area" button 194 provided in the operation area 122 is pressed on the screen of FIG. 90, the mask area setting screen 220 shown in FIG. 91 is displayed. On this mask area setting screen 220, one or more mask areas that are unnecessary for extracting the average height can be specified. The mask area can also be specified by specifying an arbitrary area such as a rectangular shape or a circular shape from the second image display area 121 as described above.

Furthermore, gain adjustment etc. can also be performed as needed. For example, when the "details setting" button 196 provided at the lower right of the operation area 122 is pressed from the screen of FIG. In addition, detailed setting items such as specification of extraction height and noise removal are displayed.

After the average extraction area is defined in this manner, the setting screen is terminated. At the time of gradation conversion, gradation conversion is performed using the average value (average reference height) of the height information contained in this average extraction area as the reference height. For example, gradation conversion is performed so that the average reference height becomes the center value of the distance range (128, which is the center value of the distance range from 0 to 255 in the case of 2 ⁸ =256 gradations). Moreover, it is not necessary to use all the height information of all the points included in the average extraction area, and the processing can be simplified by appropriately thinning out or averaging.

During operation, dynamic conversion is performed according to the procedure shown in FIG. 133, which will be described later. For example, a workpiece conveyed on a line is imaged to generate a distance image (step S13301), the average height of the average extraction area set above is calculated (step S13302), and gradation conversion is performed based on this. A low gradation distance image is generated by execution (step S13303), and the obtained low gradation distance image is inspected (step S13304). With this method, even if there is variation in the height direction of the workpiece, the reference plane for gradation conversion can be reset for each workpiece, so accurate inspection can be achieved regardless of variations in the height direction of the workpiece. can.

An example of a workpiece for which the method of designating the reference plane based on the average height is effective will now be described with reference to FIGS. 93A and 93B. FIG. 93A shows a workpiece WK7 that does not affect the inspection process even if there is no planar inclination or slight inclination on the measurement surface of the workpiece, similar to FIG. 79A described above. An inspection process for reading whether the character string is appropriate or not is performed on the three-dimensionally formed work WK7 by OCR. In such a use, a rectangular measurement region ROI for determining the average height reference is set on the second image display region 121 on the extraction region setting screen shown in FIG. 92 and the like. Here, as shown in FIG. 93A, a plane surrounding the character string on the upper surface of the work WK7 is designated as the measurement region ROI. As a result, gradation conversion is performed using the height of the measurement region ROI as a reference plane, and the low gradation distance image shown in FIG. 93B is obtained. As in FIG. 79B, the low gradation distance image also allows the character string portion projecting from the plane of the work WK7 to be cleanly extracted from the plane of the work WK7 as a background, thereby facilitating accurate OCR. In addition, since only one point on the workpiece WK7 is specified in FIG. 79A described above, the selected point may be affected by noise, whereas in FIG. It has the advantage of reducing the influence of
(plane reference)

An example of performing dynamic conversion based on the average height has been described above. Next, as another dynamic transformation, a plane reference for estimating a plane included in a reference plane estimation region specified in advance for an input image and performing gradation conversion using this estimated plane as a reference plane will be described. In this method, for example, when the surface of the workpiece is inclined, the gradient component can be canceled and the gradation conversion can be performed. Therefore, it has the advantage of being able to be utilized in the same manner as the three-point specification of the static conversion described above. A specific method for setting the plane reference will be described below. Also in the plane reference, the reference plane estimation area for determining the reference plane is set in advance (step S83 in FIG. 8 described above), similarly to the average height reference described above. An example of the procedure for specifying the reference plane estimation area in step S83 of FIG. 8 will be described below based on the GUIs of FIGS. 62, 88, and 92 to 95. FIG. First, select the "height extraction" button 116 on the GUI screen of FIG. 62 to proceed to the height extraction selection screen 140 of FIG. Then, the height dynamic extraction setting screen 190 of FIG. 88 is displayed. Next, in the "calculation method" selection column 192, as shown in FIG. 89, when "plane reference" is selected as a reference for dynamic conversion, the plane reference setting screen shown in FIG. 92 is displayed.

On the plane reference setting screen, similarly to the height extraction means in the height extraction selection screen 140 of FIG. 80, the separately set inspection target area is used as it is, or an arbitrary reference plane estimation area is specified as necessary. . The reference plane estimation area can be specified by any method such as a rectangular shape, a circle by specifying four corners, a circle by specifying a center and a radius, or a free curve. It is also possible to designate only one point on the workpiece, or conversely, to set the entire workpiece or the entire image displayed in the second image display area 121 as the reference plane estimation area.

Also, it is possible to specify a mask area in which the estimation plane is not estimated for the reference plane estimation area, as in FIG. 90 and the like described above. Furthermore, it is also possible to perform gain adjustment, extraction height specification, noise removal, etc., as required, as in the above case. For example, on the screen of FIG. 92, when the "detailed setting" button 222 provided in the operation area 122 is pressed, the plane reference detailed setting screen 240 of FIG. 94 is displayed. An "extraction height" setting field 162 for designating an extraction height, a noise removal setting field 164 for noise removal, and an invalid pixel designation field 166 for designating invalid pixels are displayed, and these detailed settings can be made. Become. In the invalid pixel specification field 166, as shown in FIG. 95, invalid pixels for which the distance cannot be obtained can be filled with a specified default value, a background pixel value, or an arbitrary value specified by the user. can.

When the reference plane estimation area is defined in this manner, the setting screen is terminated. At the time of gradation conversion, a planar estimated plane is calculated from the height information included in this reference plane estimated area. A known method such as the least squares method can be appropriately used for fitting the height information distributed within the reference plane estimation region. It should be noted that it is not always necessary to use the height information of all points included in the reference plane estimation area for calculation of the estimation plane, and it is possible to simplify the processing by appropriately thinning out or averaging, as described above. is.

After the estimated plane is determined in this way, tone conversion is performed with reference to this estimated plane. For example, gradation conversion is performed so that the estimated plane becomes the central value of the distance range. It is also possible to display information on the calculated estimated surface. For example, in the example shown in FIG. 92, in the estimated plane display field provided in the operation area 122, the tilt in the X direction, the tilt in the Y direction, and the height in the Z direction of the estimated plane are displayed. In this example, the estimated plane is one plane, but it is also possible to combine a plurality of planes to make the estimated plane. Also, in this example, the estimated surface is a flat surface, but it is also possible to calculate the estimated surface as a simple curved surface such as a spherical surface.

Examples of workpieces for which the method of designating the reference plane based on the plane reference is effective include the above-described FIG. 86A and FIG. 87A. By setting a plane reference for such a workpiece, even if the upper surface of the workpiece is inclined, the reference plane can be detected by correcting the inclination. Pattern detection is achieved.
(B3: free-form surface reference)

Finally, a free-form surface reference is described in which a free-form surface is generated by removing high-frequency components from a predetermined area (free-form surface target area) of an input image, and tone conversion is performed using this as a reference surface. For example, when the workpiece has a curved surface, and approximation using a simple plane is difficult, it is difficult to accurately extract the height information of the area to be inspected. Therefore, by generating a simplified image by removing high-frequency components from the input image and using the surface shape (free-form surface) of this image as a reference plane, rough shapes and gentle changes can be ignored, and sudden changes can be avoided. It is possible to perform an inspection while leaving only the part where the change occurs, that is, the fine shape.

A specific method for setting the free-form surface reference will be described below with reference to the GUIs of FIGS. 62 to 96. FIG. In the free-form surface reference as well, the necessary conditions for determining the reference surface are set in advance (step S83 in FIG. 8 described above), similarly to the above-described average height reference and the like. First, select the "height extraction" button 116 on the GUI screen of FIG. 62 to proceed to the height extraction selection screen 140 of FIG. Then, the height dynamic extraction setting screen 190 of FIG. 88 is displayed. Next, in the "calculation method" selection column 192, as shown in FIG. 89, "free curved surface reference" is selected as the reference for dynamic transformation. As a result, the free-form surface reference setting screen 250 shown in FIG. 96 is displayed.

Although it is possible to designate an arbitrary area as the free-form surface target area on the free-form surface reference setting screen 250 as well, preferably, the entire image displayed in the second image display area 121 or a separately designated inspection object The area is used as it is as the free-form surface target area. A free-form surface is generated by removing harmonic components from the area specified as the free-form surface target area. Then, a gradation-converted image obtained by performing gradation conversion using the free-form surface as a reference plane is superimposed and displayed on the free-form surface object area displayed on the second image display area 121 . In addition, gain adjustment, designation of extraction height, noise removal, etc., can be performed as necessary at the time of gradation conversion, as in FIG. 90 and the like described above.
(Extraction size adjustment means)

It also has an extraction size adjustment function that adjusts the fineness (extraction size) of the extraction surface extracted based on the free-form surface. Specifically, in FIG. 96, an "extraction size" designation field 252 is provided as an extraction size adjustment means in the "detailed setting of extraction surface" field of the operation field. By increasing or decreasing the numerical value in the "extraction size" designation field 252, the curvature of the free-form surface changes accordingly, and the size of extractable defects changes. Here, a free-form surface image is generated and displayed in the second image display area 121 so that unevenness having a size equal to or less than the set size is extracted. When the extraction size is increased, a smooth free-form surface is generated, and defects of a size corresponding to the set extraction size can be extracted. On the other hand, if the extraction size is reduced, a free-form surface along the surface shape of the workpiece is generated, and only small defects corresponding to the set extraction size are extracted. For example, if the extraction size is increased, the free-form surface becomes smooth with respect to the surface shape of the workpiece as shown in FIG. Conversely, if the numerical value is decreased, the free-form surface becomes closer to a detailed shape along the unevenness of the surface shape of the workpiece as shown in FIG. Become. In addition, when the numerical value of the "extraction size" designation field 252 is increased or decreased, the gradation-converted image within the free-form surface target area displayed in the second image display area 121 is also internally generated as a reference plane accordingly. The state of the free-form surface changes, and the size of the object extracted and displayed as the difference from the reference surface changes in real time. The user can optimally adjust the numerical value in the “extraction size” designation field 252 while referring to the second image display area 121 .

When the free-form surface target area is defined in this manner, the setting screen is terminated. At the time of gradation conversion, a free-form surface is calculated from the height information of the image included in this free-form surface target area. As shown in the flowchart of FIG. 161, the fitting of the height information distributed within the free-form surface target region includes image reduction processing (step S1611), filtering processing (step S1612), and image enlargement processing according to the set extraction size. A method of performing processing (step S1613) to generate a free-form surface image can be used. A median filter, for example, can be used for filtering. When the free-form surface is determined in this way, the difference can be taken from the original image (step S1614) to clarify extracted irregularities and the like. Alternatively, as described above, a known method such as the method of least squares can be used as appropriate. As described above, it is not always necessary to use the height information of all points included in the free-form surface target area for calculation of the estimated surface, and the processing can be simplified by appropriately thinning out or averaging. After the free-form surface is determined in this manner, tone conversion is performed with reference to this free-form surface. For example, gradation conversion is performed so that the free-form surface becomes the central value of the distance range.

An example of a workpiece for which the method of designating a free-form surface as a reference surface is effective will now be described with reference to FIGS. 98A and 98B. FIG. 98A shows inspection processing for detecting defects such as protrusions and depressions included in the curved surface of the work. Here, an area including a defect is designated from the curved surface as the measurement area ROI. Then, a free-form surface is obtained from the height information included in the designated measurement region ROI, and tone conversion is performed using the obtained free-form surface as a reference plane. The resulting low-gradation distance image is shown in FIG. 98B. As shown in this figure, with the free-form surface as a reference, portions higher than this surface are protrusions (represented by white dots in FIG. 98B), and portions lower than this surface are recesses (represented by black dots in FIG. 98B). detectable. Designation of a reference plane by a free-form surface in this way can be effectively used for a workpiece having a curved surface shape. Note that the processing load for detecting a free-form surface is higher than the one-point designation and three-point designation described above.
(Defect extraction processing)

In the above-described free-form surface criterion, when the depth image is compressed in the vertical and horizontal directions, that is, in the XY directions, in the image reduction process shown in step S1611 of FIG. It may not be possible to extract the scratches as expected. For example, consider an example of trying to extract an extraction target 1 and an extraction target 2 in a distance image obtained by imaging a tube-shaped inspection object as shown in FIG. 162 . If the free-form surface criterion is selected for such a range image and the image is compressed in the XY directions, as shown in FIG. On the other hand, the extraction object 2 that is originally desired to be extracted is not extracted cleanly.
(Extraction direction designating means)

Therefore, in order to realize accurate defect detection even in such cases, an extraction direction specification function is provided to extract defects, that is, local irregularities, by excluding shapes that exist continuously in a specific direction on the inspection target surface. can also be provided. For example, an extraction direction designation means is provided that can designate an extraction direction when designating inspection conditions. Specifically, when "free curved surface reference" is selected in the "calculation method" selection column 192 of the GUI shown in FIG. Let In the free-form surface reference setting screen 630 of FIG. 164, an “extraction direction” designation field 632 is added below the “extraction size” designation field 252 as a form of extraction direction designation means. In the "extraction direction" designation field 632, the direction for extracting local shape changes such as defects to be extracted is designated. In this example, one of "X", "Y" and "XY" can be selected from a drop-down list or the like as preset options. 164, the screen in FIG. 164 is similarly switched to a detailed setting screen 640 shown in FIG. A noise removal setting field 164, an invalid pixel designation field 166, and the like are displayed, and detailed settings thereof can be made.

For example, if "Y" is selected in the "extraction direction" designation field 632 on the screens of FIGS. 164 and 165, local shape changes in the Y direction (vertical direction in FIG. 162) of the image are extracted. In other words, shapes that do not change in the Y direction, such as grooves extending in the vertical direction, are ignored, and the detection results shown in FIG. 166 are obtained. Similarly, if X is selected, shape change in the X direction (horizontal direction in FIG. 162) of the image is extracted. When XY is selected, local shape changes in the XY directions are extracted as shown in FIG. As a result, by designating the extraction direction according to the shape of the object to be inspected, unnecessary shapes can be canceled and the desired shape can be appropriately extracted. For example, if character strings 1 and 2 are to be extracted from an object to be inspected as shown in FIG. By specifying the Y direction as the extraction direction, a desired character string can be accurately extracted by removing uniform shapes in the Y direction as shown in FIG.

Here, as a specific shape extraction method such as defect extraction processing, extraction processing in the Y direction shown in FIGS. 166 and 169 will be described as an example. In the image reduction process described above (step S1611 in FIG. 161), the image is reduced only in the Y direction. As a result, the reference free-form surface image obtained by filtering (step S1612) and image enlargement processing (step S1613) is a free-form surface smoothed only in the Y direction with fine irregularities in the Y direction removed. An image will be generated. In other words, it is possible to obtain an image in which only the uniform shape to be canceled in the Y direction is left. Therefore, when the difference between the target original image and this free-form surface image is taken (step S1614), the shape to be canceled that is uniform in the Y direction is removed, resulting in a finer shape in the Y direction. Only the uneven shape change is left. This realizes extraction of shape changes only in a desired direction.

Such processing is particularly effective for inspection of an inspection object that is long in one direction. By specifying the extraction direction along the longitudinal direction of the object to be inspected, uniform shapes that are continuous along the longitudinal direction can be canceled, and accurate defect extraction can be realized.

In addition, in order to acquire height information of such an object to be inspected that is continuous in one direction (for example, a product that has a uniform shape in one direction, such as a tire), in addition to structured illumination, the above-described light An inspection using a cutting method can be preferably applied. In particular, the light-section method is a method for acquiring the shape (profile) of a line-shaped cut surface, so the profile is continuously acquired while changing the cutting position for an inspection object that is long in one direction. A range image can be obtained by synthesizing the obtained profiles. With this method, even if the object to be inspected is moving, in other words, it can be inspected without being stationary. It becomes possible to detect defective products without stopping the product.

In the above example, the defect extraction process is applied to the free-form surface reference, but the present invention is not limited to the free-form surface reference and can be applied to other reference surfaces. For example, when the average height criterion is selected as the reference plane, the above-described defect extraction processing is applied to obtain the average height of each line in the specified direction, and the difference from the average height is calculated. can take. This method has the advantage that high-speed processing is possible because the processing is simple.
(Extraction area setting dialog 148)

Furthermore, when performing height extraction, the target area (extraction area) for calculating the reference plane from the input image is the same area as the inspection target area (measurement area) for inspection processing, and is separate from the measurement area. can also be set to As an example, in FIG. 90, pressing an "extraction area" button 147 displays an extraction area setting dialog 148, as shown in FIG. 99, in which an extraction area can be set. The extraction area setting dialog 148 is provided with an extraction area selection column 149 from which the user can select "same as measurement area", "rectangle", "circle", "rotated rectangle", and the like. If "same as measurement area" is selected, the extraction area will be the same as the measurement area as described above. On the other hand, by selecting other than "same as measurement area", an area different from the measurement area can be set as the extraction area. For example, when "Rectangle" is selected in the extraction area selection field 149 as shown in FIG. 100, a rectangular frame is displayed in the second image display area 121, and the user can specify the desired area by dragging the mouse. When the "Edit" button 324 provided to the right of the extraction area selection field 149 in FIG. 100 is pressed, an extraction area editing dialog 326 is displayed as shown in FIG. can be specified by When the extraction area is adjusted in the extraction area edit dialog 326, the change is reflected in the second image display area 121. FIG.
(Mask area setting column 330)

Furthermore, from the extraction area setting dialog 148, it is possible to set a mask area for specifying an area not to be an extraction area. In the example of FIG. 99, a mask area setting field 330 is provided below the extraction area selection field 149 . A plurality of mask areas can be set in the mask area setting column 330 . Here, a maximum of four mask areas from 0 to 3 can be specified, and each mask area can be set independently. For example, as shown in FIG. 102, when “circular” is selected as the mask area 0, the circular mask area 0 is displayed on the second image display area 121. FIG. Further, when an "edit" button 332 is pressed from this state, a mask area edit dialog 334 for defining details of the circular mask area 0 is displayed as shown in FIG. The user can define a circular mask area 0 with the xy coordinates of the center and the radius. The masked area is displayed in the second image display area 121 with a frame line of a color different from that of the extracted area, making it easier for the user to visually distinguish between the extracted area and the masked area. In the example of FIG. 103, the extraction area is displayed in green and the mask area is displayed in yellow. However, it is not limited to this example, and it goes without saying that it is also possible to distinguish by different colors, line thickness, line type (solid line, dashed line, etc.), highlight (blinking or emphasized), etc. .

In this way, it is also possible to set the extraction region independently of the measurement region. Also, the setting contents of the extraction area can be displayed as character information. For example, in the example of FIG. 90, the text "same as measurement area" is displayed below the "extraction area" button 147, indicating that the extraction area for calculating the plane reference is the same as the measurement area. In addition, "mask area: invalid" is displayed below it, indicating that the mask area is not set in the extraction area. This allows the user to confirm the outline of the "extraction area" as text information as well.
(Preprocessing settings)

After setting the height extraction in the "area" processing unit as described above, in the third image display area as shown in FIG. A low gradation distance image gradation-converted based on the extracted height is superimposed and displayed. Next, set the preprocessing. Preprocessing is a common filtering process performed before generating a range image as described above, and various filters can be selected here. Specifically, when the "preprocessing" button provided in the setting item button area 112 is selected from the screen of FIG. 104, a filter processing setting screen 340 of FIG. 105 is displayed, and a filter to be applied can be selected. Filters that can be selected here include an averaging filter, a median filter, a Gaussian filter, and the like. Furthermore, other than such filters, for example, it is also possible to set a binarization level. For example, on the binarization level setting screen 350 of FIG. 106, the upper limit value, the lower limit value, and the number of times of binarization can be set. It may also have a function of displaying a histogram indicating the distribution of binarized pixels or updating the histogram in conjunction with the input image. When the filter processing setting is completed in this way, the low gradation distance image binarized through the filter processing within the set region is superimposed and displayed in the third image display region as shown in FIG. .
(judgment setting)

Furthermore, in the "area" processing unit, after tone-converting the input image into a low-tone distance image according to conditions such as the set area, height extraction, and preprocessing, this low-tone distance image is It also specifies the conditions for judging depth inspections and image inspections. For example, when the "judgment condition" button provided in the setting item button area 112 is pressed on the screen of FIG. 107, the judgment condition setting screen 360 of FIG. 108 is displayed and the judgment condition is set. In this example, the pixels of the binarized low gradation distance image are counted, and OK is set when the numerical value is within a predetermined range, and NG is set when it is not. In the example of FIG. 108, since the determination condition is 0 to 30 and the current value is 166, it is determined to be NG, and as shown in FIG. is displayed. In this way, when the determination result is NG, the characters are displayed in red or otherwise conspicuous, thereby making it easier for the user to recognize them during operation.
("blob" processing unit 267)

Also, although the determination based on height inspection using height information has been described above, the present invention is not limited to this, and it is also possible to add determination processing based on conventional image inspection for luminance images. Such an image inspection is called a blob. For example, as shown in FIG. 110, a “blob” processing unit 267 that performs blob (image inspection) is added below the “area” processing unit in the flow display area 261 . In the "blob" processing unit 267, similarly to the above, the target area is set and the preprocessing is set (Fig. 111), the detection conditions are set (Fig. 112), the judgment conditions are set and judgment is made. The result can be output (Fig. 113).
(“Color Inspection” processing unit 267B)

Furthermore, when a color CCD camera is connected as imaging means, or when a color optical image can be input, color inspection can be combined. For example, as shown in FIG. 114, a “color inspection” processing unit 267B that performs color inspection as measurement processing is added below the “blob” processing unit 267 in the flow display area 261 . In the "color inspection" processing unit 267B, similarly to the above, the target area is set (FIGS. 115 and 116), detailed settings such as density average are made (FIG. 117), and judgment conditions are set. can output the determination result (FIG. 118).

After various settings are made in the setting mode as described above, an input image is acquired by actually imaging the work in the operation mode, and determination processing is performed based on the results of height inspection and image inspection. In the above example, the user can easily recognize the image of the determination result at the setting stage by adopting a configuration in which the determination result can be output even in the setting mode. However, it goes without saying that the determination result can be output only in the operation mode.
(operation mode)

Next, processing during operation inside the head unit and the controller unit in the three-dimensional image processing apparatus shown in FIGS. 4A and 4B will be described based on the flowcharts of FIGS. First, the flow of processing during operation in the head unit of the three-dimensional image processing apparatus according to Embodiment 3 shown in FIGS. 4A and 5 will be described based on the flowchart in FIG.
(Flow of processing according to Embodiment 3)

First, when a trigger is input from the outside (step S11901), one light projection pattern is projected from the first projector 20A onto the work (step S11902), and the image is picked up by the imaging means (step S11903). Next, it is determined whether or not imaging has been completed for all light projection patterns (step S11904). If not, the light projection pattern is switched (step S11905), and the process returns to step S11902 to repeat the process. Here, 8 pattern projection images are taken with the light projection pattern using the phase shift method, and 8 pattern projection images are taken with the light projection pattern using the spatial encoding method, for a total of 16 pattern projection images. .

On the other hand, if all projection patterns have been captured in step S11904, the process branches to steps S11906 and S11907. First, in step S11906, three-dimensional measurement calculation is executed to generate a distance image A. FIG.

On the other hand, in step S11907, an average image A' is calculated by averaging a plurality of pattern projection images (pattern projection image group) captured by the phase shift method.

The above steps S11902 to S11906 are three-dimensional measurement by pattern projection from the first projector 20A. Next, three-dimensional measurement is performed by pattern projection from the second projector 20B. Here, in step S11908 following step S11906, similarly to steps S11902 to S11905, the light projection pattern is projected onto the workpiece from the second projector 20B (step S11908), an image is captured by the imaging means (step S11909), It is determined whether or not imaging has been completed for all projection patterns (step S11910). If not, the projection pattern is switched (step S11911) and the process returns to step S11902 to repeat the processing. In step S11912, a three-dimensional measurement calculation is executed and a distance image B is generated. On the other hand, in step S11913, an average image B' of the pattern projection image group captured by the phase shift method is calculated. After the three-dimensional distance images A and B are generated in this manner, the three-dimensional distance images A and B are synthesized to generate a distance image in step S11914. Also, in step S11915, using the average images A' and B', a luminance image (average two-dimensional grayscale image) is generated by synthesizing them. In this manner, the distance image having the height information of the workpiece is obtained in the three-dimensional image processing apparatus of FIG. Note that steps S11907, S11913, and S11915 can be omitted if the luminance image is unnecessary.
(Flow of processing in Embodiment 4)

The processing flow of the head unit of the three-dimensional image processing apparatus according to the third embodiment shown in FIG. 4A has been described above. Next, the processing flow of the head unit of the 3D image processing apparatus according to Embodiment 4 shown in FIG. 4B will be described based on the flowchart of FIG. explain. First, when a trigger is input from the outside (step S12001), one light projection pattern is projected from the light projecting means 20 onto the workpiece (step S12002), the image is captured by the first image capturing means 10A (step S12003), At the same time, the second imaging means 10B also takes an image (step S12004). Then, it is determined whether or not imaging has been completed for all projection patterns (step S12005). If not, the projection pattern is switched (step S12006), and the process returns to step S12002 to repeat the processing. In step S12007, a three-dimensional measurement calculation is executed and a distance image A is generated. In step S12008, a three-dimensional measurement calculation is executed and a distance image B is generated. On the other hand, in step S12009, the average image A' of the pattern projection image group captured by the phase shift method is calculated, and in step S12010, the average image B' of the pattern projection image group captured by the phase shift method is calculated. In step S12011, the three-dimensional distance images A and B obtained in steps S12007 and S12008 are synthesized to generate a distance image. Also, a luminance image is generated by synthesizing the average images A' and B' obtained in steps S12009 and S10. According to this method, two images can be captured at the same time, so that the imaging time can be shortened.
(Inspection target area setting means)

For inspection during actual operation, it is necessary to specify in advance an area (inspection target area) to be inspected by the inspection execution means 50 on the workpiece. Such setting of the inspection target area is performed by the inspection target area setting means in the setting stage. The inspection target area setting means can be provided on the controller section 2 side as described above, or can be realized by a three-dimensional image processing program. Specifically, as described above, when the "set area" button 115 corresponding to the inspection area setting means of the three-dimensional image processing program shown in FIG. 62 is pressed, the screen changes to the inspection area setting screen 120 shown in FIG. On this inspection area setting screen 120, an area to be inspected can be specified.

When the inspection target area is specified in this way, the three-dimensional image processing device performs image processing on the inspection target area, and further performs inspection. That is, as shown in the flowchart of FIG. 121, an inspection target area is assigned to a distance image (step S12101), tone conversion parameters are set based on this distance image (step S12102), and tone conversion parameters are set according to the tone conversion parameters. Tone conversion is performed (step S12103), image processing is performed on the tone-converted image, and predetermined inspection is performed (step S12104).

Note that the gradation conversion process is performed on the inspection target area set by the inspection target area setting means described above. That is, in this example, the inspection target area setting means is shared with the gradation conversion target area specifying means for specifying the gradation conversion target area. However, the area used to determine the parameters of the gradation conversion process may be set independently of the inspection target area. For example, the area for gradation conversion parameter creation is specified using the inspection target area setting means or the gradation conversion parameter creation area specifying means prepared separately.
(Operation flow of the controller during operation)

Next, the procedure for processing the distance image obtained on the head side on the controller side during operation will be described with reference to the flowchart of FIG. 122 and the GUIs of FIGS. 123 to 126 show the GUI of the three-dimensional image processing program. The GUI of the three-dimensional image processing program shown in FIG. 123 shows an initial screen 260, with a flow display area 261 on the left side of the screen and a third image display area 262 on the right side. In the flow display area 261, a flow diagram is displayed in which the contents of each process performed by the three-dimensional image processing apparatus are connected in the form of processing units. Here, as processing units, an “imaging” processing unit 263 , a “Shapetrax2” processing unit 264 , a “position correction” processing unit 265 , and a “height measurement” processing unit 266 are displayed in the flow display area 261 . Also, at the stage of setting before operation, each processing unit can be selected and detailed settings can be made. In addition, in the third image display area 262, a brightness image, a distance image, or an inspection result is displayed according to the content of processing. In the example of FIG. 123, a luminance image obtained by picking up a workpiece (IC in this example) is displayed, and a search target area SA, which will be described later, is displayed in a green frame.
(imaging step)

First, in step S12201 in FIG. 122, a distance image and a luminance image are obtained from the head unit 1. Here, a pattern projection image is captured on the head unit side to generate a distance image and a luminance image. In the GUI example of FIG. 123, the “imaging” processing unit 263 displayed in the flow display area 261 corresponds.

As for the image data, the distance image is first transmitted from the head unit to the controller unit, and then the luminance image is also transmitted to the controller unit. Conversely, the luminance image may be transferred first and then the distance image may be transferred, or these transfers may be performed at the same time.
(search step)

Further, in step S12202, the controller unit performs pattern search on the input image input during operation. That is, the part to be inspected is specified so as to follow the motion of the workpiece included in the picked-up luminance image. In the example of FIG. 123, the "Shapetrax2" processing unit 264 displayed in the flow display area 261 corresponds to the pattern search. Here, the image search means 64 in FIG. 5 performs pattern search on the brightness image (input image) of the workpiece and performs positioning. Specifically, the position of the preset inspection target area is specified in the input luminance image. A search target area SA for pattern search is set in advance. For example, in the example of FIG. 124, the search target area SA is specified in a rectangular shape on the luminance image displayed in the third image display area.
(Position correction step)

Next, in step S12203, the position of the inspection area is corrected using the result of the pattern search. In the example of FIG. 123, the “position correction” processing unit 265 displayed in the flow display area 261 corresponds. The inspection target area is set in advance on the inspection target area setting screen. In the example of FIG. 125, a plurality of inspection target areas are set on the distance image displayed in the third image display area. Specifically, as shown in FIG. 126, a rectangular area is set for each pin of the IC, which is the work. The position correction is performed, for example, by a method of calculating the amount of positional deviation by normalized correlation search, a method based on the result of pattern search, or the like. In this manner, the position correction corrects the position of the inspection target area of the inspection process to be executed in the next stage.
(Inspection processing step)

Finally, in step S12204, inspection is performed using the distance image at the corrected position. In the example of FIG. 123, the “height measurement” processing unit 266 displayed in the flow display area 261 corresponds. Also, in the example of FIG. 127, height measurement is executed. That is, the height is measured and inspected in each inspection target area at the corrected position. For example, it determines whether or not the plane height is within a predetermined reference value, and outputs the determination result.

In the above procedure, an example has been described in which inspection is performed based on the distance image after the position correction is performed using the luminance image. However, the present invention is not limited to this, and an image for position correction and an image for inspection processing can be arbitrarily set. For example, conversely to the above, it is also possible to perform position correction using a distance image and then execute inspection processing using a luminance image. For example, when a white workpiece is placed against a white background, where it is difficult to perform an accurate pattern search using a luminance image, pattern searches based on height information using a depth image are effective. Become. Also, the inspection process can be performed not only based on the height information, but also based on the result of image processing using a luminance image, such as reading a character string printed on the work by OCR.
(Output format of height information)

The measured three-dimensional height information is obtained as three-dimensional point cloud data having X, Y, and Z values. In addition to the three-dimensional point cloud data, it is also possible to convert the actually obtained values into, for example, a Z image or an XY pitch Z image, in addition to the three-dimensional point cloud data.
(1:Z image)

A Z image is height image data of only Z coordinates. For example, if the unevenness of the position of the work imaged by the imaging means is important and the X and Y coordinates do not need to be accurate, the X and Y coordinates are unnecessary, so the Z image, which is data only for the Z coordinates, is used. Output is enough. In this case, the amount of data to be transmitted is reduced, and the transmission time can be shortened. In addition, since the data can be handled as an image in the same way as a normal two-dimensional imaging means, image processing can be performed using an existing image processing device for two-dimensional images.
(2: XY equal pitch Z image)

The XY equal-pitch Z image is height image data in which the XY coordinates are equal-pitch regardless of the height. Specifically, an XY equal-pitch Z image is obtained by interpolating the Z-coordinate at a position where the XY coordinates are equal-pitch from the surrounding point cloud data.

Generally, when the lens of the imaging means is not an objective telecentric lens, the position (X, Y coordinates) imaged by the imaging means differs depending on the height position (Z coordinate) of the imaged workpiece. Therefore, even if an object is captured at the same position on the imaging device, the actual XY coordinate position differs depending on the height. For example, when it is desired to inspect a three-dimensional difference such as volume, the value becomes large when the work is close to the camera, and becomes small when the work is far away, which is inconvenient. Therefore, by obtaining Z data with an XY uniform pitch from the point cloud data, it is possible to obtain a Z image having XY positions that are not affected by height.
(3: XYZ (point cloud data))

Alternatively, the point cloud data can be directly output as three-dimensional information. For example, it is used when you want to handle the measured three-dimensional data as it is. In this case, the amount of data is three times as large as that in the case of using only the Z coordinate, but since it is raw data, it can be used for purposes such as obtaining a three-dimensional difference from three-dimensional CAD data.
(Generation of uniform pitch image)

Next, FIG. 128 shows a data flow diagram for creating an equal-pitch image with a corrected angle of view in addition to the distance image and the luminance image. As shown in this figure, first, a spatial code image is generated according to the spatial encoding method from a group of spatial encoding pattern projection workpiece images captured by the head unit. On the other hand, a phase image is generated according to the phase calculation from the phase shift pattern projection work image group. From these spatial code image and phase image, phase extension calculation is performed to generate an extended phase image. Further, it is also possible to apply common filter processing, such as filtering, to the spatial encoding pattern projection work image group and the phase shift pattern projection work image group. Common filtering includes application of two-dimensional filters such as median filters, Gaussian filters, and averaging filters. On the other hand, a brightness image (average grayscale image) is generated by averaging the phase shift pattern projection work image group.
(Equal interval processing)

After the three-dimensional data and the brightness image are generated by the head unit as described above, a distance image such as a Z image or an equal-pitch Z image is created. First, a distance image is transferred from the head unit 1 to the controller unit 2, and gradation conversion is performed to convert phase information into height information. Here, an X image, a Y image, and a Z image are obtained from the phase information, and these XY are equalized to acquire an XY equal-pitch Z image and an XY equal-pitch Z-average image on the XY plane. Such equal spacing processing is performed by the spacing equalization processing setting means 47 .

In the example of FIG. 128, an example in which the phase shift method and the spatial encoding method are combined in generating the range image has been described, but the range image can also be generated only by the phase shift method without using the spatial encoding method. . The spatial encoding process can be switched ON/OFF by the spatial encoding switching means 45 shown in FIG. Such an example is shown in the data flow diagram of FIG. As shown in this figure, since the space encoding method is not used, the number of captured images can be reduced and the range image can be generated at high speed.

On the other hand, FIG. 130 shows a data flow diagram when the XY equal pitch function is turned off and a Z image is obtained. In this example, there is no need to separate the image into the X image and the Y image for tone conversion, so the processing can be simplified accordingly.

Furthermore, FIG. 131 shows an example of outputting point cloud data in which XYZ coordinate information is output as it is. In this example, since the X image, Y image, and Z image after the phase→height conversion can be output as they are, the advantage of being able to output with a light load can be obtained.
(Gradation conversion method)

Next, a procedure for automatically converting a high-gradation distance image into a low-gradation low-gradation distance image based on the distance image by the gradation conversion means 46 of the three-dimensional image processing device will be described. . Here, in an inspection device installed on a line in which a plurality of workpieces are transported, a range image (input image) that is sequentially input is subjected to gradation conversion in real time to a low gradation range image. A procedure for performing gradation conversion on the input image will be described. The gradation conversion processing in this case is roughly classified into (A) a method of determining gradation conversion parameters in advance (static conversion) and (B) a method of deciding gradation conversion parameters according to an input image. (dynamic method). These will be described below.
(A: static conversion)

First, static conversion in which tone conversion parameters are determined in advance will be described. Here, gradation conversion parameters for performing gradation conversion are adjusted for an input image or a registered image registered in advance at the time of setting. During operation, the gradation conversion of the distance image is performed using the gradation conversion parameters set at the time of setting, and the low gradation distance image after the gradation conversion is inspected. The procedure at the time of setting is as explained based on the above-described flowchart of FIG.
(Operation procedure)

When the gradation conversion parameters are adjusted, gradation conversion is performed on the input image input during operation using the gradation conversion parameters. Here, the procedure of static conversion during operation will be described based on the flowchart in FIG. First, in step S13201, a distance image is acquired. Here, the controller section 2 takes in the distance image generated by the distance image generating means 32 . Next, in step S13202, gradation conversion processing of the input distance image is performed. Here, gradation conversion processing is executed according to the gradation conversion parameters adjusted at the time of setting, and a low gradation distance image is generated by reducing the number of gradations of the distance image, that is, the dynamic range. Finally, in step S13203, the inspection execution means 50 executes inspection processing. According to this method, by setting the gradation conversion parameters in advance, there is no need to calculate the gradation conversion parameters during operation, and the processing load can be lightened.
(B: dynamic conversion)

Next, dynamic conversion for calculating tone conversion parameters for tone conversion based on an input image will be described. First, the setting procedure is performed according to the flowchart of FIG. 8 as described above. Specifically, in step S81, an input image or a registered image is obtained, and in step S82, a gradation conversion method is selected. Here, it is assumed that the user has selected dynamic conversion. Then, in step S83, the gradation conversion parameters are adjusted. Here, the conditions under which the gradation conversion parameters are to be calculated or adjusted for the input image input during operation are set based on the image acquired in step S81.

When the tone conversion parameter calculation conditions are set in this manner, the tone conversion parameters corresponding to the input image are individually calculated in accordance with the set tone conversion parameter calculation conditions during operation. Next, an operation procedure for tone-converting a distance image into a low-tone distance image by dynamic conversion will be described with reference to the flowchart in FIG. 133 . First, in step S13301, a distance image is acquired. Also here, the controller unit 2 captures the distance image generated by the distance image generating means 32, as in step S13201 described above. Next, in step S13302, tone conversion parameters are determined based on the distance image, which is the input image. As for the method of adjusting the tone conversion parameters, the methods described above can be used. Furthermore, in step S13303, tone conversion is executed. Finally, inspection processing is executed in step S13304. According to this method, since the gradation conversion parameters can be changed according to the input image, it is possible to flexibly perform gradation conversion for different works and perform accurate inspection. For example, it is possible to inspect the surface of a workpiece with height variations without lowering the accuracy.

Here, as an example of dynamic conversion, a method for optimally creating a low tone distance image after tone conversion by adjusting tone conversion parameters will be described with reference to FIG. Here, there are two methods: a method of dynamically setting one gradation conversion parameter set, and a method of preparing a plurality of gradation conversion parameter sets in advance and dynamically selecting a gradation conversion parameter set. .
(Method of setting 1A one gradation conversion parameter set)

First, a method of dynamically setting one gradation conversion parameter set will be described. In this method, the values of the tone conversion parameters used for tone conversion are adjusted based on the image information of a predetermined area of the plurality of distance images that are the input images, and the adjusted tone conversion parameters are adjusted. , the gradation conversion means 46 executes the gradation conversion processing of the distance image. Here, an example of tone conversion of a 16-tone distance image (image before tone conversion) to an 8-tone low tone distance image (image after tone conversion) will be described. Moreover, as shown in the perspective view of FIG. 134, the workpiece to be inspected is one in which the entire measurement surface of each workpiece is vertically displaced, and the range is 5 mm. Also, the range in the height direction including the thickness and distortion of the entire measurement surface is set to 0.5 mm. Consider an example of inspecting by image processing whether or not there is a flaw on the surface of each work having different surface heights. By using the distance image and converting the distance image into a two-dimensional grayscale image (low-gradation distance image), it is possible to perform inspection with an existing image processing device for two-dimensional images.

First, an area to be inspected on the workpiece is specified in advance as an inspection area by the inspection area setting means. In addition to designating a part of the range image, which is the input image, as the inspection target area, the entire range image can also be specified. In this case, the operation of designating the inspection target area may be omitted.

First, the average distance of the inspected area is determined. Next, the difference (distance range) between the maximum distance and the minimum distance of the inspection target area is obtained. Furthermore, the numerical value obtained by multiplying the distance range by 1.2 is used as the distance range of the post-conversion image. For example, in the workpiece distribution in FIG. 134, if the distance range is 0.5 mm, 0.6 mm obtained by multiplying this by 1.2 is set as the distance range of the post-gradation conversion image. Therefore, the range of ±0.3 mm around the average distance is the measurement range.

Furthermore, the span for the distance image, which is the input image, is obtained so that the range of 0.6 mm (±0.3 mm from the average distance), which is the distance range of the image after tone conversion, has 256 tones. Note that the span can also be a predetermined constant. Also, for the distance image, it is possible to set 0 when the average distance is −0.3 mm or less, and 255 when the average distance is +0.3 mm or more.

In this way, a range necessary for inspection is determined from a plurality of input distance images, and the height information in this range is appropriately maintained even in the low gradation distance image after gradation conversion. You can set gradation conversion parameters. As a result, it is possible to appropriately inspect the presence or absence and position of scratches on each work dispersed in the height direction using an existing image processing device.
(1B Method of preparing multiple gradation conversion parameter sets in advance)

In the above embodiment, a method of appropriately obtaining a gradation conversion parameter set using height information of an acquired range image without losing height information necessary for flaw detection has been described. On the other hand, a method of preparing a plurality of tone conversion parameter sets in advance will also be described below. In this method, a plurality of distance images, which are input images, are subjected to tone conversion using a plurality of preset tone conversion parameters to create post-gradation conversion images, and a predetermined inspection of the distance images is performed. A tone-converted image to be used for inspection is selected based on the image information of the target area.

Such a specific example will be described based on the schematic diagram of FIG. 135 and the flow chart of FIG. In this example as well, as in FIG. 134 described above, consider a case where the entire measurement surface of each workpiece is vertically displaced within a range of 5 mm. Nine gradation-converted images obtained by gradation conversion by changing the conversion center every 0.5 mm are created for such a work. Then, the obtained gradation-converted images are displayed, preferably side by side, on the display means, and the user selects the converted image by the conversion center closest to the average value of the range image, which is the input image. Then, a gradation conversion parameter set applied to the image selected by the user is set. Specifically, as shown in FIG. 136, the distance image of the workpiece is first generated (step S13601), and the tone conversion process is performed multiple times while changing the tone conversion parameter as described above (step S13602). . Then, the user is allowed to select a desired image from simple low gradation distance images that are gradation-converted images (step S13603), adjust the gradation conversion parameters as necessary, and obtain the obtained low gradation distance image (step S13603). The gradation distance image is inspected (step S13604).
(Details of static conversion, dynamic conversion)

Next, the details of these static conversion and dynamic conversion will be described. First, static conversion will be explained. For static conversion, as a specific method for correcting the standard of height information to be left when converting a distance image to a low-gradation distance image,
(A1) Specifying one point to correct at the specified height (distance),
(A2) A three-point specification for correction on a plane can be used.
(A1: One point designation)

One-point designation is a method of tone-converting a distance image into a low-tone distance image based on the height (distance) of a point or area designated by the user. The reference height is, for example, the middle height of the height range (distance range) in which the gradation is converted into the low gradation distance image among the height information of the distance image. Alternatively, the upper limit (highest position for tone conversion) or lower limit (lowest position for tone conversion) of the distance range can be used. The specific procedure for specifying one point is as described with reference to FIGS. 66 to 78 above.
(A2: 3 points designation)

The three-point designation is a method of tone-converting a distance image into a low-tone distance image using a plane obtained from three points designated by the user as a reference plane. Similarly to the one-point designation reference height described above, the reference plane is, for example, the middle height of the height range (distance range) in which the height information of the distance image is gradation-converted into the low-gradation distance image. Alternatively, the upper limit (highest position for tone conversion) or lower limit (lowest position for tone conversion) of the distance range can be used. Specific examples of three-point designation are as explained based on the GUI screens of FIGS. 81 to 85 above.

Here, an example in which specification of a reference plane by static conversion is effective will be described with reference to FIG. In this example, works WK7 of the same height are conveyed on the same plane by the belt conveyor BC. In this case, since there is almost no change in the height of the work WK7, dynamic conversion for changing the reference plane according to the work is unnecessary, and static conversion can be used. In particular, static conversion is faster than dynamic conversion, and benefits from static conversion can be enjoyed.

Also, FIG. 138A is an example of a workpiece whose height variation itself is desired to be suppressed. In this example, the object is inspection processing for detecting, as an abnormality, the case where the cap of the bottle, which is the work WK10 conveyed by the belt conveyor BC, has "floating". In this case, by adopting static conversion, height variations can be detected as in the low gradation distance image shown in FIG. 138B. On the other hand, if dynamic conversion is adopted, as a result of correcting fluctuations, it becomes impossible to detect abnormalities. Therefore, in such applications, static conversion can be suitably used (B: specific example of dynamic conversion)

In the above description, the static conversion is described in which the gradation conversion conditions are specified in advance at the setting stage and the gradation conversion is performed under the specified conditions during operation. Next, a specific example of dynamic conversion for adjusting tone conversion conditions according to an input image to be inspected will be described. First, for dynamic conversion, as a specific method for correcting the standard of height information to be left when converting a distance image to a low-gradation distance image,
(B1) Average height reference for gradation conversion using the average height (average distance) in the average extraction area specified for the input image as the average reference height,
(B2) a plane reference for generating an estimated plane within a specified area of the input image and using this as a reference plane for gradation conversion;
(B3) A free-form surface reference is included in which a free-form surface is generated by removing high-frequency components from the input image and used as a reference surface for gradation conversion.

The average extraction area for defining the average reference height is preset prior to operation (step S83 in FIG. 8 described above). An example of the procedure for designating the average extraction area in step S83 of FIG. 8 is as described based on the GUIs of FIGS. 88 to 92 above. On the other hand, during operation, dynamic conversion is performed according to the procedure described with reference to FIG. For example, a workpiece conveyed on a line is imaged to generate a distance image (step S13301), the average height of the average extraction area set above is calculated (step S13302), and gradation conversion is performed based on this. A low gradation distance image is generated by execution (step S13303), and the obtained low gradation distance image is inspected (step S13304). With this method, even if there is variation in the height direction of the workpiece, the reference plane for gradation conversion can be reset for each workpiece, so accurate inspection can be achieved regardless of variations in the height direction of the workpiece. can.

Next, in the plane reference as well, the reference plane estimation area for determining the reference plane is set in advance (step S83 in FIG. 8 described above), similarly to the average height reference described above. An example of the procedure for designating the reference plane estimation area in step S83 of FIG. 8 has been described based on the GUIs of FIGS. 88 and 92 to 95 above. During operation, dynamic conversion is performed according to the procedure described with reference to FIG. For example, a workpiece conveyed on a line is imaged to generate a distance image (step S13301), the reference plane estimation area set above is extracted, an estimation plane is calculated (step S13302), and the obtained estimation A low gradation distance image is generated by performing gradation conversion with the surface as a reference (step S13303), and the obtained low gradation distance image is inspected (step S13304). With this method, even if there is an inclination or the like on the surface of the workpiece, this can be canceled and accurate inspection can be realized regardless of the inclination of the workpiece.

Finally, the specific method of setting the free-form surface reference is as described with reference to the GUIs of FIGS. 126-32. In the free-form surface reference as well, the necessary conditions for determining the reference surface are set in advance (step S83 in FIG. 8 described above), similarly to the above-described average height reference and the like. During operation, dynamic conversion is performed according to the procedure described with reference to FIG. For example, a workpiece conveyed on a line is imaged to generate a distance image (step S13301), a free-form surface is calculated for the above-set free-form surface target area (step S13302), and the obtained free-form surface is used as a reference to generate a low gradation distance image (step S13303), and the obtained low gradation distance image is inspected (step S13304). With this method, it is possible to obtain the advantage of being able to perform, with high accuracy, work such as the surface inspection of curved workpieces, for which accurate inspection is difficult with conventional methods.

An example in which specification of a reference plane by dynamic conversion is effective will now be described with reference to FIG. In this example, works WK11 having different heights are conveyed on the same plane by the belt conveyor BC. In this case, since the height of each workpiece WK11 is different, the height of the workpiece can be changed by changing the reference surface according to the height of each individual and performing gradation conversion using the average height standard or the like. Optimum gradation conversion is possible even if there is

Also, FIG. 140 shows an example of a workpiece WK12 having a flat surface with minute inclined surfaces and dents. In this example, since the surface of the work WK12 has a minute inclined surface due to the dent DE, there is a possibility that the inclination affects the detection accuracy of the dent. Therefore, by using a plane reference or the like to dynamically obtain a plane for each individual workpiece and use it as a reference plane for gradation conversion, highly accurate inspection such as detection of slight dents and dents becomes possible.

Furthermore, FIG. 141 shows an example of a curved workpiece WK13 with different radii. In this example as well, by using a free-form surface reference or the like to obtain a curved surface for each workpiece and performing gradation conversion using this as a reference surface, it is possible to reduce the influence of variations in the shape of each workpiece.
(Automatic adjustment of gradation conversion parameters)

The procedure for manually setting the gradation conversion parameters based on the image after gradation conversion has been described above. On the other hand, when the gradation conversion parameters are set manually, if the gradation conversion is performed using the initially set gradation conversion parameters, an image suitable for inspection may not be obtained due to changes in the workpiece or environment. In such a case, the user corrects the gradation conversion parameters using the data after the gradation conversion without making fine adjustments by referring to the image. can also be done. In this method, after initial setting of tone conversion, tone conversion is first performed using arbitrary tone conversion parameters as initial values, and then the tone conversion parameters are adjusted.

For example, the gradation conversion condition automatic setting means can function as the gradation conversion condition automatic setting means and the gradation conversion condition manual setting means. That is, the gradation conversion condition automatic setting means sets simple gradation conversion conditions when the gradation conversion means gradation-converts the distance image into the low gradation distance image. Further, while the simple low gradation distance image that has undergone gradation conversion is displayed on the display means based on the simple gradation conversion conditions set by the gradation conversion condition automatic setting means, the gradation conversion condition manual setting means , accepts manual adjustment of tone conversion conditions. As a result, instead of prompting the user to set the gradation conversion conditions abruptly, provisional simple gradation conversion conditions are automatically set and gradation conversion is performed, and then one or more simple gradation conversion conditions are obtained. Since the desired tone conversion conditions can be set while referring to the low tone distance image, even if the user is unfamiliar with the meaning of the tone conversion parameters or is unfamiliar with the settings, such parameters can be automated to some extent. Setting work can be facilitated.

A specific procedure will be described based on the flowchart in FIG. First, in step S14201, processing for creating a distance image is executed. Next, in step S14202, initial tone conversion processing is performed using the initial values of the tone conversion parameters. Further, in step S14203, it is determined whether the obtained tone-converted image is appropriate. If not, the tone-conversion parameters are adjusted again in step S14204, and the process returns to step S14202 to repeat the process. On the other hand, if it is determined in step S14203 that an appropriate tone-converted image has been obtained, the process advances to step S14205 to execute a predetermined inspection.

In the above method, whether or not the gradation conversion parameters are appropriate is determined in step S14203, but this procedure may be omitted. The procedure in this case is shown in FIG. Each procedure is almost the same as the example of FIG. 142 described above. In step S14301, a distance image is created, and in step S14302, initial tone conversion processing is performed. After adjusting the tone conversion parameters in step S14303, a predetermined inspection is performed in step S14304.

As an initial tone conversion method, for example, a method of applying a transformation function f(x, y, z) to an input image, a method of applying a shift or span to an input image, and compressing the result to n tones, or any other method may be used. A method of taking a plane as a reference plane, taking the difference between the input image and the reference plane, applying shift and span, and compressing the result into n gradations, or taking the difference between the input image and the reference image, and applying shift and span. and compressing the result into n gradations.

Next, a specific example of automatically adjusting tone conversion parameters after tone conversion will be described. Methods for automatically adjusting tone conversion parameters include (C1) a method using the median value of the histogram of the range image data after conversion, (C2) a method using the maximum and minimum values of the histogram, and (C3) C1 and C2. A combination, etc. are conceivable.
(C1: method using the median value of the histogram)

First, the method using the median value of the histogram will be described with reference to the flowchart of FIG. First, in step S14401, a histogram of converted distance image data is calculated. Next, in step S14402, the median value of the histogram is obtained. Then, in step S14403, the tone conversion parameters are changed so that the median value becomes a predetermined value. The gradation conversion process is executed again using the gradation conversion parameters changed here. An average value of 2n+1 including n before and after the center may be used as the median. Also, the maximum value and the minimum value may be median values after removing the n largest values and the m smallest values, respectively.
(C2: method using maximum value and minimum value of histogram)

First, a histogram of the range image data after conversion is calculated. Next, the maximum value and minimum value of the histogram are obtained, and the gradation conversion parameter is changed so that the width between the maximum value and the minimum value becomes a predetermined value. Then, the gradation conversion process is executed again using the changed gradation conversion parameters. Here, the maximum value and the minimum value may be the average values of the n largest values and the m smallest values, respectively. Alternatively, the maximum value and the minimum value can be the maximum value and the minimum value after removing n from the largest and m from the smallest, respectively.
(C3: combination of C1 and C2)

You may combine said C1 and C2. That is, after calculating the histogram, the gradation conversion parameter is changed based on the median value and the width of maximum value-minimum value, and gradation conversion is performed again.

In the above method, a low-pass filter can also be applied to the histogram in advance. Alternatively, a histogram may be obtained after applying a low-pass filter to the range image after conversion in advance.

With such tone conversion, a distance image with a high number of tones including height information can be converted into a low tone distance image with a low tone. Since this low-gradation distance image can be processed as a two-dimensional image, even an existing image processing apparatus compatible with two-dimensional images can handle the low-gradation distance image. For example, height information of each pixel included in a distance image obtained by picking up an image of a work for inspecting the presence or absence of flaws is expressed as a grayscale value in binary hexadecimal. Here, when the difference between the distance image and the reference distance image is calculated, the defect information of shallow and small flaws appearing on the surface of the work is aggregated into the lower 8 gradations. Therefore, by reducing, for example, the upper eight gradations using the gradation conversion means, it is possible to greatly compress the amount of information of the differential image while preventing a decrease in detection accuracy. In this way, the gradation conversion means reduces the gradation in the upper half of the gradation when the gradation value of each pixel of the differential image is expressed in gradation, thereby preventing deterioration in the detection accuracy and converting the differential image. A large amount of information can be compressed.
(Partial execution of gradation conversion processing)

Furthermore, the gradation conversion process can be performed only on a part of the range image, not on the entire range image. Specifically, the gradation conversion means executes the gradation conversion process only for the specified inspection target area in the distance image. As a result, the gradation conversion process can be reduced, and the processing load can be reduced and the speed can be increased. As an example, height inspection processing (first inspection processing) and image inspection processing (second inspection processing) are combined using a workpiece WK14 having a shape in which cylinders with different diameters are stacked threefold, as shown in FIG. 145A. Consider a case where a plurality of inspection processes are performed. Here, as height inspection processing, the height of each cylindrical surface of this work WK14 is measured, and as image inspection processing, chips and cracks are detected in the cylindrical portion from the top of the work WK14 up to two stages. Perform a visual inspection. Selection of the inspection process is performed by the inspection process selection means. Further, specific settings for the inspection process selected by the inspection process selection means are made by the inspection process setting means.
(Inspection process selection means)

The inspection process selection means is means for selecting a plurality of inspection processes to be executed by the inspection execution means for the distance image. Here, as shown in FIGS. 44, 56, etc., the inspection process is selected from the initial screen 260 when adding a processing unit. Specifically, the desired inspection process is selected from the list of inspection processes displayed by selecting the "measurement" process from the "addition" submenu of the processing unit. In this example, "area", "pattern search", "Shapetrax2", "edge position", "edge width", "edge pitch", "edge angle", "pair edge", "blemish", "blob", ""GrayscaleBlob","Trend Edge Position", "Trend Edge Width", "Trend Edge Defect", "Grayscale Inspection", "Color Inspection", "OCR", "2D Code Reader", "1D Code Reader", "High Select the desired inspection process from "Length measurement".
(Inspection processing setting means)

On the other hand, the inspection process setting means sets the details of each inspection process selected by the inspection process selection means. Here, as shown in FIGS. 46, 63, 78, etc., each setting item is configured to be individually set from each button arranged in the setting item button area 112. FIG. In this example, specific setting items are arranged as setting item buttons. It includes a "preprocessing" button 117, a "detection condition" button 118, a "detailed setting" button 119, a "determination condition" button, a "display setting" button, a "save" button, and the like. Thus, the setting item button area 112 functions as inspection processing setting means.
(Height inspection process)

FIG. 145B shows an example of the distance image obtained for the work as shown in FIG. 145A. This distance image is expressed in 16 gradations before gradation conversion. When performing height measurement (height inspection processing) on such a distance image, high-accuracy height information can be used without performing gradation conversion, and the high-precision height information of 16 gradations can be used. inspection can be realized. Specifically, as shown in FIG. 145C, in order to measure the height of each surface of the work, inspection areas are set on three surfaces of the work, and the height of each inspection area is set to 16 floors. Measure in tune.
(Image inspection processing)

On the other hand, when performing measurement (image inspection processing) that does not use height information in inspection processing, high-gradation information is not required, and processing with a lower-gradation range image reduces the load. For this reason, the distance image of high gradation is subjected to gradation conversion to obtain the distance image of low gradation, and then the processing is performed. Here, it is not necessary to convert the entire distance image into a low-gradation distance image, and it is sufficient to perform the gradation conversion only on the object for which the image inspection process is to be performed. That is, the area to be subjected to gradation conversion is an inspection area set for image inspection processing that does not use height information, among one or more set inspection areas. In the example shown in FIG. 145D, similarly to FIG. 145B, an inspection target area for image inspection processing is set for the high gradation distance image before gradation conversion. In this example, in order to detect chipping or the like in the top two stages of the workpiece, an inspection target area is set so as to surround the cylindrical shape of the second stage. In other words, the cylindrical shape on the third stage is not subject to visual inspection, so this portion is set to be excluded from the inspection subject area. Then, gradation conversion is performed on this image processing inspection target area. The resulting low tone distance image is shown in FIG. 145E. The low gradation distance image after gradation conversion shown in this figure is expressed in 8 gradations, and although height information is somewhat lost compared to the high gradation distance image, the presence or absence of chipping can be detected. Sufficient accuracy is maintained for visual inspection applications, so there is no problem with image inspection processing. On the other hand, the area requiring tone conversion is greatly reduced compared to FIG.

In this way, the presence or absence of gradation conversion is selected according to the inspection process. For example, among the inspection processes displayed in the flow display area 261 in FIG. High-precision height measurement is realized by using the height information of the scale. On the other hand, in other inspection processes, such as the "numerical calculation" processing unit, the "area" processing unit, and the "blob" processing unit 267, the gradation-converted 8-gradation low-gradation image is used. Although not shown, when inspection processing is performed on a luminance image (for example, color inspection on a color luminance image), since height information is not necessary in the first place, gradation conversion is naturally unnecessary.
(Display of gradation conversion condition setting means 43)

For image inspection processing that requires such tone conversion processing, tone conversion conditions for performing suitable tone conversion are set by the tone conversion condition setting means 43 . At this time, the gradation conversion condition setting means 43 can be set only when gradation conversion processing is required. By disabling setting, the user can smoothly set only the necessary items without being distracted by unnecessary setting work. Therefore, in the present embodiment, when setting an inspection process that does not require height information of an image other than the height inspection process, the steps for setting the gradation conversion parameters for performing gradation conversion of the distance image are set. While the tone conversion condition setting means 43 is displayed, the gradation conversion condition setting means 43 is not displayed when setting the height inspection process.

A specific setting procedure will be described with reference to the flowchart in FIG. First, in step S14601, an inspection process is selected. Here, the inspection process to be executed by the inspection execution means is selected from the "measurement" menu, which is the inspection process selection means. Next, in step S14602, it is determined whether or not the inspection process selected in step S14601 requires tone conversion. If the inspection process requires tone conversion, the process proceeds to step S14603 to enable tone conversion. At the same time, the gradation conversion setting means is displayed in the setting item.

For example, when image inspection processing is performed on the work in FIG. 145A by the "area" processing unit, as shown in FIG. is displayed. When the "height extraction" button 116 is pressed, a height extraction setting screen is displayed as shown in FIG. From this screen, the user can set the conditions necessary for tone conversion processing.

In the example of FIG. 148, a low gradation distance image that has undergone gradation conversion according to the gradation conversion conditions set in the operation area on the right side is displayed in the image display area. As a result, the user can visually confirm whether or not the desired inspection result can be obtained under the current tone conversion conditions, and thus has the advantage of being able to easily adjust the tone conversion conditions. In particular, in the example of FIG. 148, the area to be inspected is set to be circular and arranged so as to surround the second columnar workpiece. In addition, by using dynamic conversion (real-time extraction) as the detection method and setting the plane reference as the calculation method, the top surface at the center of the workpiece and its outer peripheral surface are detected. It detects scratches and dents on the surface of the The parameter display of the reference plane detected according to the gradation conversion conditions can also be switched and displayed by operating the console.

Furthermore, in the example of FIG. 148, instead of displaying a low gradation distance image obtained by performing gradation conversion on the entire workpiece displayed on the image display area, the gradation conversion is performed only within the set inspection target area. The image of the low gradation distance image is displayed. In other words, the original distance image is displayed as it is for the image other than the inspection target area. Thereby, it is possible to visually indicate to the user that the gradation conversion is not performed on the entire input image, but is performed only within a part of the inspection target area. When the setting of the gradation conversion conditions is completed in this way, the process advances to step S14604.

On the other hand, in the case of inspection processing that does not require tone conversion, the process jumps to step S14604 without going through step S14603. In this case, the gradation conversion condition setting means is not displayed on the inspection processing setting screen. For example, when height inspection processing is performed by the "height measurement" processing unit, as shown in FIG. "Extract height" button is not displayed. This allows the user to recognize that there is no need to set conditions related to gradation conversion that are unnecessary for height inspection processing. Alternatively, by making it impossible to set tone conversion conditions for this inspection process, it is possible to avoid confusion caused by unnecessary settings.

Then, in step S14604, inspection processing is set. Finally, in step S14605, it is determined whether or not the setting of all inspection processes has been completed. If not, the process returns to step S14601 and repeats from the inspection process selection, and if completed, the process is completed.

On the other hand, the procedure during operation will be described based on the flowchart of FIG. First, in step S14901, a distance image is input as an input image. Next, in step S14902, initialization is performed. Here, 1 is set to n. Furthermore, in step S14903, the n-th inspection process is executed. Further, in step S14904, it is determined whether or not n<N (N is the number of set inspection processes).If YES, n is incremented by 1 in step S14905, and the process returns to step S14903 to perform the next inspection process. Repeat the process of executing On the other hand, if not n<N, the process is completed assuming that all inspection processes have been completed. In this way, all inspection processes are executed sequentially.

Here, the execution of the inspection process in step S14903 will be described in detail. First, if the inspection process requires tone conversion, as shown in the flowchart of FIG. In contrast, gradation conversion is performed. After that, in step S15002, inspection processing is executed for the low tone distance image after tone conversion. On the other hand, in the case of inspection processing that does not perform gradation conversion, as shown in the flowchart of FIG. step S15101).
(Hide function for image selection)

Also, at the time of setting, it is possible to restrict the selection of images for which inspection processing is to be set according to the type of inspection processing. That is, if the inspection process selected by the inspection process selection means can be performed on either the distance image or the luminance image, it is possible to call these distance images or luminance images as registered images. On the other hand, there are inspection processes that can be performed on range images but not on luminance images. For example, height measurement processing is performed on a distance image having highly accurate height information. This cannot be done for normal luminance images that do not have height information. Therefore, inspection processing that can be performed only on an image having height information, that is, a distance image, such as height measurement processing, can select only the distance image at the time of calling the registered image at the time of setting, and conversely, Intensity images are made unselectable. As a result, it is possible to prohibit or eliminate setting work that is originally impossible, such as erroneously setting the height measurement process for the luminance image, eliminate wasteful settings, and improve user operability.

The procedure for setting inspection processing conditions will be described below with reference to the flowchart of FIG. First, in step S15201, an inspection process is selected. Next, in step S15202, it is determined whether the inspection process selected in step S15201 is an inspection process capable of designating either a range image or a luminance image, or an inspection process capable of designating only a range image. Here, in the case of inspection processing in which either the distance image or the luminance image can be specified, the flow advances to step S15203 to select either the distance image or the luminance image.

For example, as shown in FIG. 61, consider the case where the "area" processing unit is selected as inspection processing. In this case, either the distance image or the luminance image can be specified. Therefore, as an inspection processing setting item in the "area" processing unit, for example, when the "image setting" button 114 in FIG. 62 is pressed to display the image setting screen 380 shown in FIG. You can choose. Here, an image to be displayed in the second image display area 121 can be selected in the “display image” selection field 124 provided in the image selection field 382 . Also, in the image setting field 384, an input image and a registered image can be selected. From this image setting column 384, an input image can be specified by an image variable. Here, when the "input image" selection field 386 is selected, the image variable selection screen 390 of FIG. 154 is displayed, and a list of selectable images is displayed. On the image variable selection screen 390, one of images captured by a plurality of imaging means (cameras) connected to the 3D image processing apparatus can be selected. Here, a different image variable is assigned to each imaging means, and the imaging means and the image variables are associated with each other. For example, the image variable "&Cam1Img" for the range image captured by camera 1, the image variable "&Cam2Img" for the range image captured by camera 2, the image variable "&Cam3Img" for the range image captured by camera 3, and the image variable "&Cam3Img" for the range image captured by camera 4. is assigned an image variable "&Cam4Img". Furthermore, an image variable “&Cam1GrayImg” is assigned to the luminance image captured by camera 1 . The user selects a desired image from the image variable selection screen 390 . In this way, since the "area" processing unit can specify either the distance image or the luminance image, not only the distance image but also the luminance image are included in the options and displayed.

Furthermore, if necessary, the image variables listed on the image variable selection screen 390 can be sorted and displayed so that the user can easily select a desired image.

On the other hand, if the inspection process can specify only the distance image, the process advances to step S15204 to select the distance image. That is, the selection of the luminance image is disabled. For example, as shown in FIG. 45, when the "height measurement" processing unit 266 is selected as inspection processing, only the distance image can be selected. Therefore, as shown in FIG. 46, when the "image setting" button 114 is selected as an inspection processing setting item in the "height measurement" processing unit, an image setting screen 380 is similarly displayed, and an image can be selected. . Here, if the "input image" selection field 386 is similarly selected, the image variable selection screen 390 of FIG. 155 is displayed, and a list of selectable images is displayed. In this image variable selection screen 390, unlike FIG. 154, the luminance image is not displayed as an option, and only the distance image is displayed as an option. With this configuration, the user can avoid erroneously selecting a luminance image that does not have height information when measuring the height, and provides an operating environment with little confusion.

After the image is selected in this manner, the flow advances to step S15205 to set inspection processing conditions for the selected image. In this way, for each inspection process, it is determined whether the inspection process can specify either the range image or the luminance image, or the inspection process can specify only the range image. Prescribing the types of selectable images avoids setting errors and contributes to user convenience.
(Hiding function for inspection processing)

The above is an example in which the user selects an inspection process first, and then restricts the selectable images when selecting an image for which the inspection process is to be set according to the type of the selected inspection process. explained. Conversely, after selecting an image first, it is also possible to restrict the types of inspection processes that can be performed on this image. That is, first, the image selection means selects a distance image or a luminance image, and then the inspection process selection means selects one or more inspection processes executed by the inspection execution means. Only inspection processing that can be executed for each image can be selected according to the type of image. With this configuration, it is possible to avoid a situation in which an unsettable inspection process is erroneously selected for a selected image or an inspection process condition related to this inspection process is set.

The procedure for setting inspection processing conditions by this method will be described below with reference to the flowchart of FIG. First, in step S15601, an image is selected. Here, either the distance image or the luminance image is selected. Next, in step S15602, it is determined whether the image selected in step S15601 is a distance image or a luminance image. In the case of the distance image, the process advances to step S15603 to allow the user to select an inspection process executable for the distance image using the inspection process selection means. On the other hand, in the case of a luminance image, the process advances to step S15604 to allow the user to select an inspection process executable for the luminance image using the inspection process selection means. When the inspection process is selected in this manner, the flow advances to step S15605 to set inspection process conditions for the selected inspection process from the inspection process condition setting means.

For example, as shown in FIG. 157, consider the case of obtaining a luminance image and a distance image for a work. As described above, when image registration is performed, a luminance image and a distance image are registered at the same time. The user selects one of these images, selects inspection processing for the next selected image, and sets the inspection processing conditions. For example, when a luminance image is selected, a tool corresponding to inspection processing performed on the luminance image is added as shown in FIG. Here, as in FIGS. 44 and 56 described above, when an image is selected and Add is selected from the displayed submenu, a list of inspection processes that can be performed on the luminance image is displayed. For example, as the "measurement" processing, an "area" processing unit, a "blemish" processing unit, and a "blob" processing unit 267 are displayed as options. At this stage, inspection processes that cannot be performed on luminance images, such as the "height measurement" processing unit, are not displayed. This prevents the user from accidentally selecting an inspection process that cannot be set to the luminance image. It should be noted that while displaying a list of all the measurement processes, it is also possible to make them unselectable by, for example, graying them out. When the inspection process is selected, the processing unit is determined, and then the inspection process conditions necessary for this inspection process are set by the inspection process condition setting means.

Also, when a distance image is selected, tools corresponding to inspection processing that can be executed on the distance image are added as shown in FIG. Here too, when the range image is selected and Add is selected from the displayed submenu, a list of inspection processes that can be performed on the luminance image is displayed. For example, as the "measurement" processing, in addition to the "area" processing unit, the "blemish" processing unit, and the "blob" processing unit 267, the "height measurement" processing unit is also displayed as an option. When the user selects a desired inspection process, the processing unit is determined, and the inspection process conditions required for this inspection process are set by the inspection process condition setting means.

By associating an inspection processing tool with an image in this way, it is possible to physically eliminate combinations of images and inspection processing that cannot be selected, and easily avoid setting errors by the user.

INDUSTRIAL APPLICABILITY The three-dimensional image processing apparatus and the three-dimensional image processing method of the present invention can be used for inspection apparatuses and the like that utilize the principle of triangulation.

100, 100', 200, 300, 400... three-dimensional image processing apparatus 1, 1B, 1C, 1D... head section 2, 2'... controller section 3... input means 4... display means 10, 10A, 10B... imaging means 20 20A First projector 20B Second projector 21 Measurement light source 22 Pattern generation unit 23, 24, 25 Lens 30 Head side control unit 31 Head side calculation unit 32 Distance image generation means 34 Filter processing section 36 Head-side communication means 36A Controller connection interface 36B PC connection interface 38 Storage means 38a Distance image storage section 38b Luminance image storage section 41 Controller-side setting means 42 Controller-side communication means 43 Gradation conversion condition setting means 44 Reference plane setting means 45 Spatial coding switching means 46 Gradation conversion means 47 Interval equalization processing setting means 48 Light projection switching means 49 Shutter speed Setting means 50 Inspection execution means 51 Main control unit 52 Controller side connection unit 53 Operation input unit 54 Display control unit 55 Communication unit 56 RAM
57 Controller-side storage means 58 Auxiliary storage means 59 Output section 60 Image processing section 62 Abnormal point highlight means 64 Image search means 66 Real-time update means 70 PLC
110... Three-dimensional image processing program GUI screen 111... First image display area 112... Setting item button area 113... "Image registration" button 114... "Image setting" button 115... "Area setting" button 116... "Extract height" Button 117 -- "Preprocessing" button 118 -- "Detection condition" button 119 -- "Detailed setting" button 120 -- Inspection target area setting screen 121 -- Second image display area 122 -- Operation area 124 -- "Display image" selection field 126 -- "Measurement area" setting field 128 --- "Edit" button 130 --- Measurement area editing screen 140 --- Height extraction selection screen 142 --- Extraction method selection means 144 --- "Extraction" button 145 --- "Extraction region" designation field 146 --- Point pointer 147..."Extraction area" button 148...Extraction area setting dialog 149...Extraction area selection field 150...One point designation screen 152..."Z height" display field 154...Enhancement method setting field 156...Gain adjustment field 158..."Detailed setting '' button 160...Enhancement method detail setting screen 162..."Extraction height" setting field 164...Noise removal setting field 166...Invalid pixel designation field 170...Three-point designation screen 172...Height extraction display field 174...Three-point designation "Details Setting" button 180 --- Three-point specification detailed setting screen 182 --- "Extraction height" setting field 190 --- Height dynamic extraction setting screen 192 --- "Calculation method" selection field 194 --- "Mask area" button 196 --- "Detailed setting" Button 210 Average height reference setting screen 220 Mask area setting screen 222 "Detailed setting" button 230 Average height reference detailed setting screen 240 Plane reference detailed setting screen 250 Free curved surface reference setting screen 252 Extraction size Designation field 260... Initial screen 261... Flow display area 262... Third image display area 263... ``Imaging'' processing unit 264... ``Shapetrax2'' processing unit 265... ``Position correction'' processing unit 266... ``Height measurement'' processing unit 266B...second "height measurement" processing unit 267..."blob" processing unit; 267B..."color inspection" processing unit 268..."edit" button 269...imaging setting menu 270...image registration screen 271..."camera selection" Column 272 --- "Register" button 280 --- Imaging setting screen 282 --- "Detailed setting" button 284 --- "Imaging setting" button 290 --- Three-dimensional measurement setting screen 292 --- "Display by continuous update" column 294 --- Shutter speed setting column 295 Numeric value display field 296 Gradation range setting field 310 Preprocessing setting field 312 Unmeasurable standard setting field 314 Equal interval processing setting field 316 Spatial code setting field 318 Projector selection setting field 322 --- "Display image" selection field 324 --- "Edit" button 326 ---Extraction region edit dialog 330 --- Mask region setting field 332 --- "Edit" button 334 --- Mask region edit dialog 340 --- Filter processing setting screen 350 Binary level setting screen 360 Judgment condition setting screen 370 First submenu 372 Second submenu 373 "Measurement" menu 380 Image setting screen 382 Image selection field 384 Image setting field 386 " Input image" selection field 390 Image variable selection screen 460 Height measurement setting screen 620 Area setting screen 630 Free curved surface reference setting screen 632 Extraction direction designation field 640 Detailed setting screen WK, WK7, WK8, WK9 , WK10, WK11, WK12, WK13, WK14 Work PC Personal computer SA Search target area SI Syringe icon ROI Measurement area BC Belt conveyor DE Hit marks

Claims (11)

A three-dimensional image processing apparatus capable of acquiring a distance image including height information of an inspection object and performing image processing based on the distance image,
an imaging means for imaging an image of an inspection object;
distance image generation means capable of generating a distance image based on the image captured by the imaging means;
Reference plane setting means for setting a reference plane that serves as a reference when tone-converting the distance image generated by the distance image generation means into a distance image having a lower tone number than the distance image. When,
Using the height of the reference plane set by the reference plane setting means as a reference, the distance image has height information of a lower gradation number than the gradation number of the distance image. gradation conversion means for performing gradation conversion to a low gradation distance image replaced with a value;
extraction direction input means for designating a direction for extracting a local shape change when the tone conversion is performed by the tone conversion means;
inspection execution means for performing a predetermined inspection process on the low gradation distance image gradation-converted by the gradation conversion means;
with
The reference plane setting means is configured to calculate a curved surface from which high-frequency components are removed from the distance image generated by the distance image generating means, and to set the calculated curved surface as a reference plane ,
A three-dimensional image processing apparatus in which the extraction direction input means can designate any of the X direction, Y direction, or XY direction of the range image as the direction for extracting the local shape change . The three-dimensional image processing device according to claim 1,
The distance image generation means sequentially generates a distance image of the inspection object based on the input image of the inspection object captured by the imaging means during operation of performing three-dimensional image processing in the three-dimensional image processing device, The reference plane setting means is configured to change the reference plane according to the height of the object to be inspected by resetting the reference plane based on the height information of the distance images that are sequentially generated. Original image processing device. The three-dimensional image processing device according to claim 1 ,
When the gradation conversion means performs gradation conversion, the distance image is once reduced, filtered and then enlarged to create a free-form surface image, and the difference between the free-form surface image and the distance image is obtained. It extracts the shape change of the distance image,
A three-dimensional image processing apparatus for reducing the distance image only in the direction specified by the extraction direction input means when reducing the distance image. The three-dimensional image processing apparatus according to any one of claims 1 to 3 , further comprising a light projecting means for projecting light onto the inspection object by a light section method,
The three-dimensional image is configured such that the image capturing means captures an image projected by the light projecting means, and the distance image generating means generates a distance image by synthesizing profiles obtained by a light section method. processing equipment. The three-dimensional image processing apparatus according to any one of claims 1 to 3 , further comprising
a light projecting means for projecting incident light from an oblique direction with respect to the optical axis of the imaging means as structured illumination having a predetermined projection pattern;
the imaging means captures a plurality of pattern projection images by acquiring reflected light projected by the light projecting means and reflected by the inspection object;
The three-dimensional image processing apparatus according to claim 1, wherein the distance image generating means generates a distance image based on a plurality of pattern projection images captured by the imaging means. The three-dimensional image processing device according to any one of claims 1 to 5, further comprising
extraction height setting means for designating height information to be extracted based on the reference plane when the gradation conversion is performed by the gradation conversion means;
The extraction height setting means can specify, as the height information to be extracted, a higher side, a lower side, or both higher and lower than the reference plane,
when the higher side is specified, the tone conversion means performs tone conversion so that the reference plane becomes the lower limit of the distance range of the low tone distance image;
when the lower side is specified, the tone conversion means performs tone conversion so that the reference plane becomes the upper limit of the distance range of the low tone distance image;
The three-dimensional image processing device according to claim 1, wherein, when both the high and low are specified, the tone conversion means performs tone conversion so that the reference plane is in the middle of the distance range of the low tone distance image. A three-dimensional image processing method for acquiring a distance image containing height information of an inspection object and performing image processing based on the distance image,
a step of projecting incident light from an oblique direction with respect to the optical axis of the imaging means as structured illumination of a predetermined projection pattern by the projection means;
a step of capturing a plurality of pattern projection images by capturing the light projected by the light projecting means and reflected by the object to be inspected;
a step of generating a distance image by a distance image generating means based on the plurality of pattern projection images captured by the imaging means;
As a gradation conversion parameter constituting a gradation conversion condition for gradation conversion of the distance image generated by the distance image generating means into a distance image having a lower gradation number than the distance image, setting a reference plane as a reference for tone conversion;
Using the height of the set reference plane as a reference, the distance image has a number of gradations lower than the number of gradations of the distance image. a step of tone conversion to a tone distance image;
a step of prompting to specify any one of the X direction, Y direction, or XY direction of the range image as a direction for extracting local shape changes when setting the reference plane;
and executing a predetermined inspection process on the tone-converted low tone distance image,
A three-dimensional image processing method for calculating a curved surface from which high-frequency components are removed from a range image, and using the calculated curved surface as a reference plane for tone conversion into a low-level range image. The three-dimensional image processing method according to claim 7 ,
During the tone conversion, the distance image is first reduced, filtered, enlarged to create a free-form surface image, and the difference between the free-form surface image and the distance image is taken to extract the shape change of the distance image. and
A three-dimensional image processing method for reducing the distance image only in a direction designated by an extraction direction input means for designating a direction for extracting a local shape change in the gradation conversion when reducing the distance image. The three-dimensional image processing method according to claim 7 or 8, further comprising
a step of prompting the user to specify any one of a higher side, a lower side, or both higher and lower than the reference plane as height information to be extracted based on the reference plane when converting the gradation;
when the high side is specified, tone conversion is performed so that the reference plane is the lower limit of the distance range of the low tone distance image;
when the lower side is specified, tone conversion is performed so that the reference plane becomes the upper limit of the distance range of the low tone distance image;
A three-dimensional image processing method for tone-converting such that when both the height and the low are specified, the reference plane is in the middle of the distance range of the low-tone distance image. A three-dimensional image processing apparatus capable of acquiring a distance image including height information of an inspection object and performing image processing based on the distance image,
an imaging means for imaging an image of an inspection object;
distance image generation means capable of generating a distance image based on the image captured by the imaging means;
Reference plane setting means for setting a reference plane that serves as a reference when tone-converting the distance image generated by the distance image generation means into a distance image having a lower tone number than the distance image. When,
Using the height of the reference plane set by the reference plane setting means as a reference, the distance image has height information of a lower gradation number than the gradation number of the distance image. gradation conversion means for performing gradation conversion to a low gradation distance image replaced with a value;
extraction height setting means for designating height information to be extracted based on the reference plane when the gradation conversion is performed by the gradation conversion means;
inspection execution means for performing a predetermined inspection process on the low gradation distance image gradation-converted by the gradation conversion means;
with
The reference plane setting means is configured to calculate a curved surface from which high-frequency components are removed from the distance image generated by the distance image generating means, and to set the calculated curved surface as a reference plane,
The extraction height setting means can specify, as the height information to be extracted, a higher side, a lower side, or both higher and lower than the reference plane,
When the high side is specified, the extraction height setting means performs tone conversion so that the reference plane becomes the lower limit of the distance range of the low tone distance image,
When the lower side is specified, the extraction height setting means performs tone conversion so that the reference plane becomes the upper limit of the distance range of the low tone distance image,
In the three-dimensional image processing device, when both the high and low are specified, the extraction height setting means carries out tone conversion so that the reference plane is in the middle of the distance range of the low tone distance image. A three-dimensional image processing method for acquiring a distance image containing height information of an inspection object and performing image processing based on the distance image,
a step of projecting incident light from an oblique direction with respect to the optical axis of the imaging means as structured illumination of a predetermined projection pattern by the projection means;
a step of capturing a plurality of pattern projection images by capturing the light projected by the light projecting means and reflected by the object to be inspected;
a step of generating a distance image by a distance image generating means based on the plurality of pattern projection images captured by the imaging means;
As a gradation conversion parameter constituting a gradation conversion condition for gradation conversion of the distance image generated by the distance image generating means into a distance image having a lower gradation number than the distance image, setting a reference plane as a reference for tone conversion;
Using the height of the set reference plane as a reference, the distance image has a number of gradations lower than the number of gradations of the distance image. a step of tone conversion to a tone distance image;
a step of prompting the user to specify any one of a higher side, a lower side, or both higher and lower than the reference plane as height information to be extracted based on the reference plane when performing the tone conversion;
a step of performing a predetermined inspection process on the tone-converted low tone distance image;
including
During the tone conversion, a curved surface is calculated by removing high-frequency components from the distance image, and the calculated curved surface is used as a reference surface for tone conversion into a low-tone distance image,
when the high side is specified, tone conversion is performed so that the reference plane is the lower limit of the distance range of the low tone distance image;
when the lower side is specified, tone conversion is performed so that the reference plane becomes the upper limit of the distance range of the low tone distance image;
A three-dimensional image processing method for tone-converting such that when both the height and the low are specified, the reference surface is in the middle of the distance range of the low-tone distance image.

Images