JP6913705B2 - 3D image processing equipment, 3D image processing methods and 3D image processing programs, computer-readable recording media and recording equipment - Google Patents

Description

The present invention relates to a three-dimensional image processing apparatus, a three-dimensional image processing method and a three-dimensional image processing program, and a computer-readable recording medium and recording device.

Many production sites such as factories have introduced image processing devices that automate and speed up inspections that rely on human visual inspection. The image processing device takes an image of a work flowing through a production line such as a belt conveyor with a camera, and executes measurement processing such as edge detection and area calculation of a predetermined area using the obtained image data. Then, based on the processing result of the measurement process, inspections such as chipping detection of the work and position detection of the alignment mark are performed, and a determination signal for determining the presence or absence of chipping or misalignment of the work is output. As described above, the image processing device may be used as one of the FA sensors.

The image to be measured by the image processing device used as the FA sensor is mainly a luminance image that does not include height information. Therefore, regarding the chipping detection of the work described above, it is good at stably detecting the two-dimensional shape of the chipped portion, but it is stable in the three-dimensional shape that is difficult to appear as a luminance image, such as the degree of denting of scratches. Is difficult to detect. For example, it is conceivable to devise the type and lighting direction of the illumination that illuminates the work during inspection, detect the shadow caused by the dent of the scratch, and indirectly detect the three-dimensional shape. Clear shadows are not always detected. In order to prevent erroneous judgment of defective products as non-defective products when unclear shadows are detected, for example, if the judgment threshold is biased to the safe side, a large number of non-defective products will be judged as defective products and the yield will deteriorate. There is a risk of inviting.

Therefore, the height is expressed two-dimensionally by using not only the luminance image whose pixel value is the shading value according to the amount of light received by the camera but also the shading value which is the pixel value corresponding to the distance between the camera and the work. A visual inspection using a distance image (see, for example, Patent Document 1) can be considered.

An example of the three-dimensional image processing apparatus is shown in the schematic diagram of FIG. 160. This three-dimensional image processing device is a controller that is connected to a head unit provided with an imaging means such as a light receiving element, is connected to the head unit, is sent image data captured by the head unit, and generates a distance image from the acquired image data. It is composed of parts.

Here, the principle of triangular ranging will be described with reference to FIG. 160. The head unit has a preset angle α between the optical axis of the incident light emitted from the light projecting unit 110 and the optical axis of the reflected light incident on the light receiving unit 120 (the optical axis of the light receiving unit 120). .. Here, when the work is not placed on the work, the incident light emitted from the light projecting unit 110 is reflected by the point O on the mounting surface of the work and is incident on the light receiving unit 120. On the other hand, when the work is placed on the work, the incident light emitted from the light projecting unit 110 is reflected by the point A on the surface of the work, becomes reflected light, and is incident on the light receiving unit 120. Then, the distance d between the point O and the point A in the X direction is measured, and the height h of the point A on the surface of the work is calculated based on this distance d.

The three-dimensional shape of the work is measured by calculating the heights of all the points on the surface of the work by applying the measurement principle of triangular distance measurement described above. In the pattern projection method, in order to irradiate all points on the surface of the work with incident light, the incident light is emitted from the light projecting unit 110 according to a predetermined structured pattern, and the reflected light reflected on the surface of the work is received and received. Efficiently measure the three-dimensional shape of the work based on the plurality of pattern images.

As such a pattern projection method, a phase shift method, a spatial coding method, a multi-slit method, and the like are known. By the three-dimensional measurement process using the pattern projection method, the projection pattern is changed, the imaging is repeated a plurality of times in the head unit, and the image is sent to the controller unit. The controller unit can perform a calculation based on the pattern projection image sent from the head unit to obtain a distance image having the height information of the work.

On the other hand, in the existing image processing apparatus, a luminance image in which the luminance is mainly used as a pixel value is used. For example, there is a system in which a monochrome camera captures a luminance image expressing the appearance of the work in optical shades on the work transported on the line, and inspects the work by image processing. In such a situation, if it is attempted to newly introduce a three-dimensional measuring device and a processing device that processes three-dimensional data (point cloud data) output from the three-dimensional measuring device, a considerable cost is required.

Therefore, if the height information is expressed as a distance image expressed as the shading of the image, the image data to be handled is the same as the existing luminance image. Therefore, even in the equipment of the image processing device using the existing luminance image, the distance You will be able to handle images.

In this case, in the conventional luminance image, an image having a relatively low gradation number is used. For example, many images in which information per pixel is expressed in eight gradations are used. On the other hand, for image data having height information, a large number of gradations (for example, 16 gradations) is required to express the height information. Therefore, in order to enable processing of a distance image by an existing image processing device that handles a low-gradation image, it is necessary to convert the multi-gradation distance image into a relatively low-gradation distance image. .. (Hereinafter, an image obtained by converting a multi-gradation distance image into a low-gradation distance image is referred to as a "low-gradation distance image", and a conventional luminance image having luminance information is referred to as a "luminance image".)

In this way, by using a gradation-converted low-gradation distance image in addition to the previously used luminance image, an image such as a conventional shape extraction is performed when inspecting whether the work is normal or abnormal. Not only the processing but also the processing such as height measurement using the height information can be used in the three-dimensional image processing apparatus.

In this case, gradation conversion that converts the height information of high gradation (for example, 16 gradations) into the same number of gradations (for example, 8 gradations) as the existing luminance image is indispensable. At this time, if the gradation conversion parameter is not set appropriately, the height information required for the inspection may be lost, and an image suitable for the inspection may not be acquired.

Therefore, it is conceivable to appropriately set the gradation conversion parameter by using the height information of the work to be inspected and determine the reference height of the gradation conversion.

However, when there are variations in the height direction of the work to be inspected by the user, it may not be possible to obtain an image suitable for inspection only by performing gradation conversion using the initially specified gradation conversion parameter.

Japanese Patent No. 4969478

The present invention has been made to solve such conventional problems. A main object of the present invention is a three-dimensional image processing device, a three-dimensional image processing method, and a third order that suppress a loss of height information and suppress a decrease in accuracy when converting a distance image into a low-gradation distance image. An object of the present invention is to provide an original image processing program and a computer-readable recording medium.

Means for Solving Problems and Effects of Invention

In order to achieve the above object, according to the three-dimensional image processing apparatus according to the first aspect of the present invention, a distance image including height information of an inspection object is acquired, and an image based on the distance image is acquired. A three-dimensional image processing device capable of performing processing, an imaging means for capturing an image of an object to be inspected, and a distance image generating means capable of generating a distance image based on the image captured by the imaging means. As a gradation conversion parameter that constitutes a gradation conversion condition when the distance image generated by the distance image generation means is gradation-converted to a distance image having a gradation number lower than the gradation number of the distance image. A reference plane setting means for setting a reference plane as a reference for performing the gradation conversion, and the distance image of the distance image based on the height of the reference plane set by the reference plane setting means. A gradation conversion means for converting a gradation into a low gradation distance image in which the height information of the distance image having a gradation number lower than the number of gradations is replaced with a grayscale value of the image, and the gradation conversion means. An inspection execution means for executing a predetermined inspection process and a low gradation distance image to be gradation-converted by the gradation conversion means for displaying the gradation-converted low-gradation distance image in during operation of a display unit, before SL performs 3D image processing by the 3D image processing apparatus, the distance of said distance image generating means, the inspection object based on the input image of the inspection object which is sequentially input from the outside image sequentially generate said reference plane setting means, based on the sequential height information of the generated range image, by resetting the reference plane, to change the reference surface in accordance with the height of the test object The gradation conversion means converts each distance image into the low gradation distance image with reference to the height of the reference plane reset for each distance image generated by the distance image generation means. It is configured to perform conversion, and when the gradation conversion means performs gradation conversion, it can be configured to perform position correction using the distance image.

Further, according to the three-dimensional image processing apparatus according to the second side surface, a light projecting means for projecting incident light from an oblique direction with respect to the optical axis of the imaging means as structured illumination of a predetermined projection pattern is provided. The imaging means is provided, the light emitting means is projected, the reflected light reflected by the inspection object is acquired, and a plurality of pattern projection images are captured, and the distance image generating means is the imaging means. It can be configured to generate a distance image based on a plurality of pattern projection images captured in. With the above configuration, inspection processing such as height measurement using a distance image using pattern projection can be realized.

Further, according to the three-dimensional image processing apparatus according to the third aspect, in addition to any of the above configurations, the reference plane setting means is externally used during the operation of performing the three-dimensional image processing by the three-dimensional image processing apparatus. The reference plane is set based on the input image of the inspection object to be input, and the reference plane setting means calculates a curved surface from the distance image displayed on the display means, and the calculation is performed. It can be configured to set the curved surface as a reference plane. With the above configuration, changes in the inspection object in the direction specified by the extraction direction input means can be extracted, and the shape that does not change in the specified direction, for example, a groove continuous in the longitudinal direction, is ignored and accurate. Shape extraction is possible.

Furthermore, according to the three-dimensional image processing apparatus according to the fourth aspect, the extraction direction input means can extract a local shape change in any of the X direction, the Y direction, and the XY direction of the distance image. Can be specified.

Furthermore, according to the three-dimensional image processing apparatus according to the fifth aspect, the gradation conversion means reduces the distance image once, performs filtering processing, and then enlarges the free curved image at the time of gradation conversion. The shape change of the distance image is extracted by creating and taking the difference between the free curved surface image and the distance image, and when the distance image is reduced, with respect to the direction specified by the extraction direction input means. Only can be reduced. With the above configuration, the distance image is reduced only in the specified direction without reducing in the direction other than the direction specified by the extraction direction input means, so that the image is uniform in the direction specified in the free-form surface image. By leaving continuous height information in and taking a difference from this, it is possible to eliminate a uniform and continuous gradual shape change in the direction and extract only a necessary shape.

Furthermore, according to the three-dimensional image processing apparatus according to the sixth aspect, a light projecting means for projecting light on the inspection object by a light cutting method is further provided, and the imaging means is said to project light. The image projected by the means is imaged, and the distance image generation means can be configured to generate a distance image by synthesizing the profiles obtained by the optical cutting method.

Furthermore, according to the three-dimensional image processing apparatus according to the seventh aspect, in addition to any of the above configurations, when the reference plane setting means is in operation to perform three-dimensional image processing with the three-dimensional image processing apparatus. , For the newly input distance image, set the reference plane as the reference for gradation conversion by the calculation method selected based on the height information in the extraction area specified on the distance image. , It can be configured to execute the gradation conversion by the gradation conversion means based on the height of the set reference plane.

Furthermore, according to the three-dimensional image processing apparatus according to the eighth aspect, the reference plane setting means is designated by designating an arbitrary point in the distance image displayed on the display means. It can be configured to set the point height information as a reference plane.

Furthermore, according to the three-dimensional image processing apparatus according to the ninth aspect, the reference plane setting means is designated by designating an arbitrary point in the distance image displayed on the display means. It can be configured to set the point height information as a reference plane.

Furthermore, according to the three-dimensional image processing apparatus according to the tenth aspect, the reference plane setting means is designated by designating an arbitrary plane in the distance image displayed on the display means. It can be configured to set the height information of the surface as a reference surface.

Furthermore, according to the three-dimensional image processing apparatus according to the eleventh aspect, the reference plane setting means is designated by designating any three points in the distance image displayed on the display means. It can be configured to set the height information of the surface including the three points as the reference surface.

Furthermore, according to the three-dimensional image processing apparatus according to the twelfth side surface, the reference plane setting means calculates a curved surface from the distance image displayed on the display means, and the calculated curved surface is used as a reference plane. Can be configured to be set as.

Furthermore , according to the three-dimensional image processing method according to the thirteenth aspect, a three-dimensional image processing method that acquires a distance image including height information of an inspection object and performs image processing based on the distance image. Therefore, there is a step of projecting incident light from an oblique direction with respect to the optical axis of the image pickup means by the light projecting means as structured illumination of a predetermined projection pattern, and a step of projecting the light by the light projecting means and reflecting it by the inspection object. A step of acquiring the reflected light and capturing a plurality of pattern projection images by the imaging means, and a step of generating a distance image by the distance image generation means based on the plurality of pattern projection images captured by the imaging means. As a gradation conversion parameter that constitutes a gradation conversion condition when the distance image generated by the distance image generation means is gradation-converted to a distance image having a gradation number lower than the gradation number of the distance image. Based on the step of setting a reference plane as a reference for performing the gradation conversion and the height of the set reference plane, the distance image is divided into a number of gradations lower than the number of gradations of the distance image. A step of gradation-converting the height information of the distance image into a low-gradation distance image in which the grayscale value of the image is replaced, and a predetermined inspection process are executed on the gradation-converted low-gradation distance image. In addition to the step of prompting to specify the direction for extracting the local shape change when setting the reference plane, the curved surface is calculated from the distance image at the time of the gradation conversion, and the calculation is performed. By gradation-converting the formed curved surface into a low-gradation distance image as a reference plane, a shape change in the specified direction is extracted, and the distance image generation means inputs an inspection object that is sequentially input from the outside. By sequentially generating a distance image of the inspection object based on the image and resetting the reference plane based on the height information of the sequentially generated distance image, the reference plane is set according to the height of the inspection target. Is changed, and each distance image is converted into the low gradation distance image based on the height of the reference plane reset for each distance image generated by the distance image generation means, and the gradation conversion is performed. In addition, position correction using the distance image can be performed. As a result, instead of simply converting a high-gradation distance image into a low-gradation distance image, a reference plane is set and gradation conversion is performed using the height of the reference plane as a reference. Highly accurate inspection can be realized by avoiding the situation where the height information in the vicinity is impaired by the gradation conversion. In addition, changes in the inspection object in the direction specified by the extraction direction input means can be extracted, and in other words, a shape that does not change in the specified direction, such as a groove continuous in the longitudinal direction, is ignored and an accurate shape is obtained. Extraction is possible.

Furthermore , according to the three-dimensional image processing method according to the fourteenth aspect, when setting the reference plane, the direction for extracting the local shape change is any of the X direction, the Y direction, and the XY direction of the distance image. It can include a step of prompting to specify.

Furthermore , according to the three-dimensional image processing method according to the fifteenth aspect, at the time of the gradation conversion, the distance image is temporarily reduced, filtered, and enlarged to create a free curved image. By taking the difference between the distance image and the distance image, the shape change of the distance image is extracted, and when the distance image is reduced, the extraction for specifying the direction for extracting the local shape change in the gradation conversion is performed. The reduction can be performed only in the direction specified by the direction input means. As a result, the height information in the direction specified in the free-form surface image is left by reducing the distance image in a direction other than the specified direction without reducing the direction specified by the extraction direction input means. By taking the difference from this, it is possible to eliminate the gradual shape change in the direction and extract only the necessary shape.
Furthermore, according to the three-dimensional image processing method according to the sixteenth aspect, in addition to any of the above, in the step of setting the reference plane, from the outside during the operation of performing the three-dimensional image processing with the three-dimensional image processing apparatus. The reference plane is set based on the input image of the input inspection object, and the position of the input image can be corrected in the step of gradation conversion.

Furthermore, according to the three-dimensional image processing program according to the seventeenth aspect, it is possible to acquire a distance image including height information of the inspection object and perform image processing based on the distance image. An image processing program that projects incident light onto a computer from an oblique direction with respect to the optical axis of the imaging means as structured illumination of a predetermined projection pattern, and acquires the reflected light reflected by the inspection object. A distance image generation function that can generate a distance image based on a plurality of captured pattern projection images, a function that displays a distance image generated by the distance image generation function, and a setting mode for making necessary settings. As a setting function for accepting operations from the user and making various settings, as a gradation conversion parameter constituting a gradation conversion condition when gradation conversion is performed on a distance image generated by the distance image generation function. In the reference plane setting function for setting a reference plane as a reference for gradation conversion and the operation mode in which three-dimensional image processing is performed after the setting mode, the distance image is set to be larger than the number of gradations of the distance image. An image gradation conversion function for converting a low gradation image into a low gradation distance image in which the height information of the distance image is replaced with a grayscale value of the image, and the low gradation distance image. , An inspection execution function for executing a predetermined inspection process is realized, and in the operation mode, the reference plane setting function is an inspection input from the outside during the operation of performing the three-dimensional image processing in the three-dimensional image processing device. The reference plane is set based on the input image of the object, the gradation conversion function is configured to correct the position of the input image, and the distance image generation function is external. By sequentially generating a distance image of the inspection object based on the input image of the inspection object sequentially input from, and resetting the reference plane based on the height information of the sequentially generated distance image, the inspection is performed. The reference plane is changed according to the height of the object, and each distance image is converted into the low gradation distance image based on the height of the reference plane reset for each distance image generated by the distance image generation function. When the conversion is performed and the gradation conversion is performed, the position correction using the distance image can be performed.

Furthermore, the computer-readable recording medium or storage device according to the eighteenth aspect stores the above-mentioned three-dimensional image processing program. Recording media include CD-ROM, CD-R, CD-RW, flexible disc, magnetic tape, MO, DVD-ROM, DVD-RAM, DVD-R, DVD + R, DVD-RW, DVD + RW, and Blu-ray (registered). Includes magnetic discs such as trademark), HD DVD (AOD), optical discs, opto-magnetic discs, semiconductor memory and other media capable of storing programs. Further, the program includes a program stored in the above-mentioned recording medium and distributed, and a program distributed by download through a network line such as the Internet. Further, the stored device includes a general-purpose or dedicated device in which the above program is implemented in a state in which it can be executed in the form of software, firmware, or the like. Furthermore, each process and function included in the program may be executed by program software that can be executed by a computer, and each part of the process may be executed by hardware such as a predetermined gate array (FPGA, ASIC), or program software. It may be realized in a form in which and a partial hardware module that realizes a part of the hardware are mixed.

It is a figure which shows the system configuration example of the 3D image processing system which includes the image processing apparatus which concerns on embodiment of this invention. It is a figure which shows the system configuration example of the 3D image processing system which concerns on the modification of this invention. It is a schematic diagram which shows the hardware structure of the 3D image processing apparatus which concerns on Embodiment 2 of this invention. FIG. 4A is a schematic view showing a head portion of the three-dimensional image processing apparatus according to the third embodiment of the present invention, and FIG. 4B is a schematic view showing a head portion of the three-dimensional image processing apparatus according to the fourth embodiment. It is a block diagram which shows the 3D image processing apparatus which concerns on Embodiment 3 of this invention. It is a block diagram which shows the controller part of FIG. It is a flowchart which shows the processing operation of the image processing apparatus which concerns on this embodiment. It is a flowchart which shows the procedure of static conversion at the time of setting. It is an image diagram which shows the initial screen which added "imaging" in a three-dimensional image processing program. It is an image diagram which shows the state which image registration was selected in the image pickup setting menu. It is an image diagram which shows the screen example of the image registration screen. It is an image diagram which shows the screen example which is registering a distance image. It is an image diagram which shows the screen example which is registering a luminance image. It is an image diagram which shows the imaging setting screen. It is an image diagram which shows the imaging valid setting screen. It is an image diagram which shows the 3D measurement setting screen. It is an image diagram which shows the type of preprocessing which can be selected. It is an image diagram which shows the option which can be set in the unmeasurable standard setting column. It is an image diagram which shows the example which selected "none" in the unmeasurable standard setting column. It is an image diagram which shows the example which selected "low" in the unmeasurable standard setting column. It is an image diagram which shows the example which selected "medium" in the unmeasurable standard setting column. It is an image diagram which shows the example which selected "high" in the unmeasurable standard setting column. It is an image diagram which shows the option which can be set in the equal interval processing setting field. It is an image diagram which shows the example which turned on the uniform interval processing, and set the "display image" selection field as "height image". It is an image diagram which shows the example which turned on the uniform interval processing, and made the "display image" selection field "shade image". It is an image diagram which shows the example which turned off the uniform interval processing, and displayed the distance image. It is an image diagram showing an example in which the uniform interval processing is turned off and a luminance image is displayed. It is an image diagram which shows the option which can be set in a space code setting field. It is an image diagram which shows the state which selected "height image" in the "display image" selection field, and displayed the distance image in the second image display area. It is an image diagram which shows the state which selected the "shade image" in the "display image" selection field, and displayed the luminance image in the second image display area. It is an image diagram which shows the example which selected OFF in a space code setting field. The state in which the "shade image" is displayed in the "display image" selection field is shown. It is an image diagram which shows the option which can be set in a projector selection setting field. It is an image diagram which shows the example which selected "1" in the projector selection setting field. It is an image diagram which shows the example which selected "2" in the projector selection setting field. It is an image diagram which shows the example which selected "1 + 2" in the projector selection setting field. It is an image diagram which shows the option which can be set in the "display image" selection field. It is an image diagram which shows the example which selected "striped projection-projector 1" in the "display image" selection field, and displayed the pattern projection image of the 1st projector in the 2nd image display area. It is an image diagram which shows the example which selected "striped projection-projector 2" in the "display image" selection field, and displayed the pattern projection image of the 2nd projector in the 2nd image display area. It is an image diagram which shows the 3D measurement setting screen which set the shutter speed setting field to "1/15". It is an image diagram which shows the 3D measurement setting screen which set the shutter speed setting field to "1/30". It is an image diagram which shows the 3D measurement setting screen which set the shading range setting field to "normal (0)". It is an image diagram which shows the 3D measurement setting screen which set the shading range setting field to "high (1)". Image showing how to add a "height measurement" processing unit from the state shown in FIG. It is an image diagram which shows the initial screen which added the "height measurement" processing unit through FIG. 44. It is an image diagram which shows the height measurement setting screen. It is an image diagram which shows the inspection target area setting screen. It is an image diagram which shows the state which displayed the drop-down menu of the "measurement area" setting column of FIG. 47. It is an image diagram which shows the state which "rotation rectangle" is selected in the "measurement area" setting field. It is an image diagram which shows the measurement area edit screen. It is an image diagram which shows the state which "circumference" is selected in the "measurement area" setting field of the measurement area edit screen. It is an image diagram which shows the initial screen which added the 2nd "height measurement" processing unit. It is an image diagram which shows the state which set "rotation rectangle" as a measurement area. It is an image diagram which shows the state which sets the "rotation rectangle" as a measurement area. It is an image diagram which shows the state which "rotation rectangle" is set as a measurement area. It is an image diagram which shows the state which tries to add a "numerical calculation" processing unit. It is an image diagram which shows the initial screen which "numerical calculation" processing unit was added. It is an image diagram which shows the state which displayed the numerical calculation edit screen. FIG. 5 is an image diagram showing a state in which an arithmetic expression is input to the numerical operation editing screen of FIG. 58. It is an image diagram which shows the initial screen in which the "numerical calculation" processing unit is set. It is an image diagram which shows the initial screen which added the "area" processing unit. It is an image diagram which shows the area setting screen. It is an image diagram which shows the area setting screen which sets the detail of a rotation rectangle. It is an image diagram which shows the area setting screen in which a rotation rectangle is set. It is an image diagram which shows the height extraction selection screen. It is an image diagram which shows GUI of one point designation screen. It is an image diagram which shows GUI of one point designation screen. It is an image diagram which shows GUI of one point designation screen. FIG. 69A is an image diagram showing a profile of an input image, and FIG. 69B is an image diagram showing a profile of a low gradation distance image obtained by gradation-converting the input image of FIG. 69A. It is an image diagram which shows the state which increased the gain from the state of FIG. 68. It is an image diagram which shows the state which lowered the gain from the state of FIG. 70. FIG. 71 is an image diagram showing a state in which the detailed setting is selected in FIG. It is an image diagram which shows GUI of the emphasis method detailed setting screen for one point designation. It is an image diagram which shows the state which displayed the drop-down list of the "extraction height" setting column from the state of FIG. 73. 75A is an image diagram showing a profile of an input image, FIG. 75B is an image diagram showing an input image of FIG. 75A with reference to a reference plane, and FIG. 75C is an image diagram showing a profile of a low gradation distance image obtained by gradation-converting FIG. 75B. Is. 76A is a brightness image, FIG. 76B is a high-gradation distance image, FIG. 76C is a low-gradation distance image obtained by gradation-converting FIG. 76B, FIG. 76D is a low-gradation distance image having a higher gain than FIG. 76C, and FIG. 76E. Is an image diagram showing a low gradation distance image in which noise removal is improved as compared with FIG. 76D, and FIG. 76F is an image diagram showing a low gradation distance image in which the “extraction height” is set to a higher side in FIG. 76E. FIG. 77A is an image diagram showing a profile of the input image, and FIG. 77B is an image diagram showing a profile of a low gradation distance image obtained by gradation conversion of the input image of FIG. 77A with the “extraction height” set to the high side. It is an image diagram which shows the state which displayed the gradation conversion image. FIG. 79A is a perspective view showing an example of a work in which a method of setting a reference plane by designating one point is effective, and FIG. 79B is an image diagram of a low-gradation distance image obtained by gradation-converting the distance image captured in FIG. 79A. It is an image diagram which shows GUI of the height extraction selection screen. It is an image diagram which shows GUI of a three-point designation screen. It is an image diagram which shows the state which specifies the first point by the height extraction means from the state of FIG. 81. It is an image diagram which shows the state which specifies the second point from the state of FIG. 82. It is an image diagram which shows the state which further specifies the third point from the state of FIG. It is an image diagram which shows GUI of the detailed setting screen for three-point designation. FIG. 86A is a perspective view showing an example of a work in which a method of setting a reference plane by designating three points is effective, FIG. 86B is an image diagram of a low-gradation distance image obtained by gradation-converting a distance image captured in FIG. 86A, and FIG. 86C is an image diagram of a low-gradation distance image. FIG. 86D is an image diagram of a binarized image of FIG. 86B, and FIG. 86D is an image diagram of a binarized image obtained by designating one point when the work of FIG. 86A is inclined. FIG. 87A is a perspective view showing an example of another work in which the method of setting the reference plane by specifying three points is effective, and FIG. 87B is an image diagram and a view of a low gradation distance image obtained by gradation-converting the distance image captured in FIG. 87A. 87C is an image diagram of a binarized image of FIG. 87B, and FIG. 87D is an image diagram of a binarized image obtained by designating one point when the work of FIG. 87A is inclined. It is an image diagram which shows the GUI of the height dynamic extraction setting screen. It is an image diagram which shows the drop-down box of the "calculation method" selection field. It is an image figure which shows the average height reference setting screen. It is an image diagram which shows the mask area setting screen. It is an image diagram which shows the plane reference detailed setting screen. FIG. 93A is a perspective view showing an example of a work in which the method of setting the reference plane based on the average height is effective, and FIG. 93B is an image diagram of a low gradation distance image obtained by gradation-converting the distance image captured in FIG. 93A. It is an image diagram which shows the plane reference detailed setting screen. It is an image diagram which shows the detail of the invalid pixel designation column on the screen of FIG. 94. It is an image diagram which shows the free-form surface reference setting screen. FIG. 96 is an image diagram showing a state in which the numerical value in the “extraction size” designation column is increased in FIG. FIG. 98A is a perspective view showing an example of a work in which a method of setting a reference plane based on a free curved surface reference is effective, and FIG. 98B is an image diagram of a low gradation distance image obtained by gradation conversion of the distance image captured in FIG. 98A. It is an image diagram which shows the state which displayed the extraction area setting dialog from the state of FIG. 90. It is an image diagram which shows the state which "rectangle" was selected in the extraction area selection column of FIG. It is an image diagram which shows the state which displayed the extraction area edit dialog from the state of FIG. 100. It is an image diagram which shows the state which "circle" was selected in the mask area selection column of FIG. It is an image diagram which shows the state which displayed the mask area edit dialog from the state of FIG. 102. It is an image diagram which shows the state which set height extraction in an "area" processing unit. It is an image diagram which shows the filter processing setting screen. It is an image diagram which shows the binarization level setting screen. It is an image diagram which shows the state which set the filter processing. It is an image diagram which shows the judgment condition setting screen. It is an image diagram which shows the state which the judgment condition was set. It is an image diagram which shows the initial screen which added the "blob" processing unit. It is an image diagram which shows the state of setting the filter processing in a "blob" processing unit. It is an image diagram which shows how the detection condition is set in a "blob" processing unit. It is an image diagram which shows how the judgment condition is set in a "blob" processing unit. It is an image diagram which shows the state which tries to add a "color inspection" processing unit to an initial screen. It is an image diagram which shows how the area of a circle is set in the "color inspection" processing unit. It is an image diagram which shows the state which set the area of a circle in the "color inspection" processing unit. It is an image diagram which shows the state which the density average was set in the "color inspection" processing unit. It is an image diagram which shows the initial screen which set the "color inspection" processing unit. It is a flowchart which shows the flow of processing at the time of operation in the head part of the 3D image processing apparatus which concerns on Embodiment 3. FIG. It is a flowchart which shows the flow of processing at the time of operation in the head part of the 3D image processing apparatus which concerns on Embodiment 4. FIG. It is a flowchart which shows the gradation conversion method which concerns on Embodiment 1. It is a flowchart which shows the flow of processing at the time of operation in the controller part of the 3D image processing apparatus which concerns on Embodiment 3. It is an image diagram which shows the initial screen of a 3D image processing program. It is an image diagram which shows the state which set the search target area on the luminance image. It is an image diagram which shows the state which set a plurality of inspection target areas on a distance image. It is an image diagram which shows the enlarged state of the distance image of FIG. 125. It is an image diagram which shows the state which performs the height measurement by the 3D image processing program. It is a data flow diagram for generating a distance image by combining a phase shift method and a spatial coding method. It is a data flow diagram for generating a distance image only by a phase shift method without using a space coding method. It is a data flow diagram which shows the procedure of obtaining a Z image by turning off pitching such as XY. It is a figure which shows the example which outputs the point cloud data. It is a flowchart which shows the procedure of static conversion at the time of operation. It is a flowchart which shows the procedure of dynamic conversion at the time of operation. It is a perspective view which shows the workpiece to be inspected. It is a schematic diagram which shows the method of preparing a plurality of gradation conversion parameter sets in advance. It is a flowchart which shows the procedure of the method of FIG. 135. It is a perspective view which shows the example of the work which hardly fluctuates in height. FIG. 138A is a perspective view showing an example of a work whose height variation is desired to be detected, and FIG. 138B is an image diagram of a low gradation distance image obtained by gradation conversion of the distance image captured in FIG. 138A. It is a perspective view which shows the example of the work in which the designation of the reference plane by dynamic conversion (the average height reference) is effective. It is a perspective view which shows the example of the work in which the designation of the reference plane by dynamic transformation (plane reference) is effective. It is a perspective view which shows the example of the work in which the designation of the reference plane by dynamic transformation (free-form surface reference) is effective. It is a flowchart which shows the procedure which repeats adjustment of a gradation conversion parameter until an appropriate image is obtained. FIG. 14 is a flowchart showing a procedure in which determination of whether or not an image is appropriate is omitted in FIG. 142. It is a flowchart which shows the specific procedure of the gradation conversion parameter adjustment. FIG. 145A is an image showing the appearance of the work, FIG. 145B is an image of a distance image obtained from the work of FIG. 145A, and FIG. 145C is an image showing a state in which an inspection target area is set for height inspection processing with respect to the distance image of FIG. 145D is an image showing a state in which an inspection target area is set for image inspection processing on the distance image of FIG. 145B, and FIG. 145E is a low gradation distance image obtained by gradation conversion of the distance image of FIG. 145D. It is a figure which shows each image of. It is a flowchart which shows the procedure at the time of setting. It is an image diagram which shows the screen which sets the "area" processing unit with respect to the work of FIG. 145A. FIG. 147 is an image diagram showing a screen for setting height extraction conditions. It is a flowchart which shows the procedure at the time of operation. It is a flowchart which shows the procedure in the case of performing the gradation conversion in the inspection process of FIG. 149. It is a flowchart which shows the procedure when the gradation conversion is not performed in the inspection process of FIG. 149. It is a flowchart which shows the procedure of setting an inspection process condition. It is an image diagram which shows the image setting screen. It is an image diagram which shows the state which called the image variable selection screen which can select a luminance image or a distance image. It is an image diagram which shows the state which called the image variable selection screen which can select only the distance image. It is a flowchart which shows the procedure which selects the inspection process after selecting an image, and sets the inspection process condition. It is a schematic diagram which shows the state which acquired the luminance image and the distance image. It is a schematic diagram which shows the inspection processing tool which can be set when the luminance image is selected in FIG. 157. It is a schematic diagram which shows the inspection processing tool which can be set when the distance image is selected in FIG. 157. It is a schematic diagram which shows the state of taking a distance image by the triangular distance measurement method. It is a flowchart which shows the procedure of fitting the height information distributed in a free curved surface target area. It is an image diagram which shows the distance image of the inspection object having a uniform shape in a vertical direction. FIG. 3 is an image diagram showing a low gradation distance image extracted from the distance image of FIG. 162 in the XY directions and subjected to gradation conversion. It is an image diagram which shows another example of a free-form surface reference setting screen. It is an image diagram which shows the detailed setting screen of the free-form surface reference setting screen of FIG. FIG. 3 is an image diagram showing a low gradation distance image extracted from the distance image of FIG. 162 in the Y direction and subjected to gradation conversion. It is an image diagram which shows the distance image of another inspection object having a uniform shape in a vertical direction. FIG. 3 is an image diagram showing a low gradation distance image extracted from the distance image of FIG. 167 in the XY directions and subjected to gradation conversion. FIG. 3 is an image diagram showing a low gradation distance image extracted from the distance image of FIG. 167 in the Y direction and subjected to gradation conversion.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiments shown below include a three-dimensional image processing apparatus, a three-dimensional image processing method, a three-dimensional image processing program, a computer-readable recording medium, and a recording device for embodying the technical idea of the present invention. The present invention does not specify a three-dimensional image processing apparatus, a three-dimensional image processing method, a three-dimensional image processing program, a computer-readable recording medium, and a recording device as follows. Further, the present specification does not specify the members shown in the claims as the members of the embodiment. In particular, the dimensions, materials, shapes, relative arrangements, etc. of the components described in the embodiments are not intended to limit the scope of the present invention to the specific description, but are merely described. It's just an example. The size and positional relationship of the members shown in each drawing may be exaggerated to clarify the explanation. Further, in the following description, members having the same or the same quality are shown with the same name and reference numeral, and detailed description thereof will be omitted as appropriate. Further, each element constituting the present invention may be configured such that a plurality of elements are composed of the same member and the plurality of elements are combined with one member, or conversely, the function of one member is performed by the plurality of members. It can also be shared and realized.

Further, in the present specification, the term "distance image (height image)" is used to mean an image including height information, for example, a three-dimensional image in which an optical brightness image is pasted as texture information on a distance image. The composite image is also used in the sense that it is included in the distance image. Further, in the present specification, the display form of the distance image is not limited to the one displayed in a two-dimensional shape, but also includes the one displayed in a three-dimensional shape.
(Embodiment 1)

FIG. 1 shows the configuration of the three-dimensional image processing apparatus according to the first embodiment of the present invention. The three-dimensional image processing device 100 includes a head unit 1 and a controller unit 2. The head unit 1 includes a light projecting means 20 that illuminates the inspection object (work) W, an imaging means 10 that captures an image of the work W, and a head-side communication means 36 for connecting to the controller unit 2.

On the other hand, the controller unit 2 executes measurement processing such as edge detection and area calculation based on the captured image. Further, a display means 4 such as a liquid crystal panel, an input means 3 such as a console for the user to perform various operations on the display means 4, a PLC (Programmable Logic Controller), and the like can be detachably connected to the controller unit 2.

In the above three-dimensional image processing apparatus 100, the light projecting means 20 of the head unit 1 projects the measurement light onto the work W, and the image pickup means 10 patterns the reflected light reflected by the measurement light incident on the work W. Image as a projected image. Further, a distance image is generated based on the pattern projection image, and the distance image is further converted into a low gradation distance image in which the height information of each pixel is replaced with the brightness. The controller unit 2 executes measurement processing such as edge detection and area calculation based on the converted low gradation distance image.

The work W, which is an object to be inspected, is, for example, an article that is sequentially transported on a line, and is moving or stationary. Further, the moving work includes a rotating work as well as a parallel movement by a conveyor or the like.
(Light projecting means 20)

The light projecting means 20 is used as an illumination that illuminates the work W in order to generate a distance image. Therefore, the light projecting means 20 is, for example, an optical projector that projects a line-shaped laser beam onto the work, or a sinusoidal striped pattern on the work, depending on the light cutting method or the pattern projection method for acquiring a distance image. Can be used as a pattern projector or the like for projecting. In addition to the light projecting means, a general lighting device for performing bright-field illumination or dark-field illumination may be separately provided. Alternatively, the light projecting means 20 can be provided with a function as a general lighting device.

The controller unit 2 executes image processing using the distance image data acquired from the head unit 1, and outputs a determination signal as a signal indicating the determination result of the quality of the work or the like to the externally connected control device such as the PLC70. do.

The imaging means 10 images the work based on a control signal input from the PLC 70, for example, an imaging trigger signal that defines a timing for capturing image data from the imaging means 10.

The display means 4 is a display device for displaying the image data obtained by imaging the work and the result of the measurement process using the image data. Generally, the user can confirm the operating state of the controller unit 2 by visually recognizing the display means 4. The input means 3 is an input device for moving the focus position and selecting a menu item on the display means 4. When a touch panel is used for the display means 4, the display means and the input means can be used in combination.

Further, the controller unit 2 can also be connected to a personal computer PC for generating a control program of the controller unit 2. Further, it is also possible to install a three-dimensional image processing program for setting the three-dimensional image processing on the personal computer PC and perform various settings for the processing performed by the controller unit 2. Alternatively, the software running on the personal computer PC can generate a processing order program that defines the processing order of image processing. In the controller unit 2, each image processing is sequentially executed according to the processing order. The personal computer PC and the controller unit 2 are connected via a communication network, and the processing sequence program generated on the personal computer PC includes, for example, layout information that defines the display mode of the display means 4, and the controller unit. Transferred to 2. On the contrary, it is also possible to take in the processing order program, layout information, etc. from the controller unit 2 and edit them on the personal computer PC. The processing sequence program may be generated not only in the personal computer PC but also in the controller unit 2.
(Modification example)

In the above example, dedicated hardware is constructed as the controller unit 2, but the present invention is not limited to this configuration. For example, like the three-dimensional image processing device 100'related to the modification shown in FIG. 2, a device in which a dedicated inspection program or a three-dimensional image processing program is installed on a general-purpose personal computer, workstation, or the like functions as a controller unit 2'. It can also be used by connecting to the head unit 1. This three-dimensional image processing device is required by performing necessary settings such as image processing with a three-dimensional image processing program and then performing image processing on a low gradation distance image according to the pattern projection image captured by the head unit 1. Perform a thorough inspection.
(Head side communication means 36)

Further, according to this, an interface that can be connected to either a dedicated controller unit 2 or a personal computer that functions as the controller unit 2 can be provided on the head unit 1 side as the head side communication means 36. For example, the head unit 1 is provided with a controller connection interface 36A for connecting to the controller unit 2 as a head-side communication means 36 as shown in FIG. 1, or for connecting to a personal computer as shown in FIG. PC connection interface 36B is provided. Further, by making such an interface replaceable as a unit, other configurations of the head portion can be made common to some extent, and the common head portion can be connected to either the controller unit or the personal computer. Alternatively, one head-side communication means provided with a dedicated controller unit 2 and an interface that can be connected to any of the personal computers may be provided. Further, as such an interface, existing communication standards such as Ethernet (trade name), USB, RS-232C and the like can be used. Further, the communication method may be a dedicated communication method, not necessarily a standardized or general-purpose communication method.
(PC connection mode)

Further, the three-dimensional image processing program may be provided with a PC connection mode for setting when a personal computer is used as the controller unit 2'connected to the head unit 1. That is, by changing the items and setting contents that can be set according to whether the controller unit is dedicated hardware or a personal computer, it is possible to appropriately set the three-dimensional image processing in any case. It will be possible. Further, the operation and functions of the connected head unit can be confirmed by installing a viewer program having a simple measurement function and an operation check purpose of the head unit 1 on a personal computer functioning as the controller unit 2'. You may do so.

The "distance image" obtained by using the image pickup means 10 and the light projecting means 20 shown in FIG. 1 is a shading value of each pixel according to the distance from the image pickup means 10 that images the work W to the work W. Is an image that changes. In other words, it can be said to be an image in which the shading value is determined based on the distance from the imaging means 10 to the work W, or it can be said to be a multi-valued image having a shading value according to the distance to the work W, or the work W. It can be said that it is a multi-valued image having a light and shade value according to the height. Further, it can be said that it is a multi-valued image in which the distance from the imaging means 10 is converted into a shading value for each pixel of the luminance image.

There are roughly two methods for generating a distance image. One is a passive method (passive measurement method) that generates a distance image using an image captured under lighting conditions for obtaining a normal image. The other is an active method (active measurement method) in which a distance image is generated by actively irradiating light for measuring in the height direction. A typical method of the passive method is a stereo measurement method. This is because a distance image can be generated only by preparing two imaging means 10 and arranging these two cameras in a predetermined positional relationship, so that a general image processing system for generating a luminance image is used. It is possible to generate a distance image and reduce the system construction cost. However, in the stereo measurement method, it is necessary to determine which point in the image obtained by one camera corresponds to which point in the image obtained by the other camera, which is a so-called corresponding point determination process. There is a problem that it takes time. Further, since the measurement position is only the corresponding point and not all the pixels, it is not suitable for speeding up the visual inspection in this respect as well.

On the other hand, typical methods of the active method are the optical cutting method and the pattern projection method. The optical cutting method is a method of optically measuring the shape and roughness of a surface. A thin slit image is projected on the surface of an object to be inspected at an angle of about 45 °, and the image is projected in the specular reflection direction. A method of observing from the side or a method of observing the shape of the cutting line generated on the surface from the side by irradiating the object to be inspected with a thin slit-shaped light bundle so as to cut it is known. In addition, although only one line of profile (cut surface) can be obtained by the optical cutting method, a distance image can be obtained by continuously changing the position where optical cutting is performed and synthesizing the profiles obtained at each position. It can also be configured.

The light cutting method here is the stereo measurement method described above, in which one camera is replaced with an optical projector, a line-shaped laser beam is projected onto the work, and the line light according to the shape of the object surface is emitted. The three-dimensional shape of the work is restored from the degree of distortion of the image. The optical cutting method does not require determination of the corresponding point, so stable measurement is possible. However, since only one line can be measured in one measurement, it is necessary to scan the object or the camera in order to obtain the measured values of all the pixels. On the other hand, the pattern projection method restores the three-dimensional shape of the work by capturing a plurality of images by shifting the shape and phase of a predetermined pattern projected on the work and analyzing the captured multiple images. Is what you do. There are several types of pattern projection methods. The phase of the sine wave striped pattern is shifted to take multiple images (at least 3 or more), and the phase of the sine wave is obtained for each pixel from the multiple images. The three-dimensional shape is restored by using the phase shift method to obtain the three-dimensional coordinates on the work surface using the obtained phase and the kind of spatial frequency swell phenomenon that occurs when two regular patterns are combined. More repotography method, the pattern itself projected on the work is different for each shooting, for example, the stripe width becomes half the screen, 1/4, 1/8, etc. with a black and white duty ratio of 50%. Patterns are projected in sequence, a pattern projection image is taken for each pattern, a spatial coding method for obtaining the absolute phase of the height of the work, and multiple fine line pattern illuminations (multi-slits) are projected on the work. A typical example is a multi-slit method in which the pattern is moved at a pitch narrower than the slit period and multiple shots are taken.

In the three-dimensional image processing apparatus 100 according to the present embodiment, a distance image is generated by combining the above-mentioned phase shift method and spatial coding method. This makes it possible to generate a distance image without moving the work or the head relatively. The present invention is not limited to generating a distance image by a phase shift method and a spatial coding method, and a distance image may be generated by another method. Further, any method other than the above-mentioned methods, such as an optical radar method (time of flight), an in-focus method, a confocal method, and a white light interferometry, may be adopted to generate a distance image. do not have.

The layout of the imaging means 10 and the light projecting means 20 shown in FIG. 1 is such that the light projecting means 20 projects light from an oblique direction with respect to the work W and receives the reflected light from the work W in a substantially vertical direction. Is arranged diagonally so as to hold the imaging means 10 in a vertical posture. By inclining the light projection direction and the imaging direction without matching them in this way, it is possible to capture a pattern projection image that captures the shadow caused by the unevenness of the surface shape of the work W.
(Embodiment 2)

However, the present invention is not limited to this arrangement example, and the imaging means 10 side is tilted with respect to the work W, for example, as in the three-dimensional image processing apparatus 200 according to the second embodiment shown in FIG. , May be an example of arrangement in which the light projecting means 20 side is held in a vertical posture. Even with the head portion 1B having such an arrangement, it is possible to image a pattern projection image that captures the shadow of the work W by inclining the light projection direction and the imaging direction in the same manner.
(Embodiment 3)

Further, one or both of the light projecting means and the imaging means can be arranged. For example, as in the three-dimensional image processing apparatus 300 shown in FIG. 4A as the third embodiment, the image pickup means 10 is held in a vertical posture with respect to the work W, while the two light projecting means 20 are centered on the image pickup means 10. It can also be arranged on both sides and configured as a head portion 1C that emits light from the left and right respectively. By capturing the pattern projection images in different directions in this way, the pattern projection image cannot be partially captured, for example, the shadow pattern is hidden by the work W itself when the light is projected from one direction. It is possible to reduce the situation where height measurement becomes inaccurate or impossible. In particular, if the light projecting means 20 is arranged so as to project light from a direction facing the work (for example, left and right or front and back), the possibility that the work itself is obstructed and the image cannot be captured can be significantly reduced.
(Embodiment 4)

Further, in the above example, the configuration in which the imaging means is one and the light projecting means is two has been described, but conversely, the configuration in which the imaging means is two and the light projecting means is one can be used. An example of this is shown in FIG. 4B as the three-dimensional image processing apparatus 400 according to the fourth embodiment. In the head portion 1D shown in this example, the light projecting means 20 is held in a vertical posture with respect to the work W, and the imaging means 10 are arranged on the left and right sides of the light projecting means 10 in an inclined posture with respect to the work W, respectively. Even with this configuration, since the work W can be imaged from different inclination angles, it is possible to suppress a situation in which the pattern projection image is partially difficult to image as in the third embodiment. Further, with this method, since two pattern projection images can be captured at the same time with one light projection, there is an advantage that the processing time can be shortened.

On the other hand, even if the same work is imaged from different angles by the two imaging means, the imaged part and the field of view are different, so it is necessary to match the positions of each pixel, and an error may occur. be. On the other hand, according to the third embodiment described above, by sharing the imaging means, it is possible to capture an image of the same field of view regardless of which light projecting means emits the measurement light. There is an advantage that the processing can be simplified by eliminating the need for various integration work and avoiding the occurrence of errors due to the integration work.

In the above examples, an example in which the imaging means 10 and the light projecting means 20 are integrally configured in each head portion has been described, but the present invention is not limited to this configuration. For example, the imaging means 10 and the light projecting means 20 can be a head portion composed of separate members. It is also possible to provide three or more imaging means and light projecting means.
(Block Diagram)

Next, FIG. 5 shows a block diagram showing the configuration of the three-dimensional image processing apparatus 300 according to the third embodiment of the present invention. As shown in FIG. 5, the three-dimensional image processing device 300 includes a head unit 1 and a controller unit 2.
(Head part 1)

The head unit 1 includes a light projecting means 20, an imaging means 10, a head side control unit 30, a head side calculation unit 31, a storage means 38, a head side communication means 36, and the like. The light projecting means 20 includes a measurement light source 21, a pattern generation unit 22, and a plurality of lenses 23, 24, and 25. The imaging means 10 includes a camera and a plurality of lenses (not shown).
(Light projecting means 20)

The light projecting means 20 is a member for projecting incident light from an oblique direction with respect to the optical axis of the imaging means as structured illumination of a predetermined projection pattern. A projector can be used in the light projecting means 20, and a lens, a pattern generation unit 22, and the like, which are optical members, are included. The light projecting means 20 is arranged obliquely above the position of the stationary or moving work. The head portion 1 may also include a plurality of light projecting means 20. In the example of FIG. 5, the head portion 1 includes two light projecting means 20. Here, the first projector 20A (on the right side in FIG. 5) capable of irradiating the work with the illumination light for measurement from the first direction and the work for measurement from a second direction different from the first direction. A second projector 20B (on the left side in FIG. 5) capable of irradiating illumination light is arranged respectively. The first projector 20A and the second projector 20B are arranged symmetrically with respect to the optical axis of the imaging means 10. Measurement light is alternately projected from the first projector 20A and the second projector 20B onto the work, and the pattern of each reflected light is imaged by the imaging means 10.

As the measurement light source 21 of each of the first projector 20A and the second projector 20B, for example, a halogen lamp that emits white light, a white LED (light emitting diode) that emits white light, or the like can be used. The measurement light emitted from the measurement light source 21 is appropriately focused by the lens and then incident on the pattern generation unit 22.

Further, in addition to the light projecting means for emitting the measurement light for acquiring the pattern projection image for generating the distance image, an observation illumination light source for capturing a normal optical image (luminance image) can also be provided. .. In addition to LEDs, semiconductor lasers (LDs), halogen lights, HIDs, and the like can be used as the illumination light source for observation. In particular, when a color image sensor is used as the image sensor, a white light source can be used as the observation illumination light source.

The measurement light emitted from the measurement light source 21 is appropriately focused by the lens 113 and then incident on the pattern generation unit 112. The pattern generation unit 22 can realize illumination of an arbitrary pattern. For example, it is possible to invert the pattern according to the color of the work or background, such as white letters for black letters and black letters for white letters, to express an appropriate pattern that is easy to see or measure. For such a pattern generation unit 22, for example, a DMD (Digital Micromirror Device) can be used. The DMD can express an arbitrary pattern by turning on / off a minute mirror for each pixel. As a result, it is possible to easily irradiate a pattern in which white and black are inverted. By using DMD for the pattern generation unit 22, an arbitrary pattern can be easily generated, and it is not necessary to prepare a mechanical pattern mask or replace it, so that there is an advantage that the device can be miniaturized and quick measurement can be performed. .. Further, since the pattern generation unit 112 using the DMD can be used in the same manner as normal illumination by irradiating all illumination patterns with all pixels turned on, it can also be used for capturing a luminance image. Further, the pattern generation unit 22 may be an LCD (liquid crystal display), LCOS (Liquid Crystal on Silicon: reflective liquid crystal element), or a mask. The measurement light incident on the pattern generation unit 22 is converted into a preset pattern and a preset intensity (brightness) and emitted. The measurement light emitted by the pattern generation unit 22 is converted into light having a diameter larger than the field of view that can be observed and measured by the imaging means 10 by a plurality of lenses, and then irradiated to the work.
(Imaging means 10)

The image pickup means 10 includes a camera for acquiring the reflected light projected by the light projecting means 20 and reflected by the work WK and capturing a plurality of pattern projection images. A CCD, CMOS, or the like can be used for such a camera. In this example, a monochrome CCD camera that can obtain high resolution is used. Needless to say, it is also possible to use a camera capable of capturing images in color. The imaging means can also capture a normal luminance image in addition to the pattern projection image.

The head-side control unit 30 is a member for controlling the first projector 20A and the second projector 20B, which are the imaging means 10 and the light projecting means 20. For example, the head unit side control unit 30 creates a projection pattern for the projection means 20 to project the measurement light onto the work to obtain a pattern projection image. As a result, the imaging means 10 projects a projection pattern for phase shift from the projection means 20 to capture a phase shift image, and a projection pattern for spatial coding is projected from the projection means 20 to obtain a spatial code image. Have an image taken. In this way, the head-side control unit 30 functions as a light projection control means for controlling the light projection means so that the image pickup means 10 captures the phase shift image and the space code image.

The head-side calculation unit 31 includes a filter processing unit 34 and a distance image generation means 32. The distance image generation means 32 generates a distance image based on a plurality of pattern projection images captured by the image pickup means 10.

The head-side storage means 38 is a member for holding various settings, images, and the like, and a storage element such as a semiconductor memory or a hard disk can be used. For example, it includes a luminance image storage unit 38b for holding a pattern projection image captured by the imaging means 10, and a distance image storage unit 38a for holding a distance image generated by the distance image generation means 32.

The head-side communication means 36 is a member for communicating with the controller unit 2. Here, it is connected to the controller-side communication means 42 of the controller unit 2 to perform data communication. For example, the distance image generated by the distance image generation means 32 is sent to the controller unit 2.
(Distance image generation means 32)

The distance image generation means 32 is a means for generating a distance image in which the shading value of each pixel changes according to the distance from the image pickup means 10 that images the work WK to the work WK. For example, when a distance image is generated by the phase shift method, the head side control unit 30 controls the light projecting means 20 so as to project the sinusoidal fringe pattern on the work in a phase-shifted manner, and the sinusoidal pattern is projected accordingly. The head-side control unit 30 controls the imaging means 10 so as to capture a plurality of images in which the phase of the wavy pattern pattern is out of phase. Then, the head side control unit 30 obtains the phase of the sine wave for each pixel from a plurality of images, and generates a distance image using the obtained phase.

When a distance image is generated by using the space coding method, the space irradiated with light is divided into a large number of small spaces having a substantially fan-shaped cross section, and a series of space code numbers are assigned to these small spaces. Therefore, even if the height of the work is high, in other words, even if the height difference is large, the height can be calculated from the space code number as long as it is in the space irradiated with light. Therefore, the shape of a high work piece can be measured as a whole.

By generating the distance image on the head unit side and transmitting it to the controller unit side in this way, the amount of data to be transmitted from the head unit to the controller unit side can be reduced, and the processing delay that may occur due to a large amount of data transfer can be reduced. It can be avoided.

In the present embodiment, the distance image generation process is performed on the head unit 1 side, but for example, the controller unit 2 side can also be responsible for the distance image generation process. Further, the gradation conversion from the distance image to the low gradation distance image can be performed not only by the controller unit but also by the head unit side. In this case, the head-side calculation unit 31 realizes the function of the gradation conversion means.
(Controller unit 2)

Further, the controller unit 2 includes a controller-side communication means 42, a controller-side control unit, a controller-side calculation unit, a controller-side storage means, an inspection execution means 50, and a controller-side setting means 41. The controller-side communication means 42 is connected to the head-side communication means 36 of the head unit 1 to perform data communication. The controller-side control unit is a member for controlling each member. The controller-side arithmetic unit realizes the function of the image processing unit 60. The image processing unit 60 realizes functions such as an image search means 64 and a gradation conversion means 46.
(Gradation conversion means)

The gradation conversion means 46 gradation-converts a high-gradation distance image into a low-gradation low-gradation distance image based on the distance image (detailed procedure will be described later). As a result, the distance image having the height information generated by the head portion can be expressed as a two-dimensional grayscale image that can be handled by the existing equipment, which can contribute to the measurement process and the inspection process. Further, there is an advantage that the load can be distributed by sharing the distance image generation process and the gradation conversion process between the head unit and the controller unit. In addition to generating the distance image on the head portion side, a low gradation distance image may be generated. Such processing can be performed by the head side calculation unit. As a result, the load on the controller unit side is further reduced, and efficient operation becomes possible.

Further, the gradation conversion means does not perform gradation conversion on all of the distance image, but preferably selects only a necessary portion and performs gradation conversion. Specifically, only the portion corresponding to the inspection target area set in advance by the inspection target area setting means (details will be described later) is gradation-converted. By doing so, the load required for gradation conversion can be reduced by limiting the process of converting a multi-gradation distance image to a low-gradation distance image only in the inspection target area. This also contributes to shortening the processing time. That is, by shortening the processing time, it can be suitably used even in applications with limited processing time such as inspection for FA applications, and real-time processing is realized.

The controller-side storage means is a member for holding various settings and images, and a semiconductor storage element, a hard disk, or the like can be used.

The controller-side setting means 41 is a member for making various settings for the controller unit, receives an operation from the user via an input means 3 such as a console connected to the controller unit, and provides necessary conditions and the like to the controller side. Instruct. For example, the functions of the gradation conversion condition setting means 43, the reference plane setting means 44, the spatial coding switching means 45, the interval equalization processing setting means 47, the light projection switching means 48, the shutter speed setting means 49, and the like are realized.

The reference plane setting means 44 is used as a gradation conversion parameter that constitutes a gradation conversion condition when performing gradation conversion for converting a distance image received by the controller-side communication means 42 into a two-dimensional low gradation distance image. , A reference plane for performing this gradation conversion is set based on the distance image. The gradation conversion means 46 uses the reference plane set by the reference plane setting means 44 as a reference, and converts the distance image into a gradation value lower than the gradation number of the distance image, and converts the height information into the grayscale value of the image. Gradation conversion is performed to the replaced low gradation distance image.

The inspection execution means 50 executes a predetermined inspection process on the low gradation distance image that has been gradation-converted by the gradation conversion means 46.
(Hardware configuration)

Next, a hardware configuration example of the controller unit 2 is shown in the block diagram of FIG. The controller unit 2 shown in this figure has a main control unit 51 that performs numerical calculation and information processing based on various programs and controls each hardware unit. The main control unit 51 stores, for example, a CPU as a central processing unit, a work memory such as a RAM that functions as a work area when the main control unit 51 executes various programs, a startup program, an initialization program, and the like. It has a program memory such as a ROM, a flash ROM, or an EEPROM.

Further, the controller unit 2 is connected to the head unit 1 including the image pickup means 10, the light projecting means 20, and the like, and controls the light projecting means 20 so as to project the sinusoidal striped pattern on the work in a shifted phase. To the controller-side connection unit 52 for capturing image data obtained by imaging with the image pickup means 10, the operation input unit 53 into which the operation signal from the input means 3 is input, and the display means 4 such as a liquid crystal panel. A display control unit 54 composed of a display DSP or the like for displaying an image, a communication unit 55 connected so as to be able to communicate with an external PLC 70, a personal computer PC, or the like, and a RAM 56 for holding temporary data. The controller-side storage means 57 for storing the contents, the auxiliary storage means 58 for holding the data set by the three-dimensional image processing program installed in the personal computer PC, and the measurement processing such as edge detection and area calculation are executed. It is provided with an image processing unit 60 composed of a DSP for calculation and the like, and an output unit 59 and the like for outputting the result of performing a predetermined inspection based on the processing result of the image processing unit 60 and the like. Each of these hardware is communicably connected via an electrical communication path (wiring) such as a bus.

In the program memory in the main control unit 51, control for controlling each unit of the controller side connection unit 52, the operation input unit 53, the display control unit 54, the communication unit 55, and the image processing unit 60 by a CPU command or the like. The program is stored. Further, the above-mentioned processing sequence program, that is, the processing sequence program generated in the personal computer PC and transferred from the personal computer PC is stored in the program memory.

The communication unit 55 functions as an interface (I / F) for receiving an image pickup trigger signal from the PLC 70 when a trigger input is received from a sensor (photoelectric sensor or the like) connected to the external PLC 70. It also functions as an interface (I / F) for receiving a three-dimensional image processing program transferred from a personal computer PC, layout information defining a display mode of the display means 4, and the like.

When the CPU of the main control unit 51 receives the image pickup trigger signal from the PLC 70 via the communication unit 55, the CPU sends an image pickup command (command) to the controller side connection unit 52. Further, based on the processing order program, a command for instructing the image processing to be executed is transmitted to the image processing unit 60. As a device for generating an imaging trigger signal, a sensor for trigger input such as a photoelectric sensor may be configured to be directly connected to the communication unit 55 instead of the PLC 70.

The operation input unit 53 functions as an interface (I / F) for receiving an operation signal from the input means 3 based on the user's operation. The display means 4 displays the operation contents of the user using the input means 3. For example, when a console is used as the input means 3, each component such as a cross key for moving the cursor displayed on the display means 4 up / down / left / right, a decision button, or a cancel button can be arranged. By operating each of these parts, the user can create a flowchart that defines the processing order of image processing on the display means 4, edit the parameter value of each image processing, and set the reference area. , You can edit the reference registration image.

The controller side connection unit 52 captures image data. Specifically, for example, when an image pickup command of the image pickup means 10 is received from the CPU, an image data acquisition signal is transmitted to the image pickup means 10. Then, after the image is taken by the image pickup means 10, the image data obtained by the image is taken in. The captured image data is once buffered (cached) and assigned to an image variable prepared in advance. The "image variable" refers to a variable that can be used as a reference for measurement processing or image display by allocating it as an input image of a corresponding image processing unit, unlike a normal variable that handles numerical values.

The image processing unit 60 executes measurement processing on the image data. Specifically, first, the controller-side connection unit 52 reads the image data from the frame buffer while referring to the above-mentioned image variables, and internally transfers the image data to the memory in the image processing unit 60. Then, the image processing unit 60 reads out the image data stored in the memory and executes the measurement process. Further, the image processing unit 60 includes a gradation conversion means 46, an abnormal point highlighting means 62, an image search means 64, and the like.

The display control unit 54 transmits a control signal for displaying a predetermined image (video) to the display means 4 based on the display command (display command) sent from the CPU. For example, a control signal is transmitted to the display means 4 in order to display the image data before or after the measurement process. The display control unit 54 also transmits a control signal for displaying the operation content of the user using the input means 3 on the display means 4.

The head unit 1 and the controller unit 2 composed of the above hardware have a configuration in which each means and function of FIG. 5 can be realized by software by various programs and the like. In this example, a mode in which a three-dimensional image processing program is installed on the computer of FIG. 1 and necessary settings for three-dimensional image processing are performed is adopted.
(Gradation conversion)

The above three-dimensional image processing apparatus acquires a distance image of the work, performs image processing on the distance image, and inspects the result. The three-dimensional image processing apparatus according to the present embodiment uses existing hardware to provide information such as area and edge, in addition to height inspection processing in which height information which is a pixel value of a distance image is used as it is for calculation. It is possible to carry out two types of inspections, that is, an image inspection process that performs an operation using. Here, in order to maintain the accuracy of the height inspection process, it is necessary to generate a multi-gradation distance image. On the other hand, with existing hardware, it is not possible to perform image inspection processing on such a multi-gradation distance image. Therefore, in order to perform image inspection processing using existing hardware, gradation conversion is performed on a multi-gradation distance image to generate a low-gradation distance image.

However, if the height information of the multi-gradation distance image is directly converted into the low-gradation distance image, there is a problem that the accuracy of the height information is impaired. Most of the general images used for FA applications are monochrome images in which the shading value of each pixel is expressed in 8 gradations. On the other hand, as the distance image, a high-gradation image such as a 16-gradation image is used. Therefore, when the multi-gradation distance image is gradation-converted to the low-gradation distance image, the height information is considerably impaired, which affects the accuracy of the inspection. However, in order to increase the number of gradations of the image handled by the existing image processing in order to improve the accuracy, the introduction cost increases, the processing load increases, and the hurdle for use increases.

Therefore, in such gradation conversion, it is necessary to set the gradation conversion conditions so that the required height information is maintained. The method and procedure will be described in detail below.
(Height inspection or image inspection)

First, the processing operation of performing the height inspection process using the three-dimensional image processing device will be described with reference to the flowchart of FIG. 7. This three-dimensional image processing device is a height inspection processing tool that performs height inspection on a distance image and various image inspection processes that perform image inspection on an existing luminance image as tools for performing calculation processing. It has tools. Here, the height inspection process will be described.

First, a distance image is generated (step S71). Specifically, the distance image generation means 32 generates a distance image by using the image pickup means 10 and the light projecting means 20. Then, the desired calculation process is selected (step S72). Here, select the tools required for the calculation process.

When selecting the image inspection processing tool, the process proceeds to step S73, and the high gradation distance image obtained in step S71 is subjected to gradation conversion processing to be converted into a low gradation distance image. As a result, even the inspection processing tool provided in the existing image processing apparatus can handle the low gradation distance image. The gradation conversion process is not performed on the entire area of the high-gradation distance image, but is preferably performed only within the inspection target area set for the image inspection process.

On the other hand, when the height inspection tool is selected, since the height information of the multi-gradation distance image is used as it is, the process proceeds to step S74 without performing gradation conversion.

Further, the inspection executing means 50 performs various calculation processes (step S74), and then determines whether or not the work is a non-defective product based on the calculation results (step S75). When the inspection execution means 50 determines that the work is a non-defective product (step S75: YES), the determination signal output means 160 outputs an OK signal to the PLC 70 as a determination signal (step S76), and the inspection execution means 50. When it is determined that the work is not a non-defective product, that is, a defective product (step S75: NO), an NG signal is output to the PLC 70 as a determination signal (step S77).
(Setting mode)

Next, an example of the procedure in the setting mode in which various settings are made to the three-dimensional image processing apparatus prior to the execution of such height inspection and image inspection processing will be described with reference to the flowchart of FIG. First, in step S81, a setting image (setting image) is selected. Here, it is also possible to input an image to be inspected in advance and recall the image saved as a registered image, or to acquire a new input image and make settings for it. Here, the input image obtained by imaging the work is registered as a registered image as an alternative to the input image that is sequentially input during operation. Further, the registered image registered in advance may be called. Next, in step S82, the gradation conversion method is selected. Here, the user is prompted to select either static conversion or dynamic conversion. Next, in step S83, the gradation conversion parameter is adjusted. Here, when static conversion is selected in step S82, the gradation conversion parameter is adjusted for the image acquired in step S81. The method of adjusting the gradation conversion parameter will be described later. The procedure described above is an example, and may be in a different order. For example, the image may be acquired after the gradation conversion method is selected.
(Details of setting procedure)

Next, the details of the procedure at the time of setting will be described. In the three-dimensional image processing apparatus, necessary settings are made in advance in the setting mode prior to the operation mode. Various setting means for making such a setting can be provided, for example, on the controller unit 2 side. For example, in the example of FIG. 1, a console, which is a form of the input means 3 connected to the controller unit 2, can be used. Further, instead of or in addition to this, the function of such a setting means can be realized in the three-dimensional image processing program installed in the personal computer connected to the controller unit 2 as described above. Hereinafter, here, the details of the procedure for performing each setting using the three-dimensional image processing program installed in the personal computer shown in FIG. 1 are described in the user interface (GUI) screen of the three-dimensional image processing program shown in FIGS. 9 to 113. The explanation will be based on. In these GUI examples, a distance image is displayed as a "height image" and a luminance image is displayed as a "shade image".
(Registration process of distance image and brightness image)

First, the distance image and the luminance image are registered. Here, the "imaging" processing unit 263 is set from the initial screen 260 of the three-dimensional image processing program shown in FIG. Specifically, the button 263 of the "imaging" processing unit is pressed. As a result, the image switching to the imaging setting menu 269 of FIG. 10 is switched.
(3D image processing program)

In the GUI screen example of FIG. 10, a first image display area 111 for displaying an image is provided on the right side of the screen, and a setting item button area 112 in which a plurality of buttons representing a plurality of setting items are arranged is provided on the left side. In the setting item button area 112, the "image registration" button 113, the "imaging setting" button 284, the "camera setting" button, the "trigger setting" button, the "flash setting" button, the "lighting volume" button, and the "lighting expansion unit" Button, "Save" button, etc. are provided. The user can select a desired setting item button from the setting item button area 112 and set necessary setting items.

In the image pickup setting menu 269 of FIG. 10, when the "image registration" button 113 provided in the setting item button area 112 is pressed, the screen is switched to the image registration screen 270 of FIG. From this screen, various settings such as registration target, camera selection, registration destination, etc. can be made. Here, after adjusting to a desired image with various buttons or the like provided in the operation area, registration, that is, saving of image data is performed. Here, the distance image is displayed in the second image display area 121, and the image variables assigned to this image are displayed in the operation area 122. In the example of FIG. 11, "Camera 1" is selected in the "Camera selection" field 271 for selecting the imaging means, and the image variable "& Cam1Img" is set to "Camera selection" as the distance image captured by this "Camera 1". It is displayed below the column 271.

When the "Register" button 272 provided at the bottom of the operation area is pressed after the setting is completed, registration of the distance image currently being displayed in the second image display area 121 is started as shown in FIG. 12, and the progress status is graphically displayed. Is displayed. Further, as shown in FIG. 13, the luminance image is also registered. In this example, the distance image is first stored in the distance image storage unit 38a, and then the luminance image is stored in the luminance image storage unit 38b. In addition, the image variable "&Cam1Img" for the distance image and the image variable "&Cam1GrayImg" for the luminance image are also recorded. Since these image variables are assigned unique variables to each image, they can be used as an index when calling a registered image. However, this example is an example, and the registration order of each image may be reversed or may be registered at the same time. In this way, by simultaneously saving the distance image and the luminance image as the registered images, the user can save labor in registering each image. However, it is also possible to individually register the distance image and the luminance image as registered images.
(Phase shift method)

Here, the phase shift method will be described as one of the methods for measuring the displacement and the three-dimensional shape of the work in a non-contact manner. The phase shift method is also called a grid pattern projection method, a fringe scanning method, or the like. In this method, a light beam having a grid pattern in which the illuminance distribution is varied in a sinusoidal manner is projected onto the work. Moreover, three or more lattice patterns with different phases of the sine wave are projected, and each brightness value of the height measurement point is imaged for each pattern from an angle different from the projection direction of the light beam, and the lattice pattern is obtained from each brightness value. Calculate the phase value of. Depending on the height of the measurement point, the phase of the lattice pattern projected on the measurement point changes, and a ray having a phase different from the phase observed by the ray reflected at the reference position is observed. Therefore, the phase of the light beam at the measurement point is calculated, and the height of the measurement point (hence the object) is measured by substituting it into the geometric relational expression of the optical device using the principle of triangulation, and the three-dimensional shape is obtained. The method. According to the phase shift method, the height of the work can be measured with high resolution by reducing the grid pattern period, but the range of the measurable height is a low height within 2π in terms of the amount of phase shift. Only those with a small height difference can be measured.
(Spatial coding method)

Therefore, the spatial coding method is also used. According to the space coding method, the space irradiated with light is divided into a large number of small spaces having a substantially fan-shaped cross section, and a series of space code numbers are assigned to these small spaces. Therefore, even if the height of the work is high, that is, even if the height difference is large, the height can be calculated from the space code number as long as it is in the space irradiated with light. Therefore, the shape of a high work piece can be measured as a whole. As described above, according to the spatial coding method, the allowable height range (dynamic range) is wide, and the shape of a high-height workpiece can be measured as a whole.

Next, a setting example for capturing a distance image with the imaging means will be described with reference to FIGS. 14 to 42. Prior to the registration of the above-mentioned image, when the "imaging setting" button 284 is pressed from the imaging setting menu 269 of FIG. 10, the imaging setting screen 280 of FIG. 14 is displayed. On the imaging setting screen 280. You can make basic settings related to imaging. For example, when the "imaging valid setting" button is pressed, the imaging valid setting screen of FIG. 15 is displayed, and the imaging means connected to the three-dimensional image processing device, that is, the camera can be selected. For example, as an imaging means, a distance image is processed in three dimensions by replacing or in addition to a monochrome CCD camera or a color CCD camera that captures a normal luminance image, or by connecting a camera capable of acquiring height information. It can be taken into the device. Further, when a plurality of image pickup means are connected to the three-dimensional image processing device, one or more image pickup means to be used can be selected.

Further, when the "detailed setting" button 282 provided in the operation area is pressed from the imaging setting screen 280 of FIG. 14, the three-dimensional measurement setting screen 290 shown in FIG. 16 is obtained. Note that in FIG. 16, different workpieces are displayed for convenience of explanation. The three-dimensional measurement setting screen 290 of FIG. 16 has a “display by continuous update” column 292, which corresponds to a real-time update means, a shutter speed setting column 294, which corresponds to a shutter speed setting means 49, a shade range setting column 296, a preprocessing setting column 310, and measurement. It includes an impossible reference setting field 312, an even interval processing setting field 314, a space code setting field 316, a projector selection setting field 318, a "display image" selection field 322, and the like.
(Real-time update means)

Here, when the setting is changed in the operation area, a real-time update means for updating the image displayed on the second image display area 121 to the changed setting is provided. The real-time update means can be switched ON / OFF. In the screen example of FIG. 16, the real-time update function can be operated by turning on the "display by continuous update" column 292 provided in the upper part of the operation area as one form of the real-time update means.

In the example of FIG. 16, items that can be set in the operation area include shutter speed, shading range, preprocessing, unmeasurable reference, even interval processing, spatial code, projector selection, display image, and the like. Hereinafter, the description will be given sequentially.
(Shutter speed setting means 49)

In the example of FIG. 16, a shutter speed setting column 294 is provided as one aspect of the shutter speed setting means 49 for adjusting the shutter speed at the time of imaging by the imaging means. The user can specify the shutter speed from the shutter speed setting field 294. Here, from the drop-down box, preset shutter speeds such as 1/15, 1/30, 1/60, 1/120, 1/240, 1/500, 1/1000 ,,, 1/20000 Select. The number of seconds corresponding to the selected numerical value is displayed in the numerical display field 295 on the right side. It is also possible to directly specify an arbitrary shutter speed numerically. For example, if "Numeric value input" is selected as a drop-down box option, the grayout of the numerical value display field 295 is canceled and the numerical value can be directly input. In this way, the exposure time of the camera (imaging element), which is an imaging means, is adjusted based on the numerical value specified in the shutter speed setting field 294. When adjusting the shutter speed, it is easier to confirm by displaying the shade image of the luminance image in the second image display area 121 rather than the distance image. Further, by the above real-time update function, the image whose shutter speed is changed in the shutter speed setting field 294 is promptly reflected in the second image display area 121, so that the user can visually check whether the current setting is appropriate or not. It can be confirmed and the adjustment work can be easily performed.
(Shadow range setting field 296)

In the shading range setting field 296, the dynamic range of the luminance image, which is a shading image, is adjusted. Here, the dynamic range is increased or decreased by selecting any of "low (-1)", "normal (0)", and "high (1)" from the drop-down box.
(Pretreatment setting field 310)

The pre-processing setting field 310 defines a common filter processing to be performed before the head portion generates a distance image. As the common filter processing, for example, an averaging filter, a median filter, a Gaussian filter, and the like can be considered. Here, as the filter processing for the pattern projection image, in the example of FIG. 17, one of "None", "Median", "Gaussian", and "Average" is selected by the drop-down box. The filter processing can be performed not only on the preprocessing performed before generating the distance image, but also on the distance image obtained on the head unit 1 side.
(Unmeasurable standard setting column 312)

In the non-measurable reference setting field 312, the level at which the noise component is cut is set. That is, the height is not measured by the amount set in the unmeasurable reference setting field 312. In the measurement of three-dimensional height information using a pattern projection image, accurate height information cannot be measured without a certain amount of light. On the other hand, when multiple reflections occur, it is too bright and it is necessary to reduce the amount of light. In this way, the noise component cut amount is selected according to the captured pattern projection image. Specifically, in order to calculate the height information of each pixel, the threshold value to be regarded as invalid data due to noise is determined for the data.

Here, as shown in FIG. 18, one of "high", "medium", "low", and "none" is selected from the drop-down box. If "None" is selected, the height is measured for all pixels without cutting the noise component. For example, in the example shown in FIG. 19, "None" is selected in the non-measurable reference setting field 312, and height data is calculated at all points including noise data. Although it is difficult to distinguish from this screen, an incorrect height is measured by noise data at the corners of the work.

On the other hand, in the example shown in FIG. 20, “Low” is selected in the unmeasurable reference setting field 312, and the points where the height information based on the noise data is invalid are reduced. Further, in the example shown in FIG. 21, “Medium” is selected in the unmeasurable reference setting field 312, and the points where the height information is invalid are further reduced.

On the other hand, it can also be confirmed from FIG. 21 that the black spot indicating that the measurement is impossible increases especially in the lower left region of the work, and the position where the height cannot be measured because it is regarded as noise is increasing. Further, in the example shown in FIG. 22, a state in which “high” is selected in the unmeasurable reference setting field 312 is shown, and as a result of excessive removal of the noise component, even the data originally desired to be retained is lost. Can be confirmed. As described above, when the setting of the non-measurable reference indicating the noise removal threshold value is too low, the height is calculated based on the noise. On the other hand, if it is too high, the part that you originally wanted to keep will be considered invalid. Therefore, by using the above real-time update function, the user adjusts the setting of the non-measurable standard, confirms the adjusted image in the second image display area, and directly refers to the setting result with the image. It can be adjusted to an appropriate value.
(Equal interval processing setting field 314)

In the uniform interval processing setting field 314, the error due to the angle of view is corrected. The even interval processing setting field 314 functions as the interval equalization processing setting means 47. In the uniform interval processing setting field 314, ON and OFF can be selected as shown in FIG. 23. By turning on the uniform interval processing, distance images arranged at equal pitches in the xy direction are acquired. Here, an even-pitch image whose position in the XY direction is evenly spaced regardless of the height (Z coordinate) is displayed in the second image display area 121. For example, in an application for inspecting dimensions on the XY plane, it is necessary to turn on the uniform interval processing. The part that has been corrected and the data is lost is treated as invalid. 24 and 25 show a state in which the uniform interval processing is turned on. In the example of FIG. 24, the “display image” selection field 322 is displayed in the second image display area 121 as a “height image”, that is, a distance image, and in FIG. 25, a “shade image” is selected to display a luminance image. The displayed examples are shown respectively.

On the other hand, when the uniform interval processing is turned off, an image (Z image) as seen visually is obtained as shown in FIG. 26, and distortion occurs in the XY direction toward the edge of the screen. However, since the even interval processing is not performed, the time required to display the image can be shortened. Note that FIG. 27 shows an example in which a “shade image” is selected in the display image field and a luminance image is displayed in the second image display area 121.
(Space code setting field 316)

In the space code setting field 316, whether or not to use the space coding method is selected. That is, the space code setting field 316 functions as the space code conversion switching means 45. In this three-dimensional image processing apparatus, the phase shift method is indispensable for generating a distance image, and in addition to the phase shift method, whether or not the spatial coding method is applied can be selected in the spatial code setting field 316. In the space code setting field 316, ON and OFF can be selected as shown in FIG. 28. When the space code setting field 316 is turned ON, the height is measured by the combination of the space coding method and the phase shift method. An example of this is shown in FIGS. 29 and 30. In these figures, FIG. 29 shows a state in which a distance image is selected as an image to be displayed in the second image display area 121. Specifically, the "height image" is selected in the "display image" selection field 322. On the other hand, FIG. 30 shows a state in which the luminance image is displayed in the second image display area 121, and the “shade image” is selected in the “display image” selection field 322. By using the spatial coding method in addition to the phase shift method, an appropriate distance image can be obtained. Specifically, since the phase jump can be corrected (phase unwrapped) by the phase shift method by the spatial coding method, it is possible to measure with high resolution while ensuring a wide dynamic range of the height. However, the imaging time is about twice as long as that in the case of OFF.

On the other hand, as shown in FIGS. 31 and 32, when OFF is selected in the space code setting field 316, the height is measured only by the phase shift method. In this case, since the height measurement dynamic range is narrowed, in the case of a workpiece having a large difference in height, if the phase shifts by one cycle or more, the height cannot be measured correctly. On the contrary, in the case of a work whose height does not change much, since the striped image is not imaged or combined by the spatial coding method, the processing can be speeded up by that amount, and the imaging time can be halved. For example, when measuring a workpiece with a small difference in the height direction, it is not necessary to take a large dynamic range, so the processing time should be shortened while maintaining highly accurate height measurement performance even with the phase shift method alone. Can be done. In this case, since the height measurement dynamic range is narrowed, in the case of a workpiece having a large difference in height, if the phase shifts by one cycle or more, the height cannot be measured correctly. On the contrary, in the case of a work whose height does not change much, the advantage that the imaging time can be halved can be obtained by turning off the spatial code.

In the example of FIG. 31, a state in which a "height image", that is, a distance image is displayed in the "display image" selection field 322 is shown, while in the example of FIG. 32, a "shade image" is displayed in the "display image" selection field 322. Is displayed.

Although the phase shift method is indispensable in this example, ON / OFF of the phase shift method may be selectable.
(Projector selection setting field 318)

The projector selection setting field 318 functions as a floodlight switching means 48 for switching ON / OFF of the first projector and the second projector. Here, in the projector selection setting field 318, the light projecting means (projector) to be used is selected from the first projector and the second projector, which are the two light projecting means. In the example of the projector selection setting field 318, as shown in FIG. 33, "1" (first projector), "2" (second projector), "1 + 2" (first projector and second projector) from the drop-down box. Select one of the projectors).

When "1" or "2" is selected in the projector selection setting field 318, that is, in the case of one-sided projection which is the projection from either the first projector or the second projector, the portion shaded by the projection. Height is not measured. In the example of FIG. 34, an example in which "1" is selected in the projector selection setting field 318 is shown, and in the example of FIG. 35, an example in which "2" is selected is shown. On each screen, the data of the shadowed part is displayed in black, and it can be confirmed from each screen that there is an area on the work where the height cannot be measured. Further, as is clear from these figures, it can be seen that the unmeasurable region differs depending on the light projecting means. In other words, even if one of the light projecting means cannot measure the area, the other light projecting means can project the light, and therefore there are many areas where the height can be measured. Therefore, by combining these, the unmeasurable region can be reduced. In particular, by arranging the workpieces so as to be sandwiched between the first projector and the second projector from both sides, the first projection from the first projector and the second projection from the second projector are in opposite directions. Since the work is irradiated, it is possible to reduce the risk of shadowing due to the projection of light from the other, which is in the opposite direction, even if the region is shadowed by the projection of either one.

Specifically, as shown in FIG. 36, when "1 + 2" is selected in the projector selection setting field 318, both the first projector and the second projector can switch to both projections. In this state, even a portion that is shaded by the light projected from one of the projectors can be interpolated as long as the other projector can project the light. However, in this case, it takes about twice as long as the one-sided projection. The user selects which projection to use according to the degree of unevenness of the workpiece to be inspected, the allowable imaging time, and the like.
("Display image" selection field 322)

In the "display image" selection field 322, an image to be displayed in the second image display area 121 is selected. For example, by selecting the display target according to the purpose of inspection, the validity of each setting can be visually confirmed from the actually displayed image. In particular, by turning on the above-mentioned real-time update means, it is possible to sequentially update the setting changes and compare them before and after the change, so that the settings can be adjusted based on the image so that the image is intended according to the application. .. In addition, even a beginner who is not familiar with the meaning of each setting parameter can obtain the advantage of being able to set while looking at the image. In this example, as shown in FIG. 37, from the “display image” selection field 322, “height image”, “shade image”, “overexposure / blackout image”, “striped projection-projector 1”, “stripes”. Select one of "flooding-projector 2". The "height image" is a distance image, and displays a colored image that is color-coded into contour lines for each height. A "shade image" is a luminance image. In this example, an image obtained by synthesizing a plurality of pattern projection images captured based on the phase shift method is used as a luminance image. However, it is also possible to irradiate the work with illumination, capture an optical image with the imaging means, and use it as a luminance image.
(Abnormal point highlighting means 62)

Further, the three-dimensional image processing apparatus includes the abnormal point highlighting means 62 as shown in FIG. For example, the "overexposed / underexposed image" that can be selected in the "display image" selection field 322 described above partially includes saturated overexposed pixels, underexposed pixels with insufficient light intensity, and the like with respect to the luminance image. It is a colored image. In this way, by highlighting the part of the image where the accurate value is not obtained and the measurement accuracy is considered to be unreliable by the coloring process, the part with low measurement accuracy is visually displayed to the user. It makes it easier to confirm whether or not an image suitable for the desired inspection application is obtained. In this example, the overexposed pixels are colored yellow and the underexposed pixels are colored blue. This allows the user to visually confirm how the whitened areas are distributed in the image using the color as a clue. Further, in the lower part of the second image display area 121, the number of overexposed pixels and underexposed pixels is counted and displayed. With reference to this, the user adjusts each setting item so that the number of these pixels approaches zero.

The colors and modes to be colored are not limited to this, and various known modes such as displaying in other colors or blinking can be appropriately used. In addition, by changing the color to be colored by the overexposed pixel and the underexposed pixel, it is possible to notify the user of the reason why the measurement reliability is low, and thus it becomes easy to take countermeasures. However, similar colors and highlights may be applied to the overexposed pixels and the underexposed pixels.

The “striped projection-projector 1” is a pattern projection image expressed by shading obtained by pattern projection only with the first projector. Further, the "striped projection-projector 2" is a pattern projection image obtained only by the second projector. FIG. 38 shows an example in which “striped projection-projector 1” is selected in the “display image” selection field 322 to display a pattern projection image of the first projector in the second image display area 121, and FIG. 39 shows “ An example in which the pattern projection image of the second projector is displayed in the second image display area 121 by selecting "Striped projection-projector 2" is shown. By checking the striped image from this screen, it is possible to identify the cause when the height of the work cannot be measured, for example, because the material of the work is translucent, the projected light is submerged, and a pattern projection can be obtained. It can be confirmed from the original pattern projection image before the distance image is generated that this is a difficult work.

The user confirms whether the shutter speed and the shading range are appropriate while referring to the image displayed in the second image display area 121, and adjusts them to appropriate values. Specifically, in a state where the overexposed / underexposed image is displayed in the second image display area 121, the adjustment is made while confirming that the overexposed pixels and the underexposed pixels are reduced. For example, by adjusting the shutter speed in the shutter speed setting field 294, blackout pixels, that is, a portion where the amount of light is insufficient and too dark is eliminated. In addition, by adjusting the shading range, overexposed pixels, that is, areas that are too bright are eliminated. In the example of FIG. 16, there are many blackout pixels because it is too dark in the whiteout / blackout image. Therefore, the shutter speed is adjusted.

For example, when the shutter speed setting field 294 is set to "1/120" in FIG. 16 and switched to "1/15" as shown in FIG. 40, the luminance image becomes slightly brighter and the number of blackout pixels becomes 0. You can see that it has become. However, the number of overexposed pixels has increased on the contrary. Therefore, when the shutter speed is switched to "1/30" as shown in FIG. 41, it can be seen that the number of overexposed pixels remains 0 and the number of overexposed pixels also decreases.

Further, when the shading range setting field 296 was set to "low (-1)" in FIG. 41 and switched to "normal (0)" as shown in FIG. 42 and raised by one step, the overexposed pixels decreased. I understand. Further, as shown in FIG. 43, when switching to “high (1)”, it can be seen that the overexposed pixels became 0. As a result, both the number of overexposed pixels and the number of underexposed pixels become 0, and the adjustment work of the shutter speed and the shading range is completed.

After setting the desired imaging conditions as described above, the distance image and the luminance image are registered as shown in FIGS. 12 to 13 described above, and the image registration work is completed.
(Height measurement setting procedure)

Next, a procedure for setting an area for height measurement with respect to the input image of the work to be inspected, which is sequentially input during operation, will be described with reference to FIGS. 44 to 55. Here, a typical work representing a plurality of input works is registered in advance as a registered image by the above procedure, and an area for height measurement is specified for the registered image.

Specifically, the "height measurement" processing unit 266 is added from the initial screen 260 of FIG. 9 as shown in FIGS. 44 to 45. In the example of FIG. 44, “Add” is selected from the first submenu 370 displayed by right-clicking or the like at the lower part of the “imaging” processing unit 263 in the flow display area 261, and “Measurement” in the second submenu 372 is selected. Among the inspection processes listed in the "Measurement" menu 373 displayed by selecting "", the "Height measurement" processing unit 266 that performs "Height measurement" is added. As a result, as shown in FIG. 45, a new “height measurement” processing unit 266 is added to the lower part of the “imaging” processing unit 263 in the flow display area 261. As described above, the "measurement" menu 373 functions as an inspection process selection means for selecting the inspection process executed by the inspection execution means.
(Inspection target area setting screen 120)

Next, a procedure for setting an area as a setting to be performed by the “height measurement” processing unit 266 will be described with reference to FIGS. 46 to 49. First, when the edit screen of the "height measurement" processing unit is called from the screen of FIG. 45, the screen shifts to the height measurement setting screen 460 shown in FIG. In the GUI screen example of FIG. 46, as in FIG. 10, a first image display area 111 for displaying an image is provided on the right side of the screen, and a setting item button area 112 in which a plurality of setting item buttons are arranged is provided on the left side. In the setting item button area 112, the "image registration" button 113, the "image setting" button 114, the "area setting" button 115, the "preprocessing" button 117, the "detection condition" button 118, the "detailed setting" button 119, A "judgment condition" button, a "display setting" button, a "save" button, and the like are provided. From this screen, when the user presses the "area setting" button 115 corresponding to the inspection target area setting means, the screen transitions to the inspection target area setting screen 120 shown in FIG. 47. On the inspection target area setting screen 120, an area to be inspected can be specified. In the example of FIG. 47, the second image display area 121 is provided on the left side of the screen, and the operation area 122 for performing various operations is arranged on the right side of the screen. In the upper part of the operation area 122, a "display image" selection field 124 for selecting an image displayed in the second image display area 121 is provided. In the example of FIG. 47, the registered image is selected in the "display image" selection field 124. Further below, a "measurement area" setting field 126 is provided as an inspection target area setting means for designating an area for executing inspection.

In the "measurement area" setting field 126, a predetermined area can be selected. Here, when the “measurement area” setting field 126 is selected, a drop-down box is displayed as shown in FIG. 48, and the shape of the desired measurement area can be selected. In this example, "None", "Rectangle", "Rotating Rectangle", "Circle", "Ellipse", "Circumference", "Arc", "Polygon", etc. "Composite area" and the like are displayed. When "None" is selected, the entire image displayed in the second image display area 121 is used as the inspection target area.

Further, detailed dimensions and the like can be set according to the shape selected in the "measurement area" setting field 126. In the example of FIG. 48, an eraser is used for the work, and in the “measurement area” setting field 126, an example in which the “rotating rectangle” is selected as shown in FIG. 49 is shown. When the "edit" button 128 is pressed in this state, the measurement area edit screen 130 shown in FIG. 50 is displayed. In the example of FIG. 50, the rotation rectangle is superimposed and displayed on the work in the second image display area 121. Here, a rectangular measurement area is drawn on the eraser case and displayed on the distance image. In addition, the basic vector of the rotating rectangle is displayed as an arrow in the frame shape of the rotating rectangle, and the width and height of the rotating rectangle, the XY coordinates of the center, the tilt angle of the basic vector, etc. are displayed on the measurement area editing screen 130. Is displayed. The user can arbitrarily adjust the shape, position, etc. of the rotating rectangle by directly inputting a numerical value from the measurement area editing screen 130 or by operating the handle displayed on the rotating rectangle with a mouse or the like.

The items that can be set on the measurement area edit screen 130 change according to the shape selected in the "measurement area" setting field 126. For example, when "circumference" is selected, as shown in FIG. 51, it is possible to set parameters related to the circumference, such as specifying the dimensions of the outer diameter and the inner diameter of the circumference. Further, it is also possible to specify the mask area from the screen of FIG. 47. As the mask area, a circular shape, a donut shape, a rectangular shape or other polygonal shape, a free curve, or the like can be specified. In this way, it is possible to appropriately set the inspection target area according to the shape of the work to be inspected, eliminate unnecessary areas for inspection such as a perforated portion and the background, and improve the processing efficiency. ..
(Second measurement display area)

When the measurement area is set in this way, the set measurement area is superimposed and displayed on the work as shown in FIG. 49. Subsequently, when designating another measurement area, the same operation is repeated. That is, as shown in FIG. 52, another second "height measurement" processing unit 266B is added below the "height measurement" processing unit 266 in the flow display area 261. Then, as shown in FIGS. 53 and 54, a rotating rectangle is set as a new measurement area on the area of the eraser, which is a work, which is not covered with the case. As a result, as shown in FIG. 55, the newly set second measurement area is superimposed and displayed on the work.

In the above example, one height measurement process is performed by one "height measurement" processing unit. That is, in order to perform a plurality of height measurement processes, it is necessary to add a plurality of "height measurement" processing units. However, it goes without saying that it is possible to configure one "height measurement" processing unit so that a plurality of height measurement processes are performed.
(Measurement processing)

When the setting of the measurement area is completed in this way, the process of actually performing the measurement is added next. Here, as shown in FIG. 56, a "numerical calculation" processing unit that performs "calculation" is added below the second "height measurement" processing unit 266B in the flow display area 261. Numerical calculation, image calculation, calibration, image concatenation, etc. can be selected as the content of the calculation executed by the "numerical calculation" processing unit. Here, as shown in FIG. 57, a "numerical calculation" processing unit in which numerical calculation is selected is added.
("Numerical calculation" processing unit)

In the "numerical calculation" processing unit, a specific calculation formula can be input. For example, as shown in FIG. 58, a numerical calculation edit screen on which a mathematical expression can be directly input is displayed, and the user defines the mathematical expression. Here, a calculator-shaped input pad is prepared, and edit buttons for copying, cutting, pasting, etc. are also prepared to facilitate the creation of arithmetic expressions. The user inputs a desired arithmetic expression from this screen. An example of the input arithmetic expression is shown in FIG.

When the content of the numerical calculation process is defined in this way, the calculation formula is displayed on the third image display area 262 on the initial screen 260 as shown in FIG.
("Area" processing unit)

Furthermore, in the example of FIG. 61, an "area" processing unit is added under the "numerical calculation" processing unit. The "area" processing unit defines the conditions for actually making a pass / fail judgment. Specifically, in order to appropriately change the gradation conversion conditions for converting the distance image to the low gradation distance image according to the registered image and the input image, the gradation conversion parameter (details will be described later). Set the area for acquiring the reference information, the condition for extracting the height from this area, the condition for performing the filtering process when generating the distance image, and the like. That is, in the "area" processing unit, area setting, height extraction, preprocessing, determination, and the like are set. First, the procedure for setting the area is the same as that of the registered image described above. That is, when the "area setting" button 115 arranged in the setting item button area 112 is pressed from the area setting screen 620 as shown in FIG. 62, the area setting screen shown in FIG. 63 is displayed, and the target area is specified. Again, select the rotating rectangle and specify more detailed coordinates as needed. In this way, the area in the "area" processing unit is determined, and as shown in FIG. 64, the rotating rectangle is superimposed and displayed on the work in the second image display area.
(Height extraction setting screen)

Next, set the height extraction. The height extraction setting is to set the gradation conversion parameter when performing the gradation conversion. That is, when the "height extraction" button 116 is pressed from the setting item button area 112 of FIG. 62, the screen shifts to the height extraction selection screen 140 shown in FIG. 65, and the display image, the extraction method, and the like can be selected. In the height extraction selection screen 140, as in FIG. 47, a second image display area 121 is provided on the left side of the screen, and an operation area 122 for performing various operations is arranged on the right side of the screen. In the upper part of the operation area 122, a "display image" selection field 124 for selecting an image displayed in the second image display area 121 is provided. In the example of FIG. 65, the registered image is selected in the "display image" selection field 124. Further below, an extraction method selection means 142 for selecting an extraction method for the height extraction function is provided.

Here, the "height extraction" button 116 functions as the gradation conversion condition setting means 43 for setting the gradation conversion parameter for performing the gradation conversion of the distance image by the gradation conversion means. In particular, the gradation conversion condition setting means 43 is displayed when a process that does not require image height information is selected by the inspection process selection means. On the contrary, when the processing that requires the height information of the image is selected by the inspection processing selection means, the gradation conversion condition setting means is not displayed. Specifically, when the "height measurement" processing unit 266 is selected as the inspection processing tool, the "height extraction" button is not displayed in the flow display area 261. For other inspection processing tools such as "area" processing unit, "blob" processing unit 267, "color inspection" processing unit 267B, "Shapetrax2" processing unit 264, "position correction" processing unit 265, etc., "height" The “Extract” button 116 is displayed, and the gradation conversion condition can be set. By doing so, when gradation conversion is required, the gradation conversion condition setting means 43 is displayed to prompt the user for the necessary settings, while when gradation conversion is not required, gradation conversion is performed. By hiding the means for setting conditions, users can avoid being confused by unnecessary settings, and an easy-to-use environment is realized.
(Extraction method selection means 142)

The extraction method selection means 142 specifies a gradation conversion method. For example, let the user select either static conversion or dynamic conversion. In the example of FIG. 65, as options in advance, either "one-point designation" or "three-point designation (plane)" corresponding to static conversion, or "real-time extraction" corresponding to dynamic conversion is selected from the drop-down box. Let me.
(One point designation screen 150)

When "one-point designation" is selected from the extraction method selection means 142 on the screen of FIG. 65, the screen shifts to the one-point designation screen 150 of FIG. Note that FIGS. 66 to 96 show an example in which a 50-yen coin is used as a work for explanation. In the one-point designation screen 150 of FIG. 66, the height of the portion designated on the second image display area 121 is set as the reference height (reference height). In the example of FIG. 66, when the “extract” button 144 provided in the operation area 122 on the right side of the screen is selected, the screen shown in FIG. 67 is displayed, and an arbitrary position on the second image display area 121 on the left side of the screen can be specified. Will be. Here, the height extraction means is used to specify a position that is the center of the height in the work displayed in the second image display area 121. In the example of FIG. 67, the height extraction means is composed of the “extract” button 144 displaying the dropper-shaped icon SI, and when the “extract” button 144 is pressed, a point is formed on the second image display area 121. A pointer 146 is displayed. The position specified by the pointer 146 is registered as the height in the middle of the distance range.

Further, the range for obtaining the height around the point designated by the pointer 146 can be specified in the "extraction area" designation field 145. In the “extraction area” designation field 145, one side of the area for which the average height is to be obtained is designated by the number of pixels. In the example of FIG. 67, “16” is specified in the “extraction area” designation field 145, and the average height in the area of 16 pixels × 16 pixels centered on the point specified by the pointer 146 is extracted. , The height extracted by the pointer 146. The size of the area designated by the pointer 146 on the second image display area 121 can be changed in conjunction with the numerical value specified in the "extraction area" designation field 145.

Further, the height information of the designated portion is displayed as a numerical value in the "Z height" display column 152 (in the example of FIG. 68, 1.253 is displayed in the "Z height" display column 152). For example, when expressing a distance range in 2 ⁸ = 256 gradations (0 to 255), the height specified height extraction means, the gain as a central value (density value / mm; details described later) 128 Is set to be. With this configuration, the user can directly specify the height to be inspected on the screen, and the gradation is converted into a low-gradation distance image within the range centered on the specified height, so that the required height is used. It is possible to avoid a situation where information is lost.
(Simple display function)

When the distance range and span are determined as the gradation conversion parameters required for gradation conversion as described above, it is possible to perform gradation conversion from a high gradation distance image to a low gradation distance image. Further, as shown in FIG. 68, a low gradation distance image that has been gradation-converted under the gradation conversion conditions currently set in the operation area 122 is simply displayed on the second image display area 121. .. Further, when the gradation conversion condition is changed in the operation area 122, the simple display of the low gradation distance image after the gradation conversion on the second image display area 121 also corresponds to the changed gradation conversion condition. Will be updated. As a result, the user can visually and quickly confirm the change after the adjustment of the gradation conversion condition, and can easily perform the adjustment work by trial and error. In this way, the image displayed in the second image display area 121 displays a mode for displaying a distance image before gradation conversion, a mode for displaying a low gradation distance image after gradation conversion, and a normal luminance image. It can be changed by switching the mode to make it.
(Gain adjustment means)

Further, the user can perform gain adjustment, which is one of the gradation conversion parameters, by using the gain adjusting means. In the example of FIG. 68, the emphasis method setting field 154 is provided in the middle of the operation area 122, and the gain adjustment field 156 is arranged here as the gain adjustment means. The current gain is numerically displayed in the gain adjustment column 156. Here, the gain [gradation / mm] is a parameter corresponding to the span when performing gradation conversion. For example, when converting a distance image of 16 gradations to 8 gradations, it is set how many gradations per 1 mm are converted among the 8 gradations. Increasing the gain value results in gradation conversion with clear contrast. For example, when the gain value is set to 100 [gradation / mm], the gradation conversion is set to 0.010 mm per gradation. If the height information of the distance image before conversion has a resolution of 0.00025 mm per gradation, the difference between the obtained reference plane and the input height data is N gradation before conversion. After conversion, it can be calculated as N [gradation] × 0.00025 [mm / gradation] × 100 [gradation / mm] = N × 0.025 gradation.

Here, the reference plane is a plane obtained by a method such as one-point designation, average height reference, three-point designation, plane reference, free-form surface reference, etc., which will be described later, and is a reference surface at the time of gradation conversion. For example, as shown in FIG. 69A, when the cross-sectional profile of the distance image (input image) before conversion of 16 gradations has a shape as shown by a solid line, the reference plane is shown by a wavy line. The profile of the low gradation distance image obtained by converting such an input image from 16 gradations to 8 gradations is as shown in FIG. 69B, and gain (conversion coefficient) as it is with respect to the difference from the reference plane. It will be in a state where it takes.

Further, the height per gradation (the reciprocal of the gain value) is automatically calculated according to the gain value described above, and can be displayed at the same time. In the example of FIG. 68, 250 [gradation / mm] is displayed as the gain value, and 0.0040 mm is displayed as the height per gradation. The user can adjust the gain value by changing the gain value. For example, when the gain value is increased, as shown in the screen of FIG. 68 to the screen of FIG. 70, the height information can be inspected in detail by emphasizing the density difference, but the inspectable height range becomes narrower. On the contrary, when the gain value is lowered, as shown in FIG. 71, it is possible to inspect a wide height range, but fine changes are impaired. When the gain is adjusted by the gain adjusting means in this way, the gradation conversion image obtained under the gradation conversion conditions can be confirmed in the second image display area 121. As a result, the user can adjust the gain value to an appropriate value according to the inspection purpose and the like while checking the gradation conversion image updated in real time.
(Extraction height setting)

Further, the setting item of the emphasis method can include the setting of the extraction height in addition to the gain value. For example, on the screen of FIG. 72, when the “detailed setting” button 158 provided at the lower right of the operation area 122 is pressed, the screen is switched to the highlighting method detailed setting screen 160 of FIG. 73, and the gain adjustment column described above is displayed in the highlighting method setting field 154. In addition to 156, the "extraction height" setting field 162 is displayed. In the "extraction height" setting field 162, as the height information to be extracted by the height extraction means, any of the high height information, the low height information, and the height information of both the high and low ones included in the area. Can be selected. Here, as shown in FIG. 74, one of “high side”, “low side”, and “both high and low” can be selected by the drop-down list provided in the “extraction height” setting field 162. For example, when "higher side" is selected, the gradation is converted so that the height of the position specified by the pointer 146 becomes the lower limit of the distance range. As a result, a low gradation distance image is generated in which only the side higher than the specified height is extracted. Similarly, when the "lower side" is selected, the gradation is converted so that the height of the position specified by the pointer 146 becomes the upper limit of the distance range. As a result, a low gradation distance image is generated in which only the side lower than the specified height is extracted. Further, in the case of "both high and low", the gradation is converted so that the height of the position specified by the pointer 146 is in the middle of the above-mentioned distance range. Pixels that are out of range after gradation conversion are clipped to black on the low side (pixel value 0 in the case of 8 gradations) and to white on the high side (pixel value 255).

Further, in the highlighting method detailed setting screen 160 of FIG. 73, a noise removal setting field 164 for removing noise and an invalid pixel designation field 166 for designating a value given to the invalid pixel are also provided.
(Noise removal setting field 164)

In the noise removal setting field 164, as one of the gradation conversion parameters, it is specified how many mm of difference from the reference plane is to be removed as noise. For example, if the noise removal parameter is set to 0.080 mm, the difference of 0.080 mm from the reference plane is removed. Here, assuming that the height information before conversion has a resolution of 0.00025 mm per gradation, the difference of 0.080 [mm] ÷ 0.00025 [mm] = 320 [gradation] is ignored. It becomes. This situation will be described with reference to FIGS. 75A to 75C. In these figures, as in FIG. 69A, FIG. 75A shows the cross-sectional profile of the distance image before conversion of 16 gradations with a solid line, the reference plane thereof with a wavy line, and the noise-removed range with a dashed-dotted line. There is. As a result of performing noise removal on such an input image with reference to the reference plane, the profile shown in FIG. 75B is obtained. Further, the profile of the low gradation distance image obtained by gradation conversion from 16 gradations to 8 gradations with respect to the distance image of FIG. 75B is as shown in FIG. 75C, and the gain (conversion coefficient) is obtained for the remaining components. It will be in the state of being applied.

Further, the effects of gain adjustment and noise removal will be described with reference to FIGS. 76A to 76F. First, it is assumed that a luminance image as shown in FIG. 76A and a distance image with high gradation (16 gradations) as shown in FIG. 76B are obtained. Here, the distance image of FIG. 76B has low gradation (8 gradations) with the initial settings (here, the gain is 100 [gradation / mm] and the noise removal is 0.000 [mm]). A low gradation distance image that has been gradation-converted to is shown in FIG. 76C. This low gradation distance image has a relatively low contrast. Therefore, when the gain is increased from this state, as shown in FIG. 76D, the low gradation distance image having high contrast is newly gradation-converted from FIG. 76B and displayed. However, the noise component is also large in this image. In the example of FIG. 76D, the gain is set to 1000 [gradation / mm] and the noise removal is set to 0.000 [mm]. Therefore, a low gradation distance image in which the amount of noise removed is increased from FIG. 76D is shown in FIG. 76E. Here, the gain is set to 1000 [gradation / mm] and the noise removal is set to 0.080 [mm]. As a result, the noise component was reduced, but on the other hand, it can be confirmed that noise having a height lower than the reference plane exists in the upper right of the "E" in the upper left. Therefore, when the "extraction height" setting field 182 shown in FIG. 74 or the like is set to the "higher side", the reference plane is converted to the lowest value (pixel value 0), so that the portion lower than the reference plane is As a result of being ignored and extracting only the side higher than the reference plane, a low gradation distance image as shown in FIG. 76F is obtained. For example, when the "extraction height" is set to the "high side" for the distance image (16 gradations) of the profile as shown in FIG. 77A, the result is that only the high side from the reference plane is extracted. A low gradation distance image (8 gradations) as shown in FIG. 77B can be obtained. In this way, it is possible to obtain a low gradation distance image having higher contrast and less noise component than that of FIG. 76C and the like. In this example, as the final gradation conversion parameters, the gain is set to 1000 [gradation / mm], the noise removal is set to 0.080 [mm], and the "extraction height" is set to "high side". Gradation conversion is performed from the distance image of 76B to the low gradation distance image of FIG. 76F.

When the conditions necessary for executing the one-point designation are set in this way, the input image changes from a high-gradation distance image to a low-gradation distance image according to the specified gradation conversion conditions, that is, the reference height and the like. The key is converted and displayed in the first image display area 111 as shown in FIG. 78.

Here, an example of a work in which the method of designating the reference plane by designating one point is effective will be described with reference to FIGS. 79A to 79B. FIG. 79A shows the work WK7 which has no flat inclination on the measurement surface of the work or does not affect the inspection process even if there is a slight inclination. Here, for the work WK7 in which numbers and character strings are three-dimensionally formed on the surface of the casting, an inspection process is performed in which whether or not the character strings are appropriate is read by OCR. In such an application, by pressing the "extract" button 144 shown in FIG. 67 or the like, a dropper-shaped icon SI is displayed on the second image display area 121. Then, the pointer 146 specifies one point on the upper surface of the work WK7, which is a plane (background surface) on which no character string is formed, as shown in FIG. 79A. As a result, gradation conversion is performed using the height of the extraction region (16 pixels in the example of FIG. 67) designated by the pointer 146 as a reference plane, and the image is converted to the low gradation distance image shown in FIG. 79B. In this low-gradation distance image, the character string portion protruding from the plane of the work WK7 is clearly extracted, so that accurate OCR can be easily executed. In this way, the one-point designation can be effectively used in cases where the inspection process is not affected even if the work is slightly tilted. In addition, the one-point designation has the advantage of being able to process at high speed with a low load.
(Specify three points)

The method of setting the gradation conversion condition by specifying one point has been described above. Next, a method of setting the gradation conversion condition by designating three points will be described based on the GUI screens of FIGS. 80 to 85. The three-point designation is a method of gradation-converting a distance image into a low-gradation distance image using a plane obtained from the three points designated by the user as a reference plane. The reference plane is also set to a height in the middle of the height range (distance range) to be gradation-converted into a low-gradation distance image, for example, in the height information of the distance image, as in the case of the reference height specified by one point described above. Alternatively, it can be set to the upper limit (highest position where gradation conversion is performed) or the lower limit (lowest position where gradation conversion is performed) of the distance range.

In the state where the "height extraction" button 116 is pressed on the GUI screen of the three-dimensional image processing program of FIG. 62 to shift to the height extraction selection screen 140 shown in FIG. 80, the extraction method selection means 142 is used as a gradation conversion method. , Select "Specify three points (plane)". As a result, the three-point designation screen 170 for setting the height extraction shown in FIG. 81 is displayed.
(Three-point designation screen 170)

On the three-point designation screen 170, three reference planes that serve as a reference for gradation conversion are designated and set on the second image display area 121. Therefore, the height extraction means is provided on the three-point designation screen 170 of FIG. 81. Specifically, by selecting the "extract" button 144 provided in the operation area 122 on the right side of the screen, the screen shown in FIG. 82 is displayed, and three arbitrary positions are set on the second image display area 121 on the left side of the screen. You will be able to specify points. Here, a point-shaped pointer 146 is displayed as the height extraction means as in FIG. 67, and the user sequentially specifies a desired position with a pointing device such as a mouse, a trackball, or a touch panel. When the first point is specified on the second image display area 121, the rectangular shape changes to a cross shape at the specified position as shown in FIG. 83 to indicate the specified position, and the next second point is displayed. Similarly, it can be specified with the pointer 146. At this point, the color distance image displayed in FIG. 82 is gradation-converted with reference to the horizontal plane including the height of the specified first point, and the low-gradation distance image after gradation conversion is a shade image. Is displayed in the second image display area. Further, when the second point is specified, the position of the second point changes from a rectangular shape to a cross shape as shown in FIG. 84, and the third point can be specified. At this point, the gradation conversion is performed again with reference to the inclined surface including the heights of the two designated points, and the low gradation distance image is updated. Then, when the third point is specified, the reference plane is set by the plane including these three designated points. Further, when each point is designated by the height extraction means, the height of the point currently designated on the height extraction screen display area may be displayed in the "Z height" display field 152.

Further, the inclination angle can be displayed as the information of the reference plane. In the example of FIG. 84, in the height extraction display field 172 provided in the operation area 122, the inclination of the reference plane in the X direction, the inclination in the Y direction, and the height of the third point in the Z direction are displayed.

Further, as with the one-point designation, the emphasis method can be specified as needed. For example, the gain adjustment means is used to adjust the gain, or the three-point designation "detailed setting" button 174 is pressed to call the three-point designation detailed setting screen 180 as shown in FIG. 85, and the emphasis method setting field 154 is displayed. In addition to the gain adjustment column 156, the "extraction height" setting column 182 is displayed, and as the height information to be extracted by the height extraction means, "high side", "low side", and "both high and low" from the drop-down list. You can choose either.

In this way, it is possible to perform gradation conversion of a distance image using an arbitrary plane defined by the three designated points as a reference plane. As a result, not only gradation conversion based on a horizontal plane as described above, but also gradation conversion using an inclined plane as a reference plane becomes possible. For example, in the application of inspecting scratches and foreign matter on the inclined surface of the work surface, the distance range becomes narrow if the inclined surface remains, but by setting the reference surface along the inclined surface, the inclined surface can be canceled efficiently. Can detect scratches and foreign matter. In this way, flexible gradation conversion can be realized by utilizing the height information according to the work and the inspection purpose.

Here, an example of a work in which the method of designating the reference plane by designating three points is effective will be described with reference to FIGS. 86A to 86D. FIG. 86A shows the work WK8 in which a flat inclination occurs on the measurement surface of the work or the presence of a minute planar inclination affects the result of the inspection process. Here, an inspection process is performed to detect the ball grid array (BGA) formed on the substrate. In such an application, by pressing the "extract" button 144 shown in FIG. 84 or the like, a dropper-shaped icon SI is displayed on the second image display area 121. Then, with the pointer 146, as shown in FIG. 86A, three points on the upper surface of the work WK8 where the BGA is not formed are designated. As a result, a plane including the designated three points is extracted as a reference plane, gradation conversion is performed, and the plane is converted into the low gradation distance image shown in FIG. 86B. In this low gradation distance image, the BGA protruding from the plane of the work WK8 is clearly extracted, so that the shape of the BGA can be confirmed by binarizing it as shown in FIG. 86C, for example. With this method, there is an advantage that even if the plane of the work WK8 is inclined, it can be accurately detected. If the work WK8 shown in FIG. 86A is inclined, for example, if one point is specified, the binarized image will be shown in FIG. 86D, and it cannot be detected correctly. On the other hand, in the three-point designation, the inclination is corrected as described above, and an accurate detection result can be obtained. As described above, the three-point designation is effective in the case where the inclination of the plane affects the inspection processing result.

Further, when the upper surface of the flat work WK9 as shown in FIG. 87A has a gentle dent, an inspection process for detecting the dent is considered. Here, as shown in FIG. 87A, the measurement area ROI is set to the area including the recess. As a result, gradation conversion is performed using a plane obtained from the height data of the entire measurement region ROI including the depression as a reference plane, and a low gradation distance image as shown in FIG. 87B is obtained. In this example, the reference plane is estimated by the method of least squares. Further, the obtained low gradation distance image is binarized to obtain the binarized image shown in FIG. 87C. As a result, the inclination can be corrected and only the dented portion can be stably extracted. If a binarized image is obtained by designating one point in a state where there is an inclination, the result shown in FIG. 87D is obtained, and it can be seen that it is difficult to detect the dent due to the inclined surface. In this way, the three-point designation is effective for estimating the reference plane with high accuracy. In terms of processing time, although it takes more processing time than specifying one point, it can be processed at a relatively high speed.

The above has described static conversion in which gradation conversion conditions are specified in advance at the setting stage and gradation conversion is performed under the specified conditions during operation. In other words, in static conversion, the gradation conversion parameter is a constant value regardless of the input image. Next, a specific example of dynamic conversion that adjusts the gradation conversion condition according to the input image to be inspected will be described. First, in the dynamic conversion, as a specific method for correcting the reference of the height information to be left when the distance image is gradation-converted to the low-gradation distance image,
(B1) An average height standard for gradation conversion using the average height (average distance) in the average extraction area specified for the input image as the average reference height.
(B2) A plane reference that generates an estimated plane in a designated area of an input image and converts gradation using this as a reference plane.
(B3) A free curved surface reference is included in which a free curved surface excluding a high frequency component is generated from an input image and gradation conversion is performed using this as a reference plane. Hereinafter, each method will be described in sequence.
(B1: Average height standard)

The average height reference is a method of calculating the average height in the designated average extraction area for each input image and converting the gradation using this as the average reference height. The average extraction area for defining the average reference height is set in advance prior to operation (step S83 in FIG. 8 described above). Hereinafter, an example of the procedure for designating the average extraction area in step S83 of FIG. 8 will be described based on the GUI of FIGS. 62, 65, 88 to 92. First, select the "height extraction" button 116 on the GUI screen of FIG. 62, proceed to the height extraction selection screen 140 of FIG. 65, and select "real-time extraction" corresponding to the dynamic conversion by the extraction method selection means 142. Then, the height dynamic extraction setting screen 190 of FIG. 88 is displayed. In this example, an example in which an eraser is used for the work again is shown. Next, in the "calculation method" selection field 192 provided below the extraction method selection means 142, a reference for dynamic conversion is specified. Here, as shown in FIG. 89, one of "average height reference", "plane reference", and "free curved surface reference" is selected in the drop-down box. Here, "average height standard" is selected. As a result, the screen shifts to the average height reference setting screen 210 of FIG. 90. Note that FIGS. 90 and 91 show an example in which a 50-yen coin is used for the work for convenience of explanation.

When setting the dynamic conversion, it is necessary to set the average reference height and the like for an image different from the distance image input in the actual operation. For this reason, an image corresponding to the work during operation is captured in advance and saved as a registered image, and when setting dynamic conversion, the registered image is read and used as a substitute for the work image during operation. Make various settings in the form. Therefore, on the screen shown in FIG. 65 or the like, the corresponding "registered image" is specified in the "display image" selection field 124.

On the average height reference setting screen 210, the separately set inspection target area is used as it is, or an arbitrary average extraction area is specified as needed. Any method can be used for designating the average extraction area, such as a rectangular shape, four corners, a circle by designating the center and radius, and a free curve. Further, only one point of the work can be specified, or conversely, the entire work or the entire image displayed in the second image display area 121 can be set as the average extraction area. Alternatively, as described above, the separately designated inspection target area can be used as the average extraction area. In these cases, the work of designating the average extraction area by the height extraction means may be omitted.
(Mask area)

It is also possible to specify a mask area in which the average height is not extracted with respect to the average extraction area. For example, when the "extraction area" button 194 provided in the operation area 122 is pressed from the screen of FIG. 90, the screen shifts to the mask area setting screen 220 shown in FIG. From the mask area setting screen 220, one or more mask areas unnecessary for extracting the average height can be specified. The mask area can also be specified by designating an arbitrary area such as a rectangular shape or a circular shape from the second image display area 121 as described above.

Further, the gain can be adjusted as needed. For example, when the "detailed setting" button 196 provided at the lower right of the operation area 122 is pressed from the screen of FIG. 90, the screen shifts to the average height reference detailed setting screen 230 shown in FIG. 92, and the gain is adjusted in the emphasis method setting field 154. In addition, detailed setting items such as specification of extraction height and noise removal are displayed.

When the average extraction area is defined in this way, the setting screen is closed. At the time of gradation conversion, gradation conversion is performed using the average value (average reference height) of the height information included in this average extraction region as the reference height. For example, the average reference height (in the case of 2 ⁸ = 256 gray scale, which is the central value of the distance range of 0 to 255 128) the center value of the distance range as a gradation conversion. Further, it is not always necessary to use all the height information of all the points included in the average extraction area, and the process can be simplified by appropriately thinning or averaging.

Then, during operation, dynamic conversion is performed by the procedure shown in FIG. 133, which will be described later. For example, a work transported on the line is imaged to generate a distance image (step S13301), the average height of the average extraction region set above is calculated (step S13302), and gradation conversion is performed based on this. This is executed to generate a low gradation distance image (step S13303), and the obtained low gradation distance image is inspected (step S13304). With this method, even if there are variations in the height direction of the workpiece, the reference plane for gradation conversion can be reset each time for each workpiece, so accurate inspection can be achieved regardless of the variations in the height direction of the workpiece. can.

Here, an example of a work in which the method of designating the reference plane based on the average height reference is effective will be described with reference to FIGS. 93A to 93B. FIG. 93A is a work WK7 which does not affect the inspection process even if there is no flat inclination on the measurement surface of the work or there is a slight inclination on the measurement surface of the work, as in FIG. 79A described above, and numbers and character strings are displayed on the surface of the casting. An inspection process is performed on the three-dimensionally formed work WK7 to read whether or not the character string is appropriate by OCR. In such an application, on the extraction area setting screen shown in FIG. 92 or the like, a measurement area ROI that determines the average height reference in a rectangular shape is set on the second image display area 121. Here, as shown in FIG. 93A, the plane surrounding the character string on the upper surface of the work WK7 is designated as the measurement area ROI. As a result, gradation conversion is performed using the height of the measurement area ROI as a reference plane, and the image is converted into the low gradation distance image shown in FIG. 93B. Even in this low gradation distance image, as in FIG. 79B, the character string portion protruding from the plane of the work WK7 is clearly extracted against the background, so that accurate OCR can be easily executed. Further, in FIG. 79A described above, since only one point on the work WK7 is specified, the selected point may be affected by noise, whereas in FIG. 93A, since it is specified as a plane, such noise. The advantage of being able to reduce the influence of
(Plane reference)

The above has described an example of performing dynamic conversion based on the average height. Next, as another dynamic conversion, a plane reference that estimates a plane included in the reference plane estimation region designated in advance for the input image and performs gradation conversion using this estimated plane as the reference plane will be described. In this method, for example, when the surface of the work is inclined, the inclination component can be canceled and the gradation conversion can be performed, so that there is an advantage that it can be utilized in the same manner as the above-mentioned three-point designation of static conversion. Hereinafter, a specific setting method of the plane reference will be described. In the plane reference as well, as in the above-mentioned average height reference, the reference plane estimation area for determining the reference plane is set in advance prior to the operation (step S83 in FIG. 8 described above). Hereinafter, in step S83 of FIG. 8, an example of the procedure for designating the reference plane estimation area will be described based on the GUI of FIGS. 62, 88, 92 to 95. First, select the "height extraction" button 116 on the GUI screen of FIG. 62, proceed to the height extraction selection screen 140 of FIG. 80, and select the "real-time extraction" corresponding to the dynamic conversion by the extraction method selection means 142. Then, the height dynamic extraction setting screen 190 of FIG. 88 is displayed. Next, in the "calculation method" selection field 192, when "plane reference" is selected as the reference for dynamic conversion as shown in FIG. 89, the screen shifts to the plane reference setting screen of FIG. 92.

On the plane reference setting screen, as in the height extraction means in the height extraction selection screen 140 of FIG. 80 described above, the separately set inspection target area is used as it is, or an arbitrary reference plane estimation area is specified as needed. .. Any method can be used for designating the reference plane estimation area, such as a rectangular shape, four corners, a circle by designating the center and radius, and a free curve. Further, only one point of the work can be specified, or conversely, the entire work or the entire image displayed in the second image display area 121 can be used as the reference plane estimation area.

Further, it is also possible to specify a mask area for which the estimated surface is not estimated for the reference surface estimation area, which is the same as in FIG. 90 and the like described above. Further, it is the same as the above that the gain can be adjusted, the extraction height can be specified, noise can be removed, and the like, if necessary. For example, on the screen of FIG. 92, when the "detailed setting" button 222 provided in the operation area 122 is pressed, the plane reference detailed setting screen 240 of FIG. 94 is displayed, and the emphasis method setting field 154 is added to the gain adjustment field 156. The "extraction height" setting field 162 for specifying the extraction height, the noise removal setting field 164 for noise removal, and the invalid pixel specification field 166 for specifying invalid pixels are displayed, and these detailed settings are possible. Become. Further, in the invalid pixel designation field 166, as shown in FIG. 95, the invalid pixel for which the distance could not be obtained can be filled with the specified specified value, the background pixel value, or an arbitrary value specified by the user. can.

When the reference plane estimation area is defined in this way, the setting screen is terminated. At the time of gradation conversion, a planar estimated surface is calculated from the height information included in this reference surface estimation area. For fitting the height information distributed in the reference plane estimation region, a known method such as the least squares method can be appropriately used. As described above, it is not always necessary to use the height information of all the points included in the reference plane estimation region for the calculation of the estimation plane, and the processing can be simplified by appropriately thinning out or averaging the height information. Is.

When the estimated surface is determined in this way, gradation conversion is performed with reference to this estimated surface. For example, gradation conversion is performed so that the estimated surface becomes the center value of the distance range. It is also possible to display the calculated information on the estimated surface. For example, in the example shown in FIG. 92, the X-direction inclination, the Y-direction inclination, and the Z-direction height of the estimated surface are displayed in the estimated surface display column provided in the operation area 122. In this example, the estimated plane is a single plane, but it can also be an estimated plane that is a combination of a plurality of planes. Further, although the estimated surface is a plane in this example, it is also possible to calculate the estimated surface as a simple curved surface such as a spherical surface.

Further, as an example of the work in which the method of designating the reference plane with a plane reference is effective, the above-mentioned FIGS. 86A and 87A can be mentioned. By setting a plane reference for such a work, even if the upper surface of the work has an inclination, the inclination is corrected and the reference surface is detected. Therefore, as with the above-mentioned three-point designation, an accurate BGA Pattern detection is realized.
(B3: Free-form surface reference)

Finally, a free curved surface reference that generates a free curved surface excluding high frequency components from a predetermined region (free curved surface target region) of an input image and performs gradation conversion using this as a reference plane will be described. For example, when the workpiece has a curved surface and it is difficult to approximate with a simple plane, it is difficult to accurately extract the height information of the area to be inspected. Therefore, by generating a simplified image by removing high-frequency components from the input image and using the surface shape (free curved surface) of this image as the reference surface, the rough shape and gentle changes are ignored, and a sudden change is made. It is possible to inspect only the part where the change is occurring, that is, the fine shape.

Hereinafter, a specific setting method of the free curved surface reference will be described based on the GUI of FIGS. 62 to 96. In the free-form surface reference as well, as in the above-mentioned average height reference and the like, the necessary conditions for determining the reference plane are set in advance prior to the operation (step S83 in FIG. 8 described above). First, select the "height extraction" button 116 on the GUI screen of FIG. 62, proceed to the height extraction selection screen 140 of FIG. 80, and select the "real-time extraction" corresponding to the dynamic conversion by the extraction method selection means 142. Then, the height dynamic extraction setting screen 190 of FIG. 88 is displayed. Next, in the "calculation method" selection field 192, "free curved surface reference" is selected as the reference for dynamic conversion as shown in FIG. 89. As a result, the screen shifts to the free curved surface reference setting screen 250 of FIG. 96.

Although it is possible to specify an arbitrary area as the free curved surface target area on the free curved surface reference setting screen 250, it is preferable that the entire image displayed in the second image display area 121 or a separately designated inspection target is specified. The area is used as it is as a free-form surface target area. Free-form surface A free-form surface is generated by removing harmonic components from the area designated as the target area. Then, a gradation conversion image that has undergone gradation conversion with the free curved surface as a reference plane is superimposed and displayed on the free curved surface target area displayed on the second image display area 121. Further, in the gradation conversion, gain adjustment, extraction height designation, noise removal, and the like can be performed as necessary, as in FIG. 90 and the like described above.
(Extraction size adjustment means)

Furthermore, it also has an extraction size adjustment function that adjusts the fineness (extraction size) of the extraction surface extracted based on the free curved surface. Specifically, in FIG. 96, an “extraction size” designation column 252 is provided as an extraction size adjusting means in the “detailed setting of extraction surface” column of the operation column. When the numerical value in the "extraction size" designation field 252 is increased or decreased, the curvature of the free-form surface changes accordingly, and the size of the defect that can be extracted changes. Here, the free-form surface image is generated and displayed in the second image display area 121 so that the unevenness of the set size or less is extracted. When the extraction size is increased, a smooth free-form surface is generated, and defects of a size corresponding to the set extraction size can be extracted. On the other hand, when the extraction size is reduced, a free curved surface along the surface shape of the work is generated, and only small defects corresponding to the set extraction size are extracted. For example, when the extraction size is increased, the free curved surface becomes smoother with respect to the surface shape of the work as shown in FIG. 97, so that the unevenness extracted by the smoothed reference surface becomes clear. On the contrary, when the numerical value is reduced, the free curved surface approaches the detailed shape along the unevenness of the surface shape of the work as shown in FIG. 90, and as a result, the unevenness extracted by such a complicated reference surface becomes unclear. Become. Further, when the numerical value in the "extraction size" designation field 252 is increased or decreased, the gradation conversion image in the free curved surface target area displayed in the second image display area 121 is also internally generated as a reference plane. The state of the free-form surface changes, and the size of the object extracted and displayed as the difference from the reference plane changes in real time. The user can optimally adjust the numerical value in the “extraction size” designation field 252 while referring to the second image display area 121.

When the free curved surface target area is defined in this way, the setting screen ends. At the time of gradation conversion, the free curved surface is calculated from the height information of the image included in the free curved surface target area. For fitting the height information distributed in the free-form surface target area, as shown in the flowchart of FIG. 161, image reduction processing (step S1611), filter processing (step S1612), and image enlargement according to the set extraction size are performed. A method of performing the process (step S1613) to generate a free-form surface image can be used. For the filtering process, for example, a median filter can be used. When the free curved surface is determined in this way, the unevenness to be extracted can be clarified by taking a difference from the original image (step S1614). Alternatively, as described above, a known method such as the least squares method can be appropriately used. As described above, it is not always necessary to use the height information of all the points included in the free curved surface target area for the calculation of the estimated surface, and the processing can be simplified by appropriately thinning or averaging. When the free curved surface is determined in this way, gradation conversion is performed with reference to this free curved surface. For example, gradation conversion is performed so that the free curved surface becomes the center value of the distance range.

Here, an example of a work in which the method of designating the reference plane as a free curved surface is effective will be described with reference to FIGS. 98A to 98B. FIG. 98A shows an inspection process for detecting defects such as protrusions and dents contained in the curved surface of the work. Here, a region including a defect is specified from the curved surface as the measurement region ROI. Then, a free curved surface is obtained from the height information included in the designated measurement area ROI, and gradation conversion is performed using the obtained free curved surface as a reference plane. The obtained low gradation distance image is shown in FIG. 98B. As shown in this figure, based on the free curved surface, the part higher than this surface is regarded as a protrusion (displayed as a white dot in FIG. 98B), and the portion lower than this surface is regarded as a depression (displayed as a black dot in FIG. 98B). Can be detected. In this way, the designation of the reference plane by the free curved surface can be effectively used for the workpiece having a curved surface shape. The processing load for detecting a free-form surface is higher than that of the one-point designation and the three-point designation described above.
(Defect extraction process)

In the above-mentioned free-form surface reference, when compression is performed in the vertical and horizontal directions of the distance image, that is, in the XY directions during the image reduction process shown in step S1611 of FIG. 161, the curved surface and the defect are separated depending on the state of the curved surface of the inspection object. It may not be possible to extract the scratches as desired. For example, consider an example in which an extraction target 1 and an extraction target 2 are to be extracted in a distance image obtained by capturing a tubular inspection object as shown in FIG. 162. When the free curved surface reference is selected and the image is compressed in the XY direction for such a distance image, as shown in FIG. 163, the peaks and valleys existing along the tubular length direction are also extracted. On the other hand, the extraction target 2 that is originally desired to be extracted is in a state of not being extracted neatly.
(Extraction direction designation means)

Therefore, in order to realize accurate defect detection even in such a case, an extraction direction specification function for extracting defects, that is, local irregularities, is provided by excluding shapes that continuously exist in a specific direction on the inspection target surface. It can also be provided. For example, an extraction direction specifying means capable of specifying an extraction direction when specifying inspection conditions is provided. Specifically, when "free curved surface reference" is selected in the "calculation method" selection field 192 of the GUI shown in FIG. 89 or the like, the GUI of the free curved surface reference setting screen 630 of FIG. 164 is displayed instead of the above-mentioned FIG. 96 or the like. Let me. In the free-form surface reference setting screen 630 of FIG. 164, an “extraction direction” designation field 632 is added as a form of the extraction direction designation means below the “extraction size” designation field 252. In the "extraction direction" designation field 632, a direction for extracting a local shape change such as a defect to be extracted is specified. In this example, as a preset option, any one of "X", "Y", and "XY" can be selected by a drop-down list or the like. Further, when the "detailed setting" button 222 is pressed on the screen of FIG. 164, the screen is similarly switched to the detailed setting screen 640 shown in FIG. The noise removal setting field 164, the invalid pixel designation field 166, and the like are displayed, and these detailed settings can be made.

For example, when "Y" is selected in the "extraction direction" designation field 632 on the screens of FIGS. 164 and 165, local shape changes in the Y direction of the image (vertical direction in FIG. 162) are extracted. In other words, a shape that does not change in the Y direction, for example, a groove extending in the vertical direction, is ignored, and a detection result as shown in FIG. 166 is obtained. Similarly, when X is selected, the shape change of the image in the X direction (horizontal direction in FIG. 162) is extracted. When XY is selected, a local shape change in the XY direction is extracted as shown in FIG. 163. As a result, by designating the extraction direction according to the shape of the inspection object, unnecessary shapes can be canceled and the desired shape can be appropriately extracted. For example, when it is desired to extract the character string 1 and the character string 2 in the inspection object as shown in FIG. 167, if the extraction is performed in the XY direction, the unevenness existing in the longitudinal direction is also detected as shown in FIG. 168. By designating the Y direction as the extraction direction, as shown in FIG. 169, a uniform shape in the Y direction can be removed, and a desired character string can be accurately extracted.

Here, as a specific shape extraction method such as defect extraction processing, the extraction processing in the Y direction shown in FIGS. 166 and 169 will be described as an example. In the image reduction process described above (step S1611 in FIG. 161), the image is reduced only in the Y direction. As a result, the reference free-form surface image obtained by performing the filter processing (step S1612) and the image enlargement processing (step S1613) has fine irregularities in the Y direction removed and is smoothed only in the Y direction. An image will be generated. In other words, it is possible to obtain an image in which only the shape to be canceled that is uniform in the Y direction is left. Therefore, when the difference between the target original image and the free-form surface image is taken (step S1614), the shape of the cancellation target that is uniform in the Y direction is removed, and as a result, the shape is finely divided in the Y direction. Only the uneven shape change will be left. As a result, it is possible to extract only the shape change in the desired direction.

Such a process is particularly effective for inspecting an inspection object that is long in one direction. By specifying the extraction direction along the longitudinal direction of the inspection object, it is possible to cancel the continuous uniform shape along the longitudinal direction, and accurate defect extraction can be realized.

Further, in order to acquire height information of such a continuous inspection object in one direction (for example, a product having a uniform shape in one direction such as a tire), in addition to the structured lighting, the above-mentioned light is used. The inspection using the cutting method can be preferably applied. In particular, since the optical cutting method is a method of acquiring the shape (profile) of a line-shaped cut surface, it is possible to continuously acquire a profile of an inspection object that is long in one direction while changing the cutting position. A distance image can be obtained by synthesizing the obtained profiles. With this method, even if the object to be inspected moves, in other words, the object to be inspected can be inspected without having to stand still. It is possible to detect defective products without stopping the product.

In the above example, the example of applying the defect extraction process to the free curved surface reference has been described, but the present invention is not limited to the free curved surface reference, and can be applied to other reference planes. For example, when the average height standard is selected as the reference plane, the above-mentioned defect extraction process is applied to obtain the average height for each line in the specified direction, and the difference from the average height is calculated. Can be taken. Since the processing is simple according to this method, there is an advantage that high-speed processing is possible.
(Extraction area setting dialog 148)

Further, when performing height extraction, the target area (extraction area) for calculating the reference plane from the input image is the same area as the inspection target area (measurement area) for performing the inspection process, and is separate from the measurement area. It can also be set to. As an example, in FIG. 90, when the “extraction area” button 147 is pressed, an extraction area setting dialog 148 in which the extraction area can be set is displayed as shown in FIG. 99. The extraction area setting dialog 148 is provided with an extraction area selection field 149, and the user can select "same as the measurement area", "rectangle", "circle", "rotation rectangle", etc. from the extraction area selection field 149. When "Same as measurement area" is selected, the extraction area becomes the same as the measurement area as described above. On the other hand, by selecting a region other than "same as the measurement region", an region different from the measurement region can be set as the extraction region. For example, as shown in FIG. 100, when "rectangle" is selected in the extraction area selection field 149, a rectangular frame is displayed in the second image display area 121, and the user can specify a desired area by dragging the mouse or the like. Further, when the "edit" button 324 provided to the right of the extraction area selection field 149 of FIG. 100 is pressed, the extraction area edit dialog 326 is displayed as shown in FIG. 101, and the rectangular extraction area is numerically expressed in xy coordinates. Can be specified by. When the extraction area is adjusted in the extraction area editing dialog 326, the change is reflected in the second image display area 121.
(Mask area setting field 330)

Further, from the extraction area setting dialog 148, a mask area for designating an area not to be extracted can be set. In the example of FIG. 99, the mask area setting field 330 is provided below the extraction area selection field 149. In the mask area setting field 330, a plurality of mask areas can be set. Here, a maximum of four mask areas 0 to 3 can be specified, and each mask area can be set independently. For example, when "circular" is selected as the mask area 0 as shown in FIG. 102, the circular mask area 0 is displayed on the second image display area 121. Further, when the "edit" button 332 is pressed from this state, the mask area editing dialog 334 for defining the details of the circular mask area 0 is displayed as shown in FIG. 103. The user can define the circular mask area 0 with the xy coordinates of the center and the radius. The mask area is displayed in the second image display area 121 with a border having a color different from that of the extraction area, which makes it easy for the user to visually distinguish the extraction area and the mask area. In the example of FIG. 103, the extraction area is displayed in green and the mask area is displayed in yellow. However, not limited to this example, it goes without saying that it is possible to distinguish by different colors, line thickness, line type (solid line, broken line, etc.), highlight (blinking or emphasis), etc. ..

In this way, the extraction area can be set independently of the measurement area. In addition, the setting contents of the extraction area can be displayed as character information. For example, in the example of FIG. 90, "same as the measurement area" is displayed in text at the bottom of the "extraction area" button 147, indicating that the extraction area for calculating the plane reference is the same as the measurement area. In addition, "mask area: invalid" is displayed at the bottom, indicating that the mask area is not set in the extraction area. As a result, the user can confirm the outline of the "extraction area" as text information.
(Pre-processing setting)

As described above, when the height extraction setting is completed in the "area" processing unit, as shown in FIG. 104, in the third image display area, within the rectangle of the area set in the "area" processing unit, The low gradation distance image that has been gradation-converted based on the extracted height is displayed in an overlapping manner. Next, the preprocessing is set. The preprocessing is a common filter processing performed before generating the distance image as described above, and various filters can be selected here. Specifically, when the "preprocessing" button provided in the setting item button area 112 is selected from the screen of FIG. 104, the filter processing setting screen 340 of FIG. 105 is displayed, and the filter to be applied can be selected. Examples of the filter that can be selected here include an averaging filter, a median filter, and a Gaussian filter. Further, in addition to such a filter, for example, a binarization level can be set. For example, on the binarization level setting screen 350 of FIG. 106, the upper limit value, the lower limit value, and the number of times of binarization can be set. Further, a histogram showing the distribution of binarized pixels may be displayed, or a function of updating the histogram in conjunction with the input image may be provided. When the filter processing setting is completed in this way, as shown in FIG. 107, in the third image display area, the low gradation distance image binarized through the filtering process is displayed in an overlapping manner within the set area. ..
(Judgment setting)

Further, in the "area" processing unit, after the input image is gradation-converted into a low-gradation distance image according to the conditions such as the set area, height extraction, and pre-processing, the height is higher than that of the low-gradation distance image. It also stipulates the conditions for making judgments such as inspection and image inspection. For example, when the "judgment condition" button provided in the setting item button area 112 is pressed from the screen of FIG. 107, the judgment condition setting screen 360 of FIG. 108 is displayed, and the judgment condition is set. In this example, the pixels of the binarized low-gradation distance image are counted, and when the value is within a predetermined range, it is set to OK, and when it is not, it is set to NG. In the example of FIG. 108, since the determination condition is 0 to 30 and the current value is 166, it is determined to be NG, and as shown in FIG. 109, “determination result: NG” and red characters are displayed in the third image display area. Is displayed. As described above, when the determination result is NG, the user can easily recognize it at the time of operation by setting the character to a red character or the like in a conspicuous manner.
("Blob" processing unit 267)

Further, although the determination based on the height inspection using the height information has been described above, the present invention is not limited to this, and it is also possible to add a determination process based on the image inspection on the conventional luminance image. Such an imaging test is called a blob. For example, as shown in FIG. 110, a "blob" processing unit 267 that performs blobs (image inspection) is added below the "area" processing unit in the flow display area 261. In the "blob" processing unit 267 as well, in the same manner as described above, the target area is set and the preprocessing is set (FIG. 111), the detection condition is set (FIG. 112), and the determination condition is set for the determination. The result can be output (Fig. 113).
("Color inspection" processing unit 267B)

Further, when a color optical image can be input, such as when a color CCD camera is connected as an imaging means, it is possible to combine color inspection. For example, as shown in FIG. 114, in the flow display area 261, a “color inspection” processing unit 267B that performs a color inspection as a measurement process is added below the “blob” processing unit 267. In the "color inspection" processing unit 267B as well, the determination conditions are set by setting the target area (FIGS. 115 and 116) and making detailed settings such as density averaging (FIG. 117) in the same manner as described above. The determination result can be output (Fig. 118).

After making various settings in the setting mode as described above, the workpiece is actually imaged in the operation mode to acquire an input image, and the determination process is performed based on the results of the height inspection and the image inspection. In the above example, the determination result can be output even in the setting mode, so that the user can easily recognize the determination result as an image at the setting stage. However, it goes without saying that the determination result can be output only in the operation mode.
(Operation mode)

Next, processing during operation inside the head unit and the controller unit in the three-dimensional image processing apparatus shown in FIGS. 4A and 4B will be described with reference to the flowcharts of FIGS. 119 and 120. First, the flow of processing during operation at the head portion of the three-dimensional image processing apparatus according to the third embodiment shown in FIGS. 4A and 5 will be described with reference to the flowchart of FIG. 119.
(Flow of processing according to the third embodiment)

First, when a trigger is input from the outside (step S11901), one projection pattern is projected onto the work from the first projector 20A (step S11902), and an image is taken by the imaging means (step S11903). Next, it is determined whether or not imaging has been completed for all the light projection patterns (step S11904), and if not, the light projection patterns are switched (step S11905), and the process returns to step S11902 to repeat the process. Here, a total of 16 pattern projection images are captured, including eight pattern projection images with a projection pattern using the phase shift method and eight pattern projection images with a projection pattern using the spatial coding method. ..

On the other hand, when all the light projection patterns have been imaged in step S11904, the process branches to step S11906 and step S11907. First, the three-dimensional measurement calculation is executed in step S11906 to generate the distance image A.

On the other hand, in step S11907, an average image A'that is averaged from a plurality of pattern projection images (pattern projection image group) captured by the phase shift method is calculated.

The above steps S11902 to S11906 are three-dimensional measurement by pattern projection from the first projector 20A. Next, three-dimensional measurement is performed by pattern projection from the second projector 20B. Here, in step S11908 following step S11906, a light projection pattern is projected onto the work from the second projector 20B (step S11908), and an image is taken by the imaging means (step S11909), as in steps S11902 to S11905. It is determined whether or not the imaging is completed for all the projection patterns (step S11910). If it is not yet, the light projection pattern is switched (step S11911), the process returns to step S11902, and the process is repeated. On the other hand, when all the light projection patterns have been imaged, the process branches to step S11912 and step S11912. In step S11912, a three-dimensional measurement operation is executed to generate a distance image B. On the other hand, in step S11913, the average image B'of the pattern projection image group captured by the phase shift method is calculated. When the three-dimensional distance images A and B are generated in this way, in step S11914, the three-dimensional distance images A and B are combined to generate a distance image. Further, in step S11915, the average images A'and B'are used to generate a luminance image (average two-dimensional shading image) obtained by synthesizing them. In this way, in the three-dimensional image processing apparatus of FIG. 5, a distance image having the height information of the work is acquired. If the luminance image is not required, steps S11907, S11913, and S11915 can be omitted.
(Flow of processing according to the fourth embodiment)

The flow of processing of the head portion of the three-dimensional image processing apparatus according to the third embodiment shown in FIG. 4A has been described above. Next, the processing flow of the head portion of the three-dimensional image processing apparatus according to the fourth embodiment shown in FIG. 4B will be described with reference to the flowchart of FIG. 120. explain. First, when a trigger is input from the outside (step S12001), one light projection pattern from the light projecting means 20 is projected onto the work (step S12002), and the first imaging means 10A takes an image (step S12003). At the same time, the second imaging means 10B also performs imaging (step S12004). Then, it is determined whether or not the imaging is completed for all the light projection patterns (step S1205). If it has not been done yet, the light projection pattern is switched (step S12006), the process returns to step S12002, and the process is repeated. On the other hand, when all the light projection patterns have been imaged, the process branches to steps S12007 to S12010. In step S12007, a three-dimensional measurement calculation is executed to generate a distance image A. In step S12008, a three-dimensional measurement operation is executed to generate a distance image B. On the other hand, in step S12009, the average image A'of the pattern projection image group captured by the phase shift method is calculated, and in step S12010, the average image B'of the pattern projection image group captured by the phase shift method is calculated. Then, in step S12011, the three-dimensional distance images A and B obtained in steps S12007 and S12008 are combined to generate a distance image. Further, a luminance image obtained by synthesizing the average images A'and B'obtained in steps S12009 and 10 is generated. According to this method, two images can be captured at the same time, so that the imaging time can be shortened.
(Inspection target area setting means)

At the time of inspection during actual operation, it is necessary to specify in advance an area (inspection target area) to be inspected by the inspection execution means 50 for the work. Such setting of the inspection target area is performed by the inspection target area setting means at the setting stage. The inspection target area setting means can be provided on the controller unit 2 side as described above, or can be realized by a three-dimensional image processing program. Specifically, as described above, when the "area setting" button 115 corresponding to the inspection target area setting means of the three-dimensional image processing program shown in FIG. 62 is pressed, the screen transitions to the inspection target area setting screen 120 shown in FIG. 47. On the inspection target area setting screen 120, an area to be inspected can be specified.

When the inspection target area is designated in this way, the three-dimensional image processing device performs image processing on the inspection target area and further executes the inspection. That is, as shown in the flowchart of FIG. 121, the inspection target area is assigned to the distance image (step S12101), the gradation conversion parameter is set based on the distance image (step S12102), and the floor is changed according to the gradation conversion parameter. Key conversion is performed (step S12103), image processing is further performed on the gradation conversion image, and a predetermined inspection is performed (step S12104).

The gradation conversion process is performed on the inspection target area set by the inspection target area setting means described above. That is, in this example, the inspection target area setting means is shared with the gradation conversion target area designating means for designating the gradation conversion target area. However, the area used to determine the parameters of the gradation conversion process may be set independently of the inspection target area. For example, the gradation conversion parameter creation area is designated by using the inspection target area setting means or the gradation conversion parameter creation area designation means prepared separately from the inspection target area setting means.
(Operation flow of controller part during operation)

Next, a procedure for processing the distance image obtained on the head portion side on the controller portion side during operation will be described with reference to the flowchart of FIG. 122 and the GUI of FIGS. 123 to 127. Here, FIGS. 123 to 126 show the GUI of the three-dimensional image processing program. The GUI of the three-dimensional image processing program shown in FIG. 123 shows an initial screen 260, and a flow display area 261 is provided on the left side of the screen and a third image display area 262 is provided on the right side. In the flow display area 261, a flow diagram in which the contents of each process performed by the three-dimensional image processing device are connected in a processing unit shape is displayed. Here, as the processing units, the "imaging" processing unit 263, the "Shapetrax2" processing unit 264, the "position correction" processing unit 265, and the "height measurement" processing unit 266 are displayed in the flow display area 261. In addition, at the setting stage before operation, each processing unit can be selected and detailed settings can be made. Further, in the third image display area 262, a luminance image, a distance image, an inspection result, or the like is displayed according to the processing content. In the example of FIG. 123, a luminance image obtained by capturing an image of a work (IC in this example) is displayed, and a search target area SA described later is displayed in a green frame shape.
(Imaging step)

First, in step S12201 of FIG. 122, a distance image and a luminance image are acquired from the head unit 1. Here, the pattern projection image is captured on the head portion side to generate a distance image and a luminance image. In the GUI example of FIG. 123, the “imaging” processing unit 263 displayed in the flow display area 261 corresponds to this.

As for the image data, the distance image is first transmitted from the head unit to the controller unit, and then the luminance image is also transmitted to the controller unit. On the contrary, the luminance image may be transferred first and then the distance image may be transferred, or these transfers may be performed at the same time.
(Search step)

Further, in step S12202, the controller unit performs a pattern search on the input image input during operation. That is, the part to be inspected is specified so as to follow the movement of the work included in the captured luminance image. In the example of FIG. 123, the “Shapetrax 2” processing unit 264 displayed in the flow display area 261 corresponds to the pattern search. Here, the image search means 64 of FIG. 5 performs a pattern search on the luminance image (input image) obtained by capturing the workpiece, and performs positioning. Specifically, the position of the preset inspection target area in the input luminance image is specified. The search target area SA for performing the pattern search is set in advance. For example, in the example of FIG. 124, the search target area SA is designated as a rectangular shape on the luminance image displayed in the third image display area.
(Position correction step)

Next, in step S12203, the position of the inspection target area is corrected using the result of the pattern search. In the example of FIG. 123, the “position correction” processing unit 265 displayed in the flow display area 261 corresponds to this. The inspection target area is preset on the inspection target area setting screen. In the example of FIG. 125, a plurality of inspection target areas are set on the distance image displayed in the third image display area. Specifically, as shown in FIG. 126, a rectangular area is set for each pin of the IC which is a work. The position correction is performed by, for example, a method of calculating the amount of misalignment by a normalized correlation search, a method based on the result of a pattern search, or the like. In this way, the position correction corrects the position of the inspection target area of the inspection process executed in the next stage.
(Inspection processing step)

Finally, in step S12204, the inspection is performed using the distance image at the corrected position. In the example of FIG. 123, the “height measurement” processing unit 266 displayed in the flow display area 261 corresponds to this. Further, in the example of FIG. 127, height measurement is performed. That is, the height is measured and inspected in each of the inspection target areas at the corrected position. For example, it is determined whether the plane height is within a predetermined reference value, and the determination result is output.

In the above procedure, an example of performing an inspection based on a distance image after performing position correction using a luminance image has been described. However, the present invention is not limited to this, and an image for which position correction is performed and an image for which inspection processing is performed can be arbitrarily set. For example, contrary to the above, it is also possible to perform the inspection process using the luminance image after performing the position correction using the distance image. As an example, in an example where an accurate pattern search is difficult with a luminance image, such as when a white work is placed on a white background, a pattern search based on height information using a distance image is effective. Become. Further, the inspection process can be determined not only by the determination process based on the height information but also by the image processing result using the luminance image such as reading the character string printed on the work by OCR.
(Height information output format)

The measured three-dimensional height information is obtained as three-dimensional point cloud data having values of X, Y, and Z, respectively. In addition to the three-dimensional point cloud data, it is also possible to convert the actually obtained value into, for example, a Z image or a pitch Z image such as XY.
(1: Z image)

The Z image is height image data having only Z coordinates. For example, if the unevenness of the position of the work imaged by the imaging means is important and the X and Y coordinates do not have to be accurate, the X and Y coordinates are unnecessary, so the Z image, which is the data of only the Z coordinate, is displayed. It is enough to output. In this case, the amount of data to be transmitted is reduced, and the transmission time can be shortened. Further, since the data can be handled as an image like a normal two-dimensional image pickup means, image processing can be performed using an existing image processing device for a two-dimensional image.
(2: XY pitch Z image)

The XY equal pitch Z image is height image data in which the XY coordinates are equal pitch regardless of the height. Specifically, the Z coordinate at the position where the XY coordinates are set to the same pitch is interpolated from the surrounding point cloud data to obtain the XY equal pitch Z image.

Generally, when the lens of the imaging means is not an objective telecentric lens, the position (X, Y coordinates) imaged by the imaging means differs depending on the height position (Z coordinate) of the imaged work. Therefore, even if the objects are captured at the same position on the image sensor, the actual XY coordinate positions will differ depending on the height. For example, when it is desired to inspect a three-dimensional difference such as volume, it is inconvenient because the value becomes large when the work is close to the camera and the value becomes small when the work is far away. Therefore, by obtaining the Z data having an XY uniform pitch from the point cloud data, it is possible to obtain a Z image having an XY position that is not affected by the height.
(3: XYZ (point cloud data))

Alternatively, the point cloud data can be output as it is as three-dimensional information. For example, it is used when you want to handle the measured three-dimensional data as it is. In this case, the amount of data is tripled as compared with the case of only the Z coordinate, but since it is raw data, it can be used for applications such as obtaining a three-dimensional difference from three-dimensional CAD data.
(Generation of equi-pitch image)

Next, FIG. 128 shows a data flow diagram for creating an equal pitch image in which the angle of view is corrected in addition to the distance image and the luminance image. As shown in this figure, first, a spatial code image is generated from the spatial coding pattern floodlight work image group imaged by the head portion according to the spatial coding method. On the other hand, a phase image is generated from the phase shift pattern floodlight work image group according to the phase calculation. Then, a phase expansion calculation is performed from these spatial code images and phase images to generate an extended phase image. It is also possible to apply common filter processing such as filtering the spatial coding pattern projection work image group and the phase shift pattern projection work image group. Examples of common filtering include the application of two-dimensional filters such as a median filter, a Gaussian filter, and an averaging filter. On the other hand, a luminance image (average shading image) is generated by averaging the phase shift pattern projection work image group.
(Equal interval processing)

As described above, after the head portion generates the three-dimensional data and the luminance image, a distance image such as a Z image or an equal pitch Z image is further created. First, the distance image is transferred from the head unit 1 to the controller unit 2, and gradation conversion is performed to convert the phase information into height information. Here, after obtaining the X image, the Y image, and the Z image from the phase information, the XY are equalized to obtain the XY equal pitch Z image and the XY equal pitch Z average image on the XY plane. Such uniform interval processing is performed by the interval equalization processing setting means 47.

In the example of FIG. 128, an example in which the phase shift method and the spatial coding method are combined is described in the generation of the distance image, but the distance image can be generated only by the phase shift method without using the spatial coding method. .. The spatial coding process can be switched ON / OFF by the spatial coding switching means 45 shown in FIG. An example of this is shown in the data flow diagram of FIG. 129. As shown in this figure, the number of images taken can be reduced and the distance image can be generated at high speed because the spatial coding method is not used.

On the other hand, FIG. 130 shows a data flow diagram when a Z image is obtained by turning off the pitching function such as XY. In this example, it is not necessary to decompose into an X image or a Y image at the time of gradation conversion, so that the processing can be simplified accordingly.

Further, FIG. 131 shows an example of outputting point cloud data that outputs XYZ coordinate information as it is. In this example, the X image, the Y image, and the Z image after the phase-to-height conversion can be output as they are, so that there is an advantage that the X image, the Y image, and the Z image can be output with a light load.
(Gradation conversion method)

Next, a procedure will be described in which the gradation conversion means 46 of the three-dimensional image processing device automatically converts a high-gradation distance image into a low-gradation low-gradation distance image based on the distance image. .. Here, in an inspection device installed on a line on which a plurality of workpieces are conveyed, a plurality of applications such as a range image (input image) that is sequentially input and a gradation conversion to a low gradation distance image in real time are used. The procedure for performing gradation conversion on the input image of is described. The gradation conversion process in this case is roughly divided into (A) a method of determining the gradation conversion parameter in advance (static conversion) and (B) a method of determining the gradation conversion parameter according to the input image. There are two methods (dynamic method). These will be described below.
(A: Static conversion)

First, a static conversion in which gradation conversion parameters are determined in advance will be described. Here, the gradation conversion parameters for performing gradation conversion are adjusted for the input image and the registered image registered in advance at the time of setting. Then, at the time of operation, the gradation conversion of the distance image is performed with the gradation conversion parameter set at the time of setting, and the inspection is executed on the low gradation distance image after the gradation conversion. The procedure at the time of setting is as described based on the flowchart of FIG. 8 described above.
(Procedure during operation)

When the gradation conversion parameter is adjusted, gradation conversion is performed on the input image input at the time of operation with this gradation conversion parameter. Here, the procedure of static conversion at the time of operation will be described based on the flowchart of FIG. 132. First, in step S13201, a distance image is acquired. Here, the controller unit 2 captures the distance image generated by the distance image generation means 32. Next, in step S13202, gradation conversion processing of the input distance image is performed. Here, the gradation conversion process is executed according to the gradation conversion parameter adjusted at the time of setting, and a low gradation distance image in which the number of gradations of the distance image, that is, the dynamic range is reduced is generated. Finally, in step S13203, the inspection process is executed by the inspection execution means 50. According to this method, by setting the gradation conversion parameter in advance, it is not necessary to calculate the gradation conversion parameter at the time of operation, and the processing can be lightly loaded.
(B: Dynamic conversion)

Next, a dynamic conversion that calculates the gradation conversion parameter at the time of gradation conversion based on the input image will be described. First, the procedure at the time of setting is performed according to the flowchart of FIG. 8 as described above. Specifically, in step S81, the input image or the registered image is acquired, and then in step S82, the gradation conversion method is selected. Here, it is assumed that the user has selected dynamic conversion. Then, in step S83, the gradation conversion parameter is adjusted. Here, under what conditions the gradation conversion parameter is calculated or adjusted with respect to the input image input during operation is set based on the image acquired in step S81.

When the calculation conditions of the gradation conversion parameters are set in this way, the gradation conversion parameters corresponding to the input image are individually calculated according to the set gradation conversion parameter calculation conditions during operation. Next, an operation procedure for gradation-converting a distance image into a low-gradation distance image by dynamic conversion will be described with reference to the flowchart of FIG. 133. First, in step S13301, a distance image is acquired. Here, as in step S13201 described above, the controller unit 2 captures the distance image generated by the distance image generation means 32. Next, in step S13302, the gradation conversion parameter is determined based on the distance image which is the input image. As a method for adjusting the gradation conversion parameter, the above-mentioned method can be used. Further, in step S13303, gradation conversion is executed. Finally, the inspection process is executed in step S13304. According to this method, since the gradation conversion parameter can be changed according to the input image, it is possible to flexibly perform gradation conversion even for different workpieces and perform an accurate inspection. For example, the inspection of the work surface having variations in height can be performed without deteriorating the accuracy.

Here, as an example of dynamic conversion, a method of optimally creating a low gradation distance image after gradation conversion by adjusting gradation conversion parameters will be described with reference to FIG. 134. Here, there are two methods, one is to dynamically set one gradation conversion parameter set, and the other is to prepare a plurality of gradation conversion parameter sets in advance and dynamically select the gradation conversion parameter set. ..
(How to set 1A gradation conversion parameter set)

First, a method of dynamically setting one gradation conversion parameter set will be described. In this method, the value of the gradation conversion parameter used for gradation conversion is adjusted based on the image information of a predetermined area of a plurality of distance images as input images, and the adjusted gradation conversion parameter is adjusted. The gradation conversion means 46 executes the gradation conversion process of the distance image using the above. Here, an example of gradation-converting a 16-gradation distance image (image before gradation conversion) into an 8-gradation low-gradation distance image (image after gradation conversion) will be described. Further, as shown in the perspective view of FIG. 134, the work to be inspected has the entire measurement surface of each work displaced vertically, and the range thereof is 5 mm. The height range including the thickness and distortion of the entire measurement surface is 0.5 mm. Consider an example in which each work having a different surface height is inspected by image processing for scratches on its surface. By using the distance image and converting the distance image into a two-dimensional grayscale image (low gradation distance image), it is possible to inspect with an existing image processing device for a two-dimensional image.

First, the area to be inspected on the work is designated in advance as the inspection target area by the inspection target area setting means. In addition to designating a part of the distance image which is the input image, the inspection target area can be the entire distance image. In this case, the work of specifying the inspection target area may be omitted.

First, the average distance of the inspection target area is calculated. Next, the difference (distance range) between the maximum distance and the minimum distance of the inspection target area is obtained. Further, a numerical value obtained by multiplying the distance range by 1.2 is used as the distance range of the converted image. For example, in the work distribution of FIG. 134, assuming that the distance range is 0.5 mm, 0.6 mm obtained by multiplying this by 1.2 is set as the distance range of the image after gradation conversion. Therefore, the measurement range is within ± 0.3 mm with respect to the average distance.

Furthermore, the span for the distance image as the input image is obtained so that the range of 0.6 mm (± 0.3 mm centered on the average distance), which is the distance range of the image after gradation conversion, becomes 256 gradations. The span may be a predetermined constant. Further, with respect to the distance image, the one having an average distance of −0.3 mm or less can be set to 0, while the one having an average distance of +0.3 mm or more can be set to 255.

In this way, a range required for inspection is determined from the plurality of input distance images, and the height information of this range is appropriately maintained even in the low gradation distance image after gradation conversion. Gradation conversion parameters can be set. As a result, it becomes possible to appropriately inspect the presence / absence and position of scratches on each work dispersed in the height direction using an existing image processing device.
(1B Method of preparing multiple gradation conversion parameter sets in advance)

In the above embodiment, a method of appropriately obtaining the gradation conversion parameter set so as not to lose the height information required for scratch detection by using the height information of the acquired distance image has been described. On the other hand, a method of preparing a plurality of gradation conversion parameter sets in advance will also be described below. In this method, for a plurality of distance images that are input images, a gradation-converted image that has been gradation-converted with a plurality of preset gradation conversion parameters is created, and a predetermined inspection of the distance image is performed. The gradation-converted image to be used for inspection is selected based on the image information of the target area.

Such a specific example will be described with reference to the schematic diagram of FIG. 135 and the flowchart of FIG. 136. In this example as well, as in FIG. 134 described above, consider a case where the entire measurement surface for each work is vertically displaced within a range of 5 mm. For such a work, nine images after gradation conversion, in which the conversion center is changed every 0.5 mm and gradation conversion is performed, are created. Then, the obtained images after gradation conversion are preferably displayed side by side on the display means, and the user is made to select the converted image by the conversion center closest to the average value of the distance image which is the input image. Then, the gradation conversion parameter set applied to the image selected by the user is set. As a specific procedure, as shown in FIG. 136, first, a distance image of the work is generated (step S13601), and then the gradation conversion process is performed a plurality of times while changing the gradation conversion parameters as described above (step S13602). .. Then, the user is made to select a desired image from the obtained gradation conversion image, which is a simple low gradation distance image (step S13603), and the gradation conversion parameter is adjusted as necessary, and the obtained low gradation is obtained. An inspection is performed on the gradation distance image (step S13604).
(Details of static conversion and dynamic conversion)

Next, the details of these static conversions and dynamic conversions will be described. First, static conversion will be described. Static conversion is a specific method for correcting the standard of height information that should be left when converting a distance image to a low gradation distance image.
(A1) One-point designation to correct at the specified height (distance) and
(A2) Three-point designation that corrects on a plane can be used.
(A1: One point designation)

The one-point designation is a method of gradation-converting a distance image into a low-gradation distance image based on the height (distance) of a point or region designated by the user. The reference height is, for example, the height in the middle of the height range (distance range) that is gradation-converted into a low-gradation distance image in the height information of the distance image. Alternatively, it can be set to the upper limit (highest position where gradation conversion is performed) or the lower limit (lowest position where gradation conversion is performed) of the distance range. The specific procedure for designating one point is as described with reference to FIGS. 66 to 78 above.
(A2: Designate three points)

The three-point designation is a method of gradation-converting a distance image into a low-gradation distance image using a plane obtained from the three points designated by the user as a reference plane. The reference plane is also set to a height in the middle of the height range (distance range) to be gradation-converted into a low-gradation distance image, for example, in the height information of the distance image, as in the case of the reference height specified by one point described above. Alternatively, it can be set to the upper limit (highest position where gradation conversion is performed) or the lower limit (lowest position where gradation conversion is performed) of the distance range. Specific examples of the three-point designation are as described based on the GUI screens of FIGS. 81 to 85 above.

Here, an example in which the designation of the reference plane by static conversion is effective will be described with reference to FIG. 137. In this example, the workpieces WK7 having the same height are conveyed on the same surface by the belt conveyor BC. In this case, since the height of the work WK7 hardly fluctuates, dynamic conversion that changes the reference plane according to the work is unnecessary, and static conversion can be used. In particular, static conversion is faster than dynamic conversion, and you can enjoy the advantages of static conversion.

Further, FIG. 138A is an example of a work in which height fluctuation itself is desired to be suppressed. In this example, the inspection process for detecting the case where the cap of the bottle, which is the work WK10 conveyed by the belt conveyor BC, has "floating" as an abnormality is targeted. In this case, by adopting the static conversion, it is possible to detect the fluctuation of the height as in the low gradation distance image shown in FIG. 138B. On the contrary, if the dynamic conversion is adopted, the fluctuation amount is corrected, and as a result, the abnormality cannot be detected. Therefore, static conversion can be preferably used in such applications (B: specific example of dynamic conversion).

The above has described static conversion in which gradation conversion conditions are specified in advance at the setting stage and gradation conversion is performed under the specified conditions during operation. Next, a specific example of dynamic conversion that adjusts the gradation conversion condition according to the input image to be inspected will be described. First, in the dynamic conversion, as a specific method for correcting the reference of the height information to be left when the distance image is gradation-converted to the low-gradation distance image,
(B1) An average height standard for gradation conversion using the average height (average distance) in the average extraction area specified for the input image as the average reference height.
(B2) A plane reference that generates an estimated plane in a designated area of an input image and converts gradation using this as a reference plane.
(B3) A free curved surface reference is included in which a free curved surface excluding a high frequency component is generated from an input image and gradation conversion is performed using this as a reference plane.

The average extraction area for defining the average reference height is set in advance prior to operation (step S83 in FIG. 8 described above). An example of the procedure for designating the average extraction area in step S83 of FIG. 8 is as described based on the GUI of FIGS. 88 to 92. On the other hand, during operation, dynamic conversion is performed according to the procedure described with reference to FIG. 133. For example, a work transported on the line is imaged to generate a distance image (step S13301), the average height of the average extraction region set above is calculated (step S13302), and gradation conversion is performed based on this. This is executed to generate a low gradation distance image (step S13303), and the obtained low gradation distance image is inspected (step S13304). With this method, even if there are variations in the height direction of the workpiece, the reference plane for gradation conversion can be reset each time for each workpiece, so accurate inspection can be achieved regardless of the variations in the height direction of the workpiece. can.

Next, also in the plane reference, the reference plane estimation area for determining the reference plane is set in advance prior to the operation (step S83 in FIG. 8 described above), as in the above-mentioned average height reference. An example of the procedure for designating the reference plane estimation area in step S83 of FIG. 8 is as described based on the GUI of FIGS. 88 and 92 to 95. During operation, dynamic conversion is performed according to the procedure described in FIG. 133. For example, a work transported on the line is imaged to generate a distance image (step S13301), the reference plane estimation region set above is extracted, the estimated plane is calculated (step S13302), and the obtained estimation is performed. A gradation conversion is performed with the surface as a reference to generate a low gradation distance image (step S13303), and the obtained low gradation distance image is inspected (step S13304). With this method, even if the surface of the work has an inclination or the like, this can be canceled and an accurate inspection can be realized regardless of the inclination of the work.

Finally, the specific setting method of the free curved surface reference is as described based on the GUI of FIGS. 126 to 32. In the free-form surface reference as well, as in the above-mentioned average height reference and the like, the necessary conditions for determining the reference plane are set in advance prior to the operation (step S83 in FIG. 8 described above). In operation, dynamic conversion is performed according to the procedure described in FIG. 133. For example, a work transported on the line is imaged to generate a distance image (step S13301), the free curved surface is calculated for the free curved surface target area set above (step S13302), and the obtained free curved surface is calculated. To generate a low gradation distance image by executing gradation conversion with reference to (step S13303), and inspect the obtained low gradation distance image (step S13304). With this method, it is possible to obtain an advantage that even a work such as a surface inspection of a curved work surface, which is difficult to perform an accurate inspection by a conventional method, can be performed with high accuracy.

Here, an example in which the designation of the reference plane by dynamic conversion is effective will be described with reference to FIG. 139. In this example, the workpieces WK11 having different heights are conveyed on the same surface by the belt conveyor BC. In this case, since the height of the work WK11 is different for each individual, the height of the work is changed by changing the reference plane according to the height of each individual and converting the gradation by using the average height reference or the like. Even if there is, optimum gradation conversion is possible.

Further, FIG. 140 shows an example of the work WK12 having a minute inclined surface and a dent on the plane. In this example, since a minute inclined surface exists on the surface of the work WK12 due to the dent DE, the inclination may affect the detection accuracy of the dent. Therefore, by using a plane reference or the like to dynamically obtain a plane for each workpiece and perform gradation conversion as a reference plane, highly accurate inspection such as detection of slight dents and dents becomes possible.

Further, FIG. 141 shows an example of a curved work WK13 having a different radius. Also in this example, by using a free-form surface reference or the like to obtain a curved surface for each work and performing gradation conversion using this as a reference surface, it is possible to perform an inspection in which the influence of shape variation for each individual is reduced.
(Automatic adjustment of gradation conversion parameters)

The procedure for manually setting the gradation conversion parameter based on the image after gradation conversion has been described above. On the other hand, when the gradation conversion parameter is manually set and the gradation conversion is performed with the gradation conversion parameter set first, an image suitable for inspection may not be obtained due to changes in the work or environment. In such a case, the user does not refer to the image for fine adjustment, but corrects the gradation conversion parameter using the data after gradation conversion and performs the conversion again to perform image conversion suitable for inspection. You can also do it. In this method, after the initial setting of the gradation conversion is performed, the gradation conversion is first performed with an arbitrary gradation conversion parameter as the initial value, and then the gradation conversion parameter is adjusted.

For example, the gradation conversion condition automatic setting means can be made to function as a gradation conversion condition automatic setting means and a gradation conversion condition manual setting means. That is, the gradation conversion condition automatic setting means sets simple gradation conversion conditions when the gradation conversion means converts a distance image into a low gradation distance image. Further, based on the simple gradation conversion condition set by the gradation conversion condition automatic setting means, the gradation conversion condition manual setting means can be used while the gradation-converted simple low-gradation distance image is displayed on the display means. , Accepts manual adjustment of gradation conversion conditions. As a result, instead of prompting the user to set the gradation conversion condition suddenly, one or more simple gradations obtained after performing the gradation conversion by automatically setting the provisional simple gradation conversion condition are performed. Since the desired gradation conversion condition can be set while referring to the low gradation distance image, even if the user is not familiar with the meaning of the gradation conversion parameter or is unfamiliar with the setting, it is automated to some extent and such. It is possible to facilitate the setting work.

A specific procedure will be described with reference to the flowchart of FIG. 142. First, in step S14201, the process of creating a distance image is executed. Next, in step S14202, the initial gradation conversion process is performed using the initial values of the gradation conversion parameters. Further, in step S14203, it is determined whether or not the obtained gradation conversion image is appropriate, and if it is not appropriate, the gradation conversion parameters are adjusted again in step S14204, and then the process returns to step S14202 to repeat the process. On the other hand, if it is determined in step S14203 that an appropriate gradation conversion image is obtained, the process proceeds to step S14205 to execute a predetermined inspection.

In the above method, it is determined in step S14203 whether or not the gradation conversion parameter is appropriate, but this procedure may be omitted. The procedure in this case is shown in FIG. 143. Each procedure is substantially the same as the example of FIG. 142 described above. In step S14301, the distance image creation process is executed, and then in step S14320, the initial gradation conversion process is performed. Then, after adjusting the gradation conversion parameter in step S14303, a predetermined inspection is performed in step S14304.

As an initial gradation conversion method, for example, a method of applying a conversion function f (x, y, z) to an input image, a method of shifting or spanning an input image and compressing the result to n gradations, or an arbitrary method. Using the plane as the reference plane, take the difference between the input image and the reference plane, apply shift and span, and compress the result to n gradations, or take the difference between the input image and the reference image and shift and span. Examples thereof include a method of multiplying and compressing the result to n gradations.

Next, a specific example of automatically adjusting the gradation conversion parameter after gradation conversion will be described. The automatic adjustment method of the gradation conversion parameter includes (C1) a method of using the median value of the histogram of the distance image data after conversion, (C2) a method of using the maximum and minimum values of the histogram, and (C3) a method of using C1 and C2. Combinations, etc. are possible.
(C1: Method using the median histogram)

First, a method using the median value of the histogram will be described based on the flowchart of FIG. 144. First, in step S14401, a histogram of the converted distance image data is calculated. Next, in step S14402, the median value of the histogram is obtained. Then, in step S14403, the gradation conversion parameter is changed so that the median value becomes a predetermined value. The gradation conversion process is executed again using the gradation conversion parameter changed here. The median value may be the average value of 2n + 1 including n pieces before and after the center. Further, the maximum value and the minimum value may be the median values after removing n from the larger one and m from the smaller one, respectively.
(C2: Method using the maximum and minimum values of the histogram)

First, the histogram of the converted distance image data is calculated. Next, the maximum and minimum values of the histogram are obtained, and the gradation conversion parameters are changed so that the width between the maximum value and the minimum value becomes a predetermined value. Then, the gradation conversion process is executed again with the changed gradation conversion parameters. Here, the maximum value and the minimum value may be average values of n from the larger one and m from the smaller one, respectively. Alternatively, the maximum value and the minimum value can be the maximum value and the minimum value after removing n from the larger one and m from the smaller one, respectively.
(C3: Combination of C1 and C2)

The above C1 and C2 may be combined. That is, after the calculation of the histogram, the gradation conversion parameter is changed based on the width of the median value and the maximum value-minimum value, and the gradation conversion is performed again.

In the above method, a low-pass filter can be applied to the histogram in advance. Further, the histogram may be obtained after applying a low-pass filter to the converted distance image in advance.

By such gradation conversion, a distance image having a high number of gradations including height information can be converted into a low gradation distance image with low gradation. Since this low-gradation distance image can be processed as a two-dimensional image, it is possible to handle the low-gradation distance image even with an image processing device compatible with the existing two-dimensional image. For example, the height information of each pixel included in the distance image obtained by photographing the work in order to inspect the presence or absence of scratches is expressed as a hexadecimal binary number as a shading value. Here, when the difference between the distance image and the reference distance image is calculated, the defect information of shallow and small scratches appearing on the surface of the work is aggregated in the lower eight gradations. Therefore, for example, by reducing the upper eight gradations by the gradation conversion means, it is possible to greatly reduce the amount of information of the difference image while preventing the detection accuracy from being lowered. In this way, the gradation conversion means reduces the gradation of the upper half of the gradation when the gradation value of each pixel of the difference image is expressed in gradation, thereby preventing the detection accuracy from being lowered and the difference image. The amount of information can be greatly compressed.
(Partial execution of gradation conversion processing)

Further, the gradation conversion process is not performed on all the distance images, but can be performed only on a part of the distance image. Specifically, the gradation conversion means executes the gradation conversion process only for the designated inspection target area in the distance image. As a result, the gradation conversion process is reduced, the processing load is reduced, and the speed is increased. As an example, a height inspection process (first inspection process) and an image inspection process (second inspection process) are combined using a work WK14 having a shape in which cylinders having different diameters are triple-layered as shown in FIG. 145A. Consider the case where multiple inspection processes are performed. Here, as a height inspection process, the height of each surface of the columnar surface of the work WK14 is measured, and as an image inspection process, chipping or cracking is detected in the columnar portion from the top to the second stage of the work WK14. Perform a visual inspection. The inspection process is selected by the inspection process selection means. Further, specific settings for the inspection process selected by the inspection process selection means are performed by the inspection process setting means.
(Inspection process selection means)

The inspection process selection means is a means for selecting a plurality of inspection processes executed by the inspection execution means for the distance image. Here, as shown in FIGS. 44, 56, etc., the inspection process is selected when the processing unit is added from the initial screen 260. Specifically, the desired inspection process is selected from the list of inspection processes displayed by selecting the "measurement" process from the "addition" submenu of the processing unit. In this example, "Area", "Pattern Search", "Shapetrax2", "Edge Position", "Edge Width", "Edge Pitch", "Edge Angle", "Pair Edge", "Scratch", "Blob", ""Shadeblob","Trend edge position", "Trend edge width", "Trend edge defect", "Shade inspection", "Color inspection", "OCR", "2D code reader", "1D code reader", "High" Select the desired inspection process from "Measurement".
(Inspection processing setting means)

On the other hand, the inspection process setting means sets the details of each inspection process selected by the inspection process selection means. Here, as shown in FIGS. 46, 63, 78, and the like, each setting item is individually set from each button arranged in the setting item button area 112. In this example, specific setting items are arranged as setting item buttons, for example, "image registration" button 113, "image setting" button 114, "area setting" button 115, "height extraction" button 116, " Includes a "pre-processing" button 117, a "detection condition" button 118, a "detailed setting" button 119, a "judgment condition" button, a "display setting" button, a "save" button, and the like. In this way, the setting item button area 112 functions as an inspection process setting means.
(Height inspection process)

An example of the obtained distance image for the work as shown in FIG. 145A is shown in FIG. 145B. This distance image is represented by 16 gradations before gradation conversion. When height measurement (height inspection processing) is performed on such a distance image, high accuracy is achieved by using height information with 16 gradations of high accuracy without performing gradation conversion. Inspection can be realized. Specifically, as shown in FIG. 145C, in order to measure the height of each surface of the work, the inspection target area is set on each of the three surfaces of the work, and the height of each inspection target area is set to the 16th floor. Measure as it is.
(Image inspection processing)

On the other hand, when measurement (image inspection processing) that does not use height information is performed in the inspection processing, high-gradation information is unnecessary, and processing with a lower-gradation distance image is less burdensome. Therefore, gradation conversion is performed on a high-gradation distance image to obtain a low-gradation distance image, and then processing is performed. Here, it is not necessary to convert the entire distance image into a low-gradation distance image, and it is sufficient to perform the gradation conversion only on the target to be image-inspected. That is, the area for performing gradation conversion is an inspection target area set for image inspection processing that does not use height information, among the inspection target areas set for one or more. In the example shown in FIG. 145D, as in FIG. 145B, an inspection target area for image inspection processing is set for a high gradation distance image before gradation conversion. In this example, the inspection target area is set so as to surround the columnar column of the second stage in order to detect chipping or the like from the top of the work to the second stage. In other words, since the third column is not subject to visual inspection, this part is set to be excluded from the inspection target area. Then, gradation conversion is performed on the target area for image processing inspection. The low gradation distance image obtained as a result is shown in FIG. 145E. The low gradation distance image after gradation conversion shown in this figure is represented by eight gradations, and although some height information is lost as compared with the high gradation distance image, the presence or absence of chipping is detected. Since sufficient accuracy is maintained for visual inspection applications, there is no problem in image inspection processing. On the other hand, the area requiring gradation conversion is significantly reduced as compared with FIG. 145B and the like, which contributes to simplification and speeding up of processing.

In this way, the presence or absence of gradation conversion is selected according to the inspection process. For example, among the inspection processes displayed in the flow display area 261 of FIG. 110, in the "height measurement" processing unit and the second "height measurement" processing unit, the 16th floor of the distance image without gradation conversion. Highly accurate height measurement is realized by using the height information of the key. On the other hand, in other inspection processes, for example, a "numerical calculation" processing unit, an "area" processing unit, and a "blob" processing unit 267, a gradation-converted 8-gradation low-gradation grayscale image is used. Although not shown, when performing an inspection process on a luminance image (for example, a color inspection on a color luminance image), height information is not required in the first place, so naturally gradation conversion is also unnecessary.
(Display of gradation conversion condition setting means 43)

Further, for the image inspection process that requires such a gradation conversion process, the gradation conversion condition for performing an appropriate gradation conversion is set from the gradation conversion condition setting means 43. At this time, the gradation conversion condition setting means 43 can be set only when the gradation conversion processing is required, and conversely, in the inspection processing that does not require the gradation conversion processing, for example, the height inspection processing, the gradation conversion condition By making it impossible to set, the user can smoothly set only the necessary items without being confused by the extra setting work. Therefore, in the present embodiment, when setting the inspection process that does not require the height information of the image other than the height inspection process, the floor for setting the gradation conversion parameter for performing the gradation conversion of the distance image. While the key conversion condition setting means 43 is displayed, the gradation conversion condition setting means 43 is not displayed when setting the height inspection process.

A specific setting procedure will be described with reference to the flowchart of FIG. 146. First, in step S14601, the inspection process is selected. Here, the inspection process to be executed by the inspection execution means is selected in the "Measurement" menu which is the inspection process selection means. Next, in step S14602, it is determined whether or not the inspection process selected in step S14601 requires gradation conversion, and if the inspection process requires gradation conversion, the process proceeds to step S14603 to enable gradation conversion. At the same time, the gradation conversion setting means is displayed in the setting item.

For example, when performing image inspection processing on the work of FIG. 145A by the "area" processing unit, as shown in FIG. 147, the gradation conversion condition setting means 43 is provided in the setting item button area 112 which is the inspection processing setting means. The "height extraction" button 116 is displayed. When the "height extraction" button 116 is pressed, the height extraction setting screen is displayed as shown in FIG. 148. From this screen, the user can set the conditions required for the gradation conversion process.

In the example of FIG. 148, the gradation-converted low-gradation distance image is displayed on the image display area according to the gradation conversion conditions set in the operation area on the right side. As a result, the user can visually confirm whether or not the desired inspection result can be obtained under the current gradation conversion conditions, and has an advantage that the adjustment work of the gradation conversion conditions can be easily performed. In particular, in the example of FIG. 148, the inspection target area is set to be circular and arranged so as to surround the second columnar shape of the work. In addition, by setting the detection method to dynamic conversion (real-time extraction) and setting the plane reference as the calculation method, two surfaces, the top surface at the center of the work and the outer peripheral surface thereof, are detected, and these are used as the reference planes for the work. Detects scratches and dents on the surface of. Further, the parameter display of the reference plane detected according to the gradation conversion condition can be switched and displayed by operating the console.

Further, in the example of FIG. 148, the low gradation distance image obtained by gradation-converting the entire work displayed on the image display area is not displayed, but the gradation is converted only within the set inspection target area. The image of the low gradation distance image is displayed. In other words, the original distance image is displayed as it is for the image other than the inspection target area. As a result, it is possible to visually indicate to the user that the gradation conversion is not performed on the entire input image but is performed only within the inspection target area which is a part of the input image. When the setting of the gradation conversion condition is completed in this way, the process proceeds to step S14604.

On the other hand, in the case of the inspection process that does not require gradation conversion, the process jumps to step S14604 without going through step S14603. In this case, the gradation conversion condition setting means is not displayed on the inspection process setting screen. For example, when the height inspection process is performed by the "height measurement" processing unit, as shown in FIG. 46 and the like, the setting item button area 112, which is the inspection process setting means, is the gradation conversion condition setting means 43. The "Extract Height" button is not displayed. As a result, the user can recognize that it is not necessary to set conditions related to gradation conversion which is unnecessary for the height inspection process. Alternatively, by making it impossible to set the gradation conversion condition for this inspection process, it is possible to avoid confusion caused by unnecessary setting.

Then, in step S14604, the inspection process is set. Finally, in step S14605, it is determined whether or not all the inspection process settings have been completed. If not, the process returns to step S14601 and is repeated from the selection of the inspection process. If so, the process is completed.

On the other hand, the procedure during operation will be described with reference to the flowchart of FIG. 149. First, in step S14901, a distance image is input as an input image. Next, in step S14902, initialization is performed. Here, 1 is set to n. Further, in step S14903, the nth inspection process is executed. Further, in step S14904, it is determined whether or not n <N (N is the number of set inspection processes), and if YES, n is incremented by 1 in step S14905, and the process returns to step S14903 to perform the next inspection process. Repeat the process of executing. On the other hand, if n <N, it is assumed that all the inspection processes have been completed, and the processes are completed. In this way, all inspection processes are sequentially executed.

Here, the execution of the inspection process in step S14903 will be described in detail. First, when gradation conversion is required in the inspection process, as shown in the flowchart of FIG. On the other hand, gradation conversion is performed. Then, in step S1502, the inspection process is executed on the low gradation distance image after the gradation conversion. On the other hand, in the case of the inspection process that does not perform the gradation conversion, as shown in the flowchart of FIG. 151, the inspection process in the inspection target area is executed with the high gradation distance image as it is without undergoing the gradation conversion (). Step S15101).
(Hide function of image selection)

Further, at the time of setting, it is possible to limit the selection of the image to be set for the inspection process according to the type of the inspection process. That is, when the inspection process selected by the inspection process selection means can be executed for any of the distance image and the luminance image, the distance image or the luminance image can be called as a registered image. On the other hand, although the inspection process can be performed on a distance image, it cannot be performed on a luminance image. For example, the height measurement process is executed for a distance image having highly accurate height information. This cannot be done for a normal luminance image that does not have height information. Therefore, in the inspection process that can be executed only for an image having height information, that is, a distance image, such as a height measurement process, only the distance image can be selected at the time of calling the registered image at the time of setting, and conversely. Make it impossible to select a brightness image. As a result, it is possible to prohibit or eliminate setting work that is originally impossible, such as setting a height measurement process for a luminance image by mistake, eliminate waste of setting, and improve user operability.

Hereinafter, the procedure for setting the inspection processing conditions will be described with reference to the flowchart of FIG. 152. First, in step S15201, the inspection process is selected. Next, in step S15202, it is determined whether the inspection process selected in step S15201 is an inspection process in which either a distance image or a luminance image can be specified, or an inspection process in which only the distance image can be specified. Here, in the case of the inspection process in which either the distance image or the luminance image can be specified, the process proceeds to step S15203, and either the distance image or the luminance image is selected.

For example, consider the case where an "area" processing unit is selected as the inspection processing, as shown in FIG. In this case, either a distance image or a luminance image can be specified. Therefore, as an inspection processing setting item in the "area" processing unit, for example, when pressing the "image setting" button 114 in FIG. 62 to display the image setting screen 380 shown in FIG. 153, an image is displayed from the operation field 122. You can choose. Here, the image to be displayed in the second image display area 121 can be selected in the "display image" selection field 124 provided in the image selection field 382. Further, in the image setting field 384, an input image and a registered image can be selected. From this image setting field 384, the input image can be specified by an image variable. Here, when the "input image" selection field 386 is selected, the image variable selection screen 390 of FIG. 154 is displayed, and a list of selectable images is displayed. On the image variable selection screen 390, any one of the images captured by a plurality of imaging means (cameras) connected to the three-dimensional image processing device can be selected. Here, different image variables are assigned to each imaging means, and the imaging means and the image variables are associated with each other. For example, the distance image captured by the camera 1 has an image variable "& Cam1Img", the distance image of the camera 2 has an image variable "& Cam2Img", and the distance image of the camera 3 has an image variable "& Cam3Img". The image variable "& Cam4Img" is assigned to each. Further, an image variable "& Cam1GrayImg" is assigned to the luminance image captured by the camera 1. The user selects a desired image from the image variable selection screen 390. As described above, since the "area" processing unit can specify either the distance image or the luminance image, not only the distance image but also the luminance image is included in the options and displayed.

Further, if necessary, the image variables listed on the image variable selection screen 390 can be sorted and displayed, making it easy for the user to select a desired image.

On the other hand, if the inspection process is a process in which only the distance image can be specified, the process proceeds to step S15204 and the distance image is selected. That is, the luminance image cannot be selected. For example, as shown in FIG. 45, when the "height measurement" processing unit 266 is selected as the inspection process, only the distance image can be selected. Therefore, as shown in FIG. 46, when the "image setting" button 114 is selected as the inspection processing setting item in the "height measurement" processing unit, the image setting screen 380 is displayed in the same manner, and the image can be selected. .. Here, similarly, when the "input image" selection field 386 is selected, the image variable selection screen 390 of FIG. 155 is displayed, and a list of selectable images is displayed. In this image variable selection screen 390, unlike FIG. 154, the luminance image is not displayed as an option, and only the distance image is displayed as an option. With this configuration, the user can avoid accidentally selecting a luminance image that does not have height information when measuring the height, and an operating environment with less confusion is provided.

When the image is selected in this way, the process proceeds to step S15205, and inspection processing conditions are set for the selected image. In this way, by utilizing the fact that the inspection process in which either the distance image or the luminance image can be specified or the inspection process in which only the distance image can be specified is determined for each inspection process, each inspection process can be performed. By defining the types of images that can be selected, setting mistakes are avoided and it contributes to user convenience.
(Hide function of inspection process)

The above is an example in which the user is allowed to select the inspection process first, and then the selectable images are restricted when selecting the image to be set for the inspection process according to the selected inspection process type. Was explained. Conversely, it is possible to select an image first and then limit the types of inspection processing that can be performed on this image. That is, when a distance image or a brightness image is first selected by the image selection means, and then one or more inspection processes executed by the inspection execution means are selected by the inspection process selection means, the selected image is the distance image or the brightness. Depending on the image, only the inspection processing that can be executed for each image can be selected. With this configuration, it is possible to avoid a situation in which an inspection process that cannot be set by mistake is selected for the selected image or an inspection process condition related to this inspection process is set.

Hereinafter, the procedure for setting the inspection processing conditions by this method will be described with reference to the flowchart of FIG. 156. First, in step S15601, an image is selected. Here, a distance image or a luminance image is selected. Next, in step S15602, it is determined whether the image selected in step S15601 is either a distance image or a luminance image. In the case of a distance image, the process proceeds to step S15603, and the user is made to select an inspection process that can be executed on the distance image by the inspection process selection means. On the other hand, in the case of a luminance image, the process proceeds to step S15604, and the user is made to select an inspection process that can be executed on the luminance image by the inspection process selection means. When the inspection process is selected in this way, the process proceeds to step S15605, and the inspection process conditions for the selected inspection process are set from the inspection process condition setting means.

For example, as shown in FIG. 157, consider a case where a luminance image and a distance image are acquired for the work. As described above, when the image is registered, the luminance image and the distance image are registered at the same time. The user selects one of the images, selects the inspection process for the next selected image, and sets the inspection process conditions. For example, when a luminance image is selected, as shown in FIG. 158, a tool corresponding to the inspection process performed on the luminance image is added. Here, as in FIGS. 44 and 56 described above, when an image is selected and Add is selected from the displayed submenu, a list of inspection processes that can be performed on the luminance image is displayed. For example, as "measurement" processing, "area" processing unit, "scratch" processing unit, and "blob" processing unit 267 are displayed as options. At this stage, inspection processes that cannot be performed on the luminance image, such as the "height measurement" processing unit, are not displayed. As a result, it is possible to prevent the user from accidentally selecting an inspection process that cannot be set for the luminance image. It is also possible to make it impossible to select by graying out while displaying a list of all measurement processes. When the inspection process is selected, the processing unit is determined, and then the inspection process conditions required for this inspection process are set from the inspection process condition setting means.

When a distance image is selected, a tool corresponding to an inspection process that can be executed on the distance image is added as shown in FIG. 159. Again, if you select a distance image and select Add from the displayed submenu, a list of inspection processes that can be performed on the luminance image is displayed. For example, as "measurement" processing, in addition to "area" processing unit, "scratch" processing unit, and "blob" processing unit 267, "height measurement" processing unit is also displayed as an option. When the user selects a desired inspection process, the processing unit is determined, and the inspection processing conditions required for this inspection processing are set from the inspection processing condition setting means.

In this way, by associating the inspection processing tool with the image, it is possible to physically eliminate the combination of the unselectable image and the inspection processing, and easily avoid the setting error by the user.

The three-dimensional image processing apparatus, the three-dimensional image processing method, the three-dimensional image processing program, the computer-readable recording medium, and the recording device of the present invention can be used for an inspection apparatus or the like utilizing the principle of triangular distance measurement.

100, 100', 200, 300, 400 ... Three-dimensional image processing device 1, 1B, 1C, 1D ... Head unit 2, 2'... Controller unit 3 ... Input means 4 ... Display means 10, 10A, 10B ... Imaging means 20 ... Light projecting means; 20A ... First projector; 20B ... Second projector 21 ... Measuring light source 22 ... Pattern generation unit 23, 24, 25 ... Lens 30 ... Head side controller 31 ... Head side calculation unit 32 ... Distance image generation means 34 ... Filter processing unit 36 ... Head side communication means; 36A ... Controller connection interface; 36B ... PC connection interface 38 ... Storage means; 38a ... Distance image storage unit; 38b ... Brightness image storage unit 41 ... Controller side setting means 42 ... Controller side communication means 43 ... Gradation conversion condition setting means 44 ... Reference plane setting means 45 ... Spatial coding switching means 46 ... Gradation conversion means 47 ... Interval equalization processing setting means 48 ... Projection switching means 49 ... Shutter speed Setting means 50 ... Inspection execution means 51 ... Main control unit 52 ... Controller side connection unit 53 ... Operation input unit 54 ... Display control unit 55 ... Communication unit 56 ... RAM
57 ... Controller side storage means 58 ... Auxiliary storage means 59 ... Output unit 60 ... Image processing unit 62 ... Abnormal point highlighting means 64 ... Image search means 66 ... Real-time update means 70 ... PLC
110 ... 3D image processing program GUI screen 111 ... First image display area 112 ... Setting item button area 113 ... "Image registration" button 114 ... "Image setting" button 115 ... "Area setting" button 116 ... "Height extraction" Button 117 ... "Pre-processing" button 118 ... "Detection condition" button 119 ... "Detailed setting" button 120 ... Inspection target area setting screen 121 ... Second image display area 122 ... Operation area 124 ... "Display image" selection field 126 ... "Measurement area" setting field 128 ... "Edit" button 130 ... Measurement area edit screen 140 ... Height extraction selection screen 142 ... Extraction method selection means 144 ... "Extraction" button 145 ... "Extraction area" designation field 146 ... Dot pointer 147 ... "Extraction area" button 148 ... Extraction area setting dialog 149 ... Extraction area selection field 150 ... One point designation screen 152 ... "Z height" display field 154 ... Emphasis method setting field 156 ... Gain adjustment field 158 ... "Detailed setting" Button 160 ... Emphasis method detailed setting screen 162 ... "Extraction height" setting field 164 ... Noise removal setting field 166 ... Invalid pixel designation field 170 ... Three-point designation screen 172 ... Height extraction display field 174 ... Three-point designation "Details""Setting" button 180 ... Three-point specification detailed setting screen 182 ... "Extraction height" setting field 190 ... Height dynamic extraction setting screen 192 ... "Calculation method" selection field 194 ... "Mask area" button 196 ... "Detailed setting" Button 210 ... Average height standard setting screen 220 ... Mask area setting screen 222 ... "Detailed setting" Button 230 ... Average height standard detailed setting screen 240 ... Plane standard detailed setting screen 250 ... Free curved surface standard setting screen 252 ... "Extraction size" Designation field 260 ... Initial screen 261 ... Flow display area 262 ... Third image display area 263 ... "Image capture" processing unit 264 ... "Shapetrax2" processing unit 265 ... "Position correction" processing unit 266 ... "Height measurement" processing unit 266B ... Second "height measurement" processing unit 267 ... "Blob" processing unit; 267B ... "Color inspection" processing unit 268 ... "Edit" button 269 ... Image setting menu 270 ... Image registration screen 271 ... "Camera selection" Column 272 ... "Register" button 280 ... Image setting screen 282 ... "Detailed setting" button 284 ... "Image setting" button 290 ... Three-dimensional measurement setting screen 292 ... "Display by continuous update" column 294 ... Shutter speed setting column 295 ... Numerical display field 296 ... Light and shade range setting field 310 ... Pre-processing setting field 312 ... Unmeasurable standard setting field 314 ... Equal interval processing setting field 316 ... Spatial code setting field 318 ... Projector selection setting field 322 ... "Display image" selection field 324 ... "Edit" button 326 ... Extraction area edit dialog 330 ... Mask area setting field 332 ... "Edit" button 334 ... Mask area edit dialog 340 ... Filter processing setting screen 350 ... Binarization level setting screen 360 ... Judgment condition setting screen 370 ... First submenu 372 ... Second submenu 373 ... "Measurement" menu 380 ... Image setting screen 382 ... Image selection field 384 ... Image setting field 386 ... " Input image "selection field 390 ... image variable selection screen 460 ... height measurement setting screen 620 ... area setting screen 630 ... free curved surface reference setting screen 632 ..." extraction direction "designation field 640 ... detailed setting screen WK, WK7, WK8, WK9 , WK10, WK11, WK12, WK13, WK14 ... Work PC ... Personal computer SA ... Search target area SI ... Spot-shaped icon ROI ... Measurement area BC ... Belt conveyor DE ... Dust

Claims (18)

A three-dimensional image processing device capable of acquiring a distance image including height information of an inspection object and performing image processing based on the distance image.
An imaging means for capturing an image of an object to be inspected,
A distance image generating means capable of generating a distance image based on an image captured by the imaging means, and a distance image generating means.
As a gradation conversion parameter that constitutes a gradation conversion condition when the distance image generated by the distance image generation means is gradation-converted to a distance image having a gradation number lower than the gradation number of the distance image A reference plane setting means for setting a reference plane that serves as a reference for gradation conversion, and
With reference to the height of the reference plane set by the reference plane setting means, the distance image is based on the height information of the distance image having a gradation number lower than the gradation number of the distance image. Gradation conversion means for converting gradation to a low-gradation distance image replaced with a value,
An inspection execution means for executing a predetermined inspection process on a low gradation distance image whose gradation has been converted by the gradation conversion means, and
It is provided with a display means for displaying a low gradation distance image that has been gradation-converted by the gradation conversion means.
During the operation of performing the three-dimensional image processing with the three-dimensional image processing apparatus, the distance image generation means sequentially generates a distance image of the inspection object based on the input image of the inspection object sequentially input from the outside, and the distance image of the inspection object is sequentially generated. The reference plane setting means is configured to change the reference plane according to the height of the inspection object by resetting the reference plane based on the height information of the distance image generated sequentially.
The gradation conversion means is configured to convert each distance image into the low gradation distance image based on the height of the reference plane reset for each distance image generated by the distance image generation means. ,
A three-dimensional image processing apparatus characterized in that the gradation conversion means is configured to perform position correction using the distance image when performing gradation conversion. The three-dimensional image processing apparatus according to claim 1, further comprising a light projecting means for projecting incident light as structured illumination of a predetermined projection pattern from an oblique direction with respect to the optical axis of the imaging means. Ori,
The image pickup means acquires the reflected light projected by the light projecting means and reflected by the inspection object to capture a plurality of pattern projection images.
The three-dimensional image processing apparatus is characterized in that the distance image generation means is configured to generate a distance image based on a plurality of pattern projection images captured by the image pickup means. The three-dimensional image processing apparatus according to claim 1, further comprising an extraction direction input means for designating a direction for extracting a local shape change in gradation conversion by the gradation conversion means.
With
The reference plane setting means is configured to set a reference plane based on an input image of an inspection object input from the outside during operation in which the three-dimensional image processing apparatus performs three-dimensional image processing.
The three-dimensional image processing apparatus is configured such that the reference surface setting means calculates a curved surface from a distance image displayed on the display means and sets the calculated curved surface as a reference surface. .. The three-dimensional image processing apparatus according to claim 3.
A three-dimensional image processing apparatus, wherein the extraction direction input means can specify any one of the X direction, the Y direction, and the XY direction of a distance image as a direction for extracting a local shape change. The three-dimensional image processing apparatus according to claim 3 or 4.
At the time of gradation conversion, the gradation conversion means reduces the distance image once, performs filtering processing, and then enlarges it to create a free curved surface image, and obtains a difference between the free curved surface image and the distance image. It extracts the shape change of the distance image,
A three-dimensional image processing apparatus characterized in that when the distance image is reduced, the reduction is performed only in the direction specified by the extraction direction input means. The three-dimensional image processing apparatus according to any one of claims 3 to 5, further comprising a light projecting means for projecting light on an inspection object by a light cutting method.
The imaging means is configured to capture an image projected by the light projecting means, and the distance image generating means is configured to synthesize a profile obtained by a light cutting method to generate a distance image. Three-dimensional image processing device. The three-dimensional image processing apparatus according to any one of claims 1 to 6.
When the reference plane setting means performs three-dimensional image processing with the three-dimensional image processing device, the height information in the extraction region designated on the distance image is obtained with respect to the newly input distance image. A reference plane as a reference for gradation conversion is set by the calculation method selected based on the above, and the gradation conversion by the gradation conversion means is executed based on the height of the set reference plane. A three-dimensional image processing device characterized by being The three-dimensional image processing apparatus according to claim 7.
The reference plane setting means is configured to set the height information of the designated point as the reference plane by designating an arbitrary point in the distance image displayed on the display means. A featured three-dimensional image processing device. The three-dimensional image processing apparatus according to claim 7.
The reference plane setting means is configured to set the height information of the designated point as a reference plane by designating an arbitrary point in the distance image displayed on the display means. A featured three-dimensional image processing device. The three-dimensional image processing apparatus according to claim 7.
The reference plane setting means is configured to set the height information of the designated plane as a reference plane by designating an arbitrary plane in the distance image displayed on the display means. A featured three-dimensional image processing device. The three-dimensional image processing apparatus according to claim 7.
By designating any three points in the distance image displayed on the display means, the reference plane setting means sets the height information of the plane including the designated three points as the reference plane. A three-dimensional image processing device characterized by being made of. The three-dimensional image processing apparatus according to claim 7.
The three-dimensional image processing apparatus is configured such that the reference surface setting means calculates a curved surface from a distance image displayed on the display means and sets the calculated curved surface as a reference surface. .. It is a three-dimensional image processing method that acquires a distance image including height information of an inspection object and performs image processing based on the distance image.
A process of projecting incident light from an oblique direction with respect to the optical axis of the imaging means by the light projecting means as structured illumination of a predetermined projection pattern.
A step of acquiring the reflected light projected by the light projecting means and reflected by the inspection object and capturing a plurality of pattern projection images by the imaging means.
A step of generating a distance image by the distance image generation means based on a plurality of pattern projection images captured by the image pickup means, and
As a gradation conversion parameter that constitutes a gradation conversion condition when the distance image generated by the distance image generation means is gradation-converted to a distance image having a gradation number lower than the gradation number of the distance image The process of setting a reference plane that serves as a reference for gradation conversion, and
Based on the height of the set reference plane, the distance image has a gradation number lower than the gradation number of the distance image, and the height information of the distance image is replaced with a grayscale value of the image. Gradation distance The process of converting gradation to an image and
A step of executing a predetermined inspection process on the gradation-converted low-gradation distance image, and a step of prompting to specify a direction for extracting a local shape change when setting the reference plane. Includes
At the time of gradation conversion, a curved surface is calculated from the distance image, and the calculated curved surface is used as a reference plane for gradation conversion to a low gradation distance image to extract a shape change in the specified direction.
The distance image generation means sequentially generates a distance image of the inspection object based on the input image of the inspection object sequentially input from the outside, and based on the height information of the sequentially generated distance image, the reference plane. By resetting, the reference plane can be changed according to the height of the inspection object.
Each distance image is converted into the low gradation distance image based on the height of the reference plane reset for each distance image generated by the distance image generation means.
A three-dimensional image processing method characterized by performing position correction using the distance image when performing the gradation conversion. The three-dimensional image processing method according to claim 13.
The three-dimensional image processing comprising a step of prompting to specify one of the X direction, the Y direction, and the XY direction of the distance image as the direction for extracting the local shape change when setting the reference plane. Method. The three-dimensional image processing method according to claim 13 or 14.
At the time of the gradation conversion, the distance image is once reduced, filtered, enlarged to create a free curved surface image, and the difference between the free curved surface image and the distance image is taken to extract the shape change of the distance image. It is something to do
When reducing the distance image, the three-dimensional image is characterized in that the reduction is performed only in the direction specified by the extraction direction input means for designating the direction for extracting the local shape change in the gradation conversion. Processing method. The three-dimensional image processing method according to any one of claims 13 to 15.
In the process of setting the reference plane, the reference plane is set based on the input image of the inspection object input from the outside during the operation of performing the three-dimensional image processing with the three-dimensional image processing device.
A three-dimensional image processing method characterized in that position correction of an input image is performed in the step of gradation conversion. A three-dimensional image processing program capable of acquiring a distance image including height information of an inspection object and performing image processing based on the distance image, which is applied to a computer.
Based on a plurality of pattern projection images captured by projecting incident light from an oblique direction with respect to the optical axis of the imaging means as structured illumination of a predetermined projection pattern and acquiring the reflected light reflected by the inspection object. And a distance image generation function that can generate a distance image,
A function to display the distance image generated by the distance image generation function and
In the setting mode for making necessary settings, as a setting function for accepting operations from the user and making various settings, the gradation conversion condition for gradation conversion of the distance image generated by the distance image generation function is set. As the constituent gradation conversion parameters, a reference plane setting function for setting a reference plane as a reference for performing the gradation conversion, and a reference plane setting function.
In the operation mode in which the three-dimensional image processing is performed after the setting mode, the distance image is replaced with the height information of the distance image having a gradation number lower than the gradation number of the distance image with the shading value of the image. Image gradation conversion function for gradation conversion to low gradation distance images
An inspection execution function for executing a predetermined inspection process on the low gradation distance image is realized.
In the operation mode
The reference plane setting function is configured to set the reference plane based on an input image of an inspection object input from the outside at the time of operation in which the three-dimensional image processing device performs three-dimensional image processing.
The gradation conversion function is configured to correct the position of the input image.
The distance image generation function sequentially generates a distance image of the inspection object based on the input image of the inspection object sequentially input from the outside, and based on the height information of the sequentially generated distance image, the reference plane. By resetting, the reference plane can be changed according to the height of the inspection object.
Each distance image is converted into the low gradation distance image based on the height of the reference plane reset for each distance image generated by the distance image generation function.
A three-dimensional image processing program characterized in that it is configured to perform position correction using the distance image when performing the gradation conversion. A computer-readable recording medium or device that records the three-dimensional image processing program according to claim 17.

Images