JP2020193867A - Image processing device, control method, control program, and computer-readable recording medium - Google Patents

Abstract

To allow an image inspection such as quality judgment of parts can be performed stably and at high speed even in an environment where a complicated background is reflected in a camera.SOLUTION: An image processing device is provided with: a grayscale image acquisition lighting device 102 that radiates a first illumination light including a first wavelength, and a distance image acquisition lighting device 103 that radiates a second illumination light including a second wavelength different from the first wavelength. An imaging device 101 takes an image of an object in a state where the object is illuminated by irradiating the object with the first illumination light and the second illumination light from the respective lighting devices. A grayscale image generation unit 110 generates a grayscale image composed of pixels imaged through a first optical filter that transmits the first wavelength. A distance image generation unit 111 generates a distance image including pixels imaged through a second optical filter that transmits the second wavelength. An image processing unit 104 determines an inspection area of the grayscale image or the distance image by using distance information of the object of the distance image.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing apparatus, a control method, a control program, and a computer-readable recording medium that inspect the object according to the result of image processing on the image data obtained by capturing the object.

Conventionally, in the manufacturing process of various industrial products, image data acquired by imaging parts is image-processed, and the processing results are used to perform various images such as presence / absence inspection, front / back inspection, position inspection, posture inspection, and appearance inspection of parts. Inspection is being carried out.

In the image processing used in these image inspections, the image feature amount expressing the characteristic that can judge the quality of the component with respect to the image data acquired by imaging the component, for example, the brightness average value and the position of the feature point / feature edge. , Edge strength, etc. are calculated. For example, in the case of quality determination, a threshold value of an image feature amount capable of quantitatively determining the quality of a component is set, and the quality is determined by whether or not the image feature amount obtained by image processing exceeds the threshold value.

On the other hand, in the image data obtained by imaging the actual parts, not only the parts that should be inspected originally, but also the pattern on the surface of the inspection jig, the structure of the machine that conveys the parts, and the parts are placed. Various objects with complicated patterns such as trays are often reflected as the background. If such a complicated background is reflected, an erroneous judgment may occur in the image inspection. For example, in such an erroneous judgment, the image feature amount calculated from the background image information is compared with the image feature amount that should be originally used to judge the quality calculated from the image information of the surface or the end of the part. It is generated by calculating the superimposed image feature amount.

As a technique for stably inspecting the quality of parts from an image in which a complicated background is reflected, a configuration as shown in Patent Document 1 below has been proposed. In the apparatus of Patent Document 1, a normal shading image is captured and acquired, and a distance image having a shading value according to a distance is acquired. Then, the inspection area for performing the inspection based on the other image information is specified by the image information of one of the shading image and the distance image. With such a configuration, for example, even in a component inspection in an environment where complex backgrounds that cannot be distinguished from each other are reflected in a shade image, the area of the component is specified by the distance image, and the area of the identified component is covered. Only can the inspection using the shade image be carried out. In such a configuration, the grayscale image and the distance image need to handle the area information at the same position and the same scale, and it is desirable that they can be acquired by the same camera.

Japanese Unexamined Patent Publication No. 2012-21914

Here, in order to stably inspect the parts by the conventional technique described in Patent Document 1 in the environment in the actual factory, the image feature amount used for specifying the area of the parts or inspecting the quality of the parts. It is necessary to irradiate with appropriate illumination light to emphasize. Further, as described above, it is desirable that the shading image and the distance image can be acquired by the same camera.

However, the appropriate illumination method is not always the same when the shading image is acquired and when the distance image is acquired. For example, in the case of identifying a predetermined part in a shading image, quality inspection, etc., an illumination light that emphasizes only the outline of the part or the edge part of the unevenness, an illumination light that makes the surface of the part glow uniformly white, or a surface surface. Illumination light that emphasizes unevenness is preferable. As a result, the image feature amount used in the subsequent image processing can be calculated stably. On the other hand, when identifying the area of a component or inspecting the quality of a component using a distance image, illumination light may be used so that the distance information between the camera and the component can be obtained accurately and stably. In the passive stereo method, the pattern that makes it easier to associate the left and right cameras is irradiated as an auxiliary, or in the active stereo method, slit light or pattern light is irradiated by laser or condensing to obtain position and distance information on the image. The accuracy of the association can be improved.

As described above, the preferred mode of illumination light differs between the shading image and the distance image. Therefore, it is necessary to perform at least two imaging such as the first imaging using the illumination light suitable for the grayscale image and the second imaging using the illumination light suitable for the distance image, and the imaging takes time. There is such a problem. In addition, in the method of performing multiple imaging, the parts move due to impact or vibration between the two imaging, and the inspection area shifts between the grayscale image and the distance image, and the quality inspection may fail. There is sex.

In view of the above problems, the subject of the present invention is that the background can be reliably separated even in an environment where a complicated background is reflected in the camera, and image inspection such as quality determination of parts can be performed stably and at high speed. To do so.

In order to solve the above problems, in the present invention, in the present invention, a first illuminating device that irradiates a first light including the first wavelength and a second light including a second wavelength different from the first wavelength. The object is irradiated with the first light and the second light from the second lighting device for irradiating the object, the imaging device for imaging the object, and the first lighting device and the second lighting device. A configuration including a controller for imaging the object using the imaging device in this state was adopted.

According to the above configuration, by wavelength separation of the first light and the second light, a first image or a second image corresponding to a grayscale image or a distance image can be acquired by one imaging. Therefore, for example, even in an environment where a complicated background is reflected in the camera, the background can be reliably separated, and image inspection such as quality determination of parts can be performed stably and at high speed.

It is explanatory drawing which showed the system structure of the image processing apparatus which concerns on Embodiment 1 of this invention. It is explanatory drawing which showed the appearance of the part which concerns on Embodiment 1 of this invention. It is explanatory drawing which showed the structure of the coaxial epi-illumination type LED lighting apparatus which concerns on Embodiment 1 of this invention. It is explanatory drawing which showed the ring type LED lighting apparatus which concerns on Embodiment 1 of this invention. It is explanatory drawing which showed the structure of the wavelength filter which concerns on Embodiment 1 of this invention. It is explanatory drawing which showed the filter order on the image imaged through the wavelength filter which concerns on Embodiment 1 of this invention. It is explanatory drawing which showed the method of generating the shade image which concerns on Embodiment 1 of this invention. It is explanatory drawing which showed the method of generating the distance image which concerns on Embodiment 1 of this invention. It is explanatory drawing which showed the example of the shading image which concerns on Embodiment 1 of this invention. It is explanatory drawing which showed the example of the distance image which concerns on Embodiment 1 of this invention. It is explanatory drawing which showed the example of the mask image which concerns on Embodiment 1 of this invention. It is explanatory drawing which showed the example of the masked shading image which concerns on Embodiment 1 of this invention. It is explanatory drawing which showed the system structure of the image processing apparatus which concerns on Embodiment 2 of this invention. It is explanatory drawing which showed the example of the pattern light projected by the projector which concerns on Embodiment 2 of this invention. It is explanatory drawing which showed the structure of the wavelength filter which concerns on Embodiment 3 of this invention. It is explanatory drawing which showed the filter order on the image imaged through the wavelength filter which concerns on Embodiment 3 of this invention. It is explanatory drawing which showed the method of generating the shade image which concerns on Embodiment 3 of this invention. It is explanatory drawing which showed the method of generating the distance image which concerns on Embodiment 3 of this invention. It is explanatory drawing which showed the structure of the image processing apparatus which concerns on Embodiment 4 of this invention. It is explanatory drawing which showed the structure of the image processing apparatus which concerns on Embodiment 5 of this invention. It is explanatory drawing which showed the example of the masked distance image which concerns on Embodiment 6 of this invention. It is explanatory drawing which showed the example of the image obtained by projecting the pattern light by the projector which concerns on Embodiment 2 of this invention. It is a flowchart which showed the processing procedure of the image inspection of the image processing apparatus which concerns on Embodiments 1-6 of this invention. It is a block diagram which showed the structural example of the image processing part of the image processing apparatus which concerns on Embodiments 1-6 of this invention.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the accompanying drawings. The configuration shown below is merely an example. For example, a person skilled in the art can appropriately change the detailed configuration without departing from the spirit of the present invention. Further, the numerical values taken up in the present embodiment are reference numerical values and do not limit the present invention.

<Embodiment 1>
The first embodiment of the image processing apparatus according to the present invention will be described.

FIG. 1 shows the overall configuration of the image processing apparatus according to the present embodiment. As shown in the figure, this image processing device includes an image pickup device 101, an image processing unit 104, a shading image acquisition lighting device 102, a distance image acquisition lighting device 103, and a controller 106.

The image pickup device 101 includes two image pickup elements 107 and 107, two wavelength filters 108 and 108, two lenses 109 and 109, a shading image generation unit 110, and a distance image generation unit 111.

The image processing unit 104 includes an inspection area specifying unit 112 and an inspection processing unit 113. The controller 106 controls the operation of the imaging device 101 by the imaging command, and controls the operation of the shading image acquisition lighting device 102 and the distance image acquisition lighting device 103 by the lighting control command. The controller 106 controls the operation of the transfer robot 116 by a robot control command according to the processing result of the image processing unit 104. The transfer robot 116 operates a component 105 placed on or around a stand 114 arranged on the stand 115. By operating the transfer robot 116 (robot device), an article such as an industrial product or its component assembly is manufactured from the component 105.

In the present embodiment, the image pickup device 101 is implemented in the form of, for example, a stereo camera including two image pickup elements 107, 107, wavelength filters 108, 108, and lenses 109, 109. However, the constituent members of these image pickup devices 101 may be composed of more image pickup elements, wavelength filters, and lenses, such as three or four.

The image inspection unit of the present embodiment is configured to be able to acquire a grayscale image and a distance image used for image processing by one stereo shooting using one imaging device 101. Therefore, in the present embodiment, the shading image acquisition lighting device 102 and the distance image acquisition lighting device 103 are configured to irradiate illumination light having different wavelengths. Further, the shading image acquisition illuminating device 102 and the distance image acquisition illuminating device 103 are configured to have an illumination light irradiation form suitable for each application, as will be described later. Then, for example, the image sensors 107 and 107 are read out in pixel units of each of the R, G, and B colors, and pixel data having the spectral spectrum of each of the above lighting devices is acquired. If the image pickup devices 107 and 107 have such a read-out mechanism, such read-out may be performed using the read-out mechanism.

The wavelength filters 108 and 108 are usually arranged in front of the image pickup devices 107 and 107 and on the object side, and have their respective wavelengths in front of the sensor that acquires the R, G, and B spectral intensities for each color pixel. It is configured as a filter array through which.

Here, with reference to FIG. 23, the flow of the process of inspecting the parts in the above configuration will be described. First, both the shading image acquisition lighting device 102 and the distance image acquisition lighting device 103 are turned on by the lighting control command of the controller 106 (FIGS. 23 and S10). In this state, the image pickup apparatus 101 simultaneously captures an image with the two image pickup elements 107 and 107 in response to the image pickup command of the controller 106, and obtains two image data (FIGS. 23, S12: imaging step).

Here, for example, the shading image acquisition lighting device 102 irradiates the blue (B) wavelength (first wavelength), and the distance image acquisition lighting device 103 irradiates the red (R) wavelength (second wavelength). The lighting form is to be used. Based on the two image data, the image pickup device 101 generates a shading image and a distance image obtained by the illumination light of the shading image acquisition illuminating device 102 and the distance image acquisition illuminating device 103 by the shading image generation of the image processing unit 104. Transfer to unit 110 and distance image generation unit 111.

In step S14 of FIG. 23, a grayscale image is generated using a set of first pixel data captured by the imaging light that has passed through only the first optical filter through which blue light is transmitted among the wavelength filters 108 and 108. .. In addition, a distance image is generated using a set of second pixel data captured by the imaging light that has passed only through the second optical filter through which red light is transmitted.

The inspection area specifying unit 112 of the image processing unit 104 identifies the inspection area based on the image information of the generated distance image (FIGS. 23 and S16). For example, the range of an image in which only an object having a specific imaging distance obtained from a distance image is captured is specified as an inspection area. The imaging distance of the imaged object can be calculated from, for example, the parallax of each of the images stereo-imaged by the image sensors 107 and 107.

Further, the inspection processing unit 113 performs inspection processing only on the inspection area specified by the inspection area specifying unit 112 in the grayscale image, and determines the quality of the part (FIGS. 23, S18: inspection step). The determination result is transferred to the controller 106, and the transfer robot 116 or other production equipment (details not shown) is controlled in the subsequent process based on the transferred quality determination result (FIGS. 23 and S20). By using the image processing device of the present embodiment, the robot system as described above can be configured.

As described above, in the present embodiment, since the shading image and the distance image can be acquired by one imaging, the quality inspection of the component using the shading image and the distance image can be performed at high speed. .. Further, since the imaging is not performed multiple times, it is extremely unlikely that an inspection error will occur due to the time lag of imaging as described above. In the following, each component will be described in more detail.

The part 105 is an object that constitutes an article that is a product. In this embodiment, as shown in FIG. 2, the parts have, for example, a gear-like shape. Further, the parts are placed on the pedestal 114 arranged on the gantry 115, for example, by hand, and the image inspection unit of FIG. 1 inspects the gear-like shape for defects such as chips. It shall be.

As shown in FIG. 2, the component 105 is arranged on the stand 114 so that the gears can be seen as a circle from the image pickup apparatus 101. It is assumed that the surface of the component 105 has a pattern (texture). In the example of FIG. 2, a large number of objects 105a, 105b, etc. having a shape similar to that of the component 105 are arranged around the table 114 on which the component 105 is placed. Therefore, when the component 105 is imaged and image data is obtained, it is assumed that a large number of shapes similar to the component are reflected in the background. It is assumed that the positions of the component 105 and the stand 114 on which the component 105 is placed are indefinite and may change to various positions. Then, for example, if the result of the image inspection of the component 105 is a good product, the controller 106 is transported to a subsequent process by the transfer robot, and if the inspection result of the component 105 is a defective product, the system is stopped and a warning sound is emitted. It shall be generated.

The controller 106 in FIG. 1 is a device that uses the image processing result of the image processing unit 104. Various mounting forms of the controller 106 can be considered, and examples thereof include PLC (Programmable Logical Controller) that controls the production line in an integrated manner. Alternatively, the function of the controller 106 may be implemented in a robot control device or the like that controls the transfer robot 116. When the controller 106 is a PLC, the input / output of the IO signal to the other blocks in the figure is performed by using a communication interface such as Ethernet or RS-232C communication. Various commands shown in parentheses in FIG. 1 are transmitted and received via this communication interface.

The controller 106 issues a lighting control command to the shading image acquisition lighting device 102 and the distance image acquisition lighting device 103 to emit two types of lighting. Next, the controller 106 issues an imaging command to the imaging device 101, causes the imaging device 101 to generate a grayscale image and a distance image, further causes the image processing unit 104 to execute an inspection process, and the image processing unit 104 processes the image. Receive the result. Then, the controller 106 issues a robot control command according to the processing result to the transfer robot 116 based on the image processing result received from the image processing unit 104. For example, in the case of a good / bad inspection, the transfer robot 116 is controlled to convey the component 105 to the subsequent process in the non-defective product determination, and the transfer robot 116 is controlled to remove the component 105 from the line in the defective product determination.

In this embodiment, the controller 106 issues an imaging command to the imaging device 101 to start imaging. However, the controller 106 may issue an imaging command to the image processing unit 104 instead of the image processing device 101, and may issue an imaging command to the imaging device 101 via the image processing unit 104.

The shading image acquisition lighting device 102 is, for example, a coaxial epi-illumination type LED lighting device as shown in FIG. In this type of coaxial epi-illumination type LED lighting device, the illumination light of the LED element 1021 arranged on the side surface as shown in the drawing is reflected by the half mirror 1023 to illuminate the object arranged below. A diffuser plate 1022 or the like can be arranged in front of the LED element 1021 if necessary. As the light source, not only the LED element but also another light emitting element, for example, a laser light source such as a semiconductor laser element may be used.

By using the coaxial epi-illumination type LED lighting device as shown in FIG. 3, the image pickup device 101 is illuminated with parallel light illumination light having an illumination direction substantially parallel to the image pickup optical axis via a half mirror. Equivalent image data can be obtained. Therefore, it is particularly effective when the surface of an object is specularly reflected and the specularly reflected light is imaged. In the present embodiment, the emission wavelength (emission color) of the LED element 1021 of the shading image acquisition lighting device 102 is, for example, blue (B). However, as will be described later, in the shading image generation, the blue (B) and green (G) pixels excluding the red (R) are read out from the image sensor 107 and used. Therefore, the emission wavelength (emission color) of the LED element 1021 may include, for example, a spectrum of green (G).

When inspecting a component, the shading image acquisition lighting device 102 illuminates the component with a predetermined brightness based on a lighting control command transmitted from the controller 106 prior to imaging. In the present embodiment, the shading image acquisition lighting device 102 is configured to capture the specularly reflected light of the component by the coaxial epi-illumination type LED lighting device as described above. However, the shading image acquisition lighting device 102 may be configured by a ring-type LED lighting device that irradiates low-angle illumination light in order to emphasize patterns and scratches on the surface. Further, for the shading image acquisition lighting device 102, the optimum shape and the optimum arrangement position can be selected for a specific image inspection. Further, as the light emitting element, any type of light emitting element other than the LED element 1021 may be used as long as the light emitting wavelength band can be selected.

On the other hand, in the present embodiment, the distance image acquisition lighting device 103 is composed of, for example, a ring-shaped LED lighting device as shown in the upper half of FIG. This LED lighting device has a cross-sectional shape as shown in the lower half of the figure. That is, a large number of LED elements 1031 are arranged in a ring-shaped housing portion in which the inner circumference and the lower portion are opened from metal or plastic so that the irradiation direction faces the direction of the object below. The front of the LED element 1031 is preferably covered with a ring-shaped diffuser plate 1033.

In the present embodiment, the ring-shaped LED lighting device as shown in FIG. 4 constitutes the distance image acquisition lighting device 103, and is arranged at a height close to the component 105 of the object, for example, as shown in FIG. .. As a result, the component 105 is illuminated from an angle close to the side or the side of the periphery. With such a configuration, it is possible to control the specular reflection light of the illumination light so as not to be reflected in the direction of the image pickup apparatus 101, and to perform imaging mainly by the diffuse reflection light of the object. In the present embodiment, the emission wavelength (emission color) of the distance image acquisition lighting device 103 is, for example, red (R).

The distance image acquisition lighting device 103 illuminates the component with a predetermined brightness based on the illumination control command transmitted from the controller 106 at the time of imaging at the time of inspecting the component. It is assumed that the lighting state when the image pickup device 101 performs imaging is that the parts are illuminated by both the shading image acquisition lighting device 102 and the distance image acquisition lighting device 103.

In the present embodiment, the distance image acquisition lighting device 103 uses a ring-shaped LED lighting device, but has an optimum shape for emphasizing the surface pattern according to the component 105 to be inspected. Other LED lighting devices that can be used in the optimum position can be used. Further, also in the case of the distance image acquisition lighting device 103, as the light emitting element, any type of light emitting element other than the LED can be used as long as the emission wavelength band can be selected.

The image pickup device 101 includes two image pickup elements 107, 107, two wavelength filters 108, 108, two lenses 109, 109, a grayscale image generation unit 110 (first image generation unit), and a distance image generation unit 111 (second image generation unit). Image generator). Preferably, the image pickup apparatus 101 is manufactured in the form of, for example, an integral camera in which these members are incorporated in a single housing. However, each of the two image pickup elements, the wavelength filter, and the lens portion may be configured by using, for example, a separate camera device. Hereinafter, the components of the image pickup apparatus 101 will be described in detail.

For example, the image pickup devices 107 and 107 are configured by a CMOS image sensor having a bit depth of 8 bits for each pixel, and output image data to the outside of the element by an LVDS signal. The image pickup optical system that combines the image pickup elements 107 and 107 and the lenses 109 and 109 has an image pickup field of view (angle of view) that is about twice the diameter of the component in the horizontal and vertical directions with respect to the image pickup field of view of the image pickup elements 107 and 107. Configure to have. This type of image sensor 107, 107 can read image data after writing a set value to a register built in the image sensor 107, 107 via IO specification communication such as SPI or I2C.

The image pickup devices 107, 107 start the image pickup operation in response to a shooting trigger issued inside or outside the device. The image pickup elements 107 and 107 take two images at the timing of an imaging command issued from the controller 106 after the shading image acquisition lighting device 102 and the distance image acquisition lighting device 103 are in a predetermined lighting state. .. Further, it is assumed that the image pickup elements 107 and 107 can perform thinning imaging and binning imaging by writing the set value to the register. The two image pickup elements 107, 107 are arranged on the circuit board so as to be arranged in the horizontal direction, for example, as shown in FIG.

The wavelength filters 108 and 108 are color filters that are arranged on-chip with respect to the image pickup elements 107 and 107 so that the positions correspond to the pixels of the image pickup elements 107 and 107, for example. For example, the color filters constituting the wavelength filters 108 and 108 are R (red) and G (green) for each 2 × 2 pixel of the image pickup elements 107 and 107, for example, as shown by 108R, 108G and 108B in FIG. ), G (green), B (blue) are arranged in the order of Bayer arrangement. As shown in FIGS. 5 and 6, the image data captured through this color filter has R (red) component (upper left), G (green) component (lower left), and G (green) for each 2 × 2 pixel. Pixel data corresponding to the wavelength of the pattern of the component (upper right) and the B (blue) component (lower right) can be obtained. The above filter array pattern is only an example. The arrangement patterns of R (red), G (green), and B (blue) of the color filters may be in any order or combination different from those in the present embodiment. In the present embodiment, the same wavelength filters 108 and 108 are arranged on-chip for the two image sensors 107 and 107, respectively.

As shown in FIG. 7, the grayscale image generation unit 110 ignores the pixels of the R (red) component for either one of the two image data obtained by imaging (1071). Only the pixels of the G (green) component and the B (blue) component are extracted (1072). In addition, when the image is output after thinning or binning with 2 × 2 pixels of the color filter as one block by the function of the image sensor, R (red), G (green) to B (blue) are similarly obtained. Only the desired pixels of can be extracted. After that, the grayscale image generation unit 110 performs a known Bayer development process on the extracted image, grayscales it, and generates a grayscale image.

When the stand 114 on which the component 105 is placed and its surroundings are imaged in the state of FIG. 2, the shade image generation unit 110 generates a shade image as shown in FIG. In FIG. 9, the image 302 is an image of the pedestal 114, the image 303 is an image of the component 105 of FIG. 2 to be inspected, and the images 304 and 305 are images of the objects 105a, 105b ... Around the pedestal 114.

Although the configuration in which the grayscale image generation unit 110 is provided in the image pickup apparatus 101 is shown here, a configuration in which the grayscale image generation unit is provided in the image processing unit 104 may be used. In that case, the image captured by the image pickup apparatus 101 may be transferred to the image processing unit 104 as it is, and a shade image may be generated in the image processing unit 104. It is desirable that the shade image generated by the shade image generation unit 110 is in a format that holds at least 2 channels of G (green) and B (blue) as data in order to reduce the capacity. However, in order to divert data processing, a format may be used in which data for 3 channels of R (red), G (green), and B (blue) are retained. Further, in order to further reduce the capacity, a format may be used in which data for one channel of B (blue) is held.

On the other hand, as shown in FIG. 8, the distance image generation unit 111 ignores the pixels of the G (green) component and the B (blue) component for both of the two image data obtained by imaging, and R (red). ) Only the pixel of the component is extracted (1071, 1073). In addition, when the image is output after thinning or binning with 2 × 2 pixels of the color filter as one block by the function of the image sensor, R (red), G (green) to B ( Only the predetermined pixels of blue) can be extracted. Then, the extracted image is subjected to a known Bayer development process and grayscaled to obtain two grayscale images. Further, using a known stereo method, two grayscale images are associated with each other in stereo, parallax is calculated for each pixel, and a distance image is generated.

As shown in FIG. 10, the distance image generated here is represented by a brightness value that is brighter as the distance from the image pickup apparatus 101 is longer and darker as the distance is shorter (FIG. 10: 301). The distance image in this embodiment has 32 bits of floating point data as distance data for each pixel. Although the present embodiment shows an example in which the distance image generation unit 111 is provided in the image pickup apparatus 101, a configuration in which the distance image generation unit 111 is provided in the image processing unit 104 may be used. In that case, the image captured by the image pickup apparatus 101 can be transferred to the image processing unit 104 as it is, and the distance image can be generated on the side of the image processing unit 104.

Further, in the present embodiment, the illumination light of the blue (B) wavelength band is used for the shade image, and the illumination light of the red (R) wavelength band is used for the distance image. However, the assignment of the wavelength (emission color of the illumination) of the illumination light (photographed light) handled by the grayscale image generation unit 110 and the distance image generation unit 111 is arbitrary, and an assignment different from the above may be adopted. In short, it suffices that the wavelength band (emission color of the illumination) of the illumination light (photographed light) is different between the shade image and the distance image, and the wavelengths of the other illumination light (photographed light) for each of the shade image and the distance image. A band (emission color of lighting) may be assigned.

The image processing unit 104 includes an inspection area specifying unit 112 and an inspection processing unit 113. The inspection area specifying unit 112 determines the inspection area using the grayscale image and the distance image input from the image pickup apparatus 101, and the inspection processing unit 113 determines the inspection processing for the inspection area specified by the inspection area specifying unit 112. Perform processing. The image processing unit 104 can be mounted by, for example, a PC equipped with a CPU, a board computer, an arithmetic unit such as a GPU or FPGA.

Here, FIG. 24 shows an example of a specific hardware configuration of the image processing unit 104 of FIG. The control system as shown in the figure can be realized, for example, by implementing a so-called PC form.

The control system of FIG. 24 can be configured by a CPU 1601 as a main control means, a ROM 1602 as a storage device, and PC hardware including a RAM 1603. The ROM 1602 can store a control program of the CPU 1601 and constant information for realizing the manufacturing procedure of the present embodiment. Further, the RAM 1603 is used as a work area of the CPU 1601 when executing the control procedure. Further, an external storage device 1606 is connected to the control system of FIG. 24. The external storage device 1606 is not necessarily necessary for the practice of the present invention, but can be configured from an HDD, an SSD, an external storage device of another network-mounted system, or the like.

The control program of the CPU 1601 for realizing the image processing of the present embodiment, for example, the processing of the inspection area specifying unit 112 and the inspection processing unit 113, is a storage unit such as the above-mentioned external storage device 1606 or ROM 1602 (for example, EEPROM area). Store in. In that case, the control program of the CPU 1601 for realizing the control procedure of the present embodiment is supplied to each of the above-mentioned storage units via the network interface 1607 (NIF), and is updated to a new (another) program. Can be done. Alternatively, the control program of the CPU 1601 for realizing the control procedure described later is supplied to each of the above storage units via storage means such as various magnetic disks, optical disks, and flash memories, and a drive device for that purpose. The contents can also be updated. Various storage means, storage units, or storage devices in a state in which the control program of the CPU 1601 for realizing the control procedure of the present invention is stored constitutes a computer-readable recording medium in which the control procedure of the present invention is stored. It will be.

The network interface 1607 can be configured using, for example, a wired communication standard such as IEEE 802.3 or a wireless communication standard such as IEEE 802.11, 802.11. The CPU 1601 can communicate with another device 1104 via the network interface 1607. The device 1104 corresponds to, for example, a controller 106 controlling control device for manufacturing an article, another management server, or the like. Communication with the transfer robot 116 of FIG. 1 or other production equipment on the production line arranged in the transfer robot 116 can also be performed via the network interface 1607.

Further, the control device of FIG. 24 includes a user interface device 400 (UI device). As the user interface device 400, a GUI device including an LCD display, a keyboard, a pointing device (mouse, joystick, etc.) and the like may be arranged. Through such a user interface device 400, it is possible to notify the user of the progress of image processing, set various parameters for controlling image processing, and the like.

The image processing unit 104 includes the following functional elements. These functional elements can be realized by the hardware and software of the control system shown in FIG. 24 described above.

The inspection area specifying unit 112 determines an image area in a specific distance range in the image data captured by the image pickup apparatus 101 as an inspection area based on the distance image generated by the distance image generation unit 111.

Here, for example, each pixel of the distance image shown in FIG. 10 is binarized by threshold processing, and then the pixel value is logically inverted (“1” ⇔ “0”) to obtain a binary image as shown in FIG. .. As a method of selecting the threshold value for this binarization, for example, a method of extracting the maximum value of the distance information of each pixel of the distance image and setting a value half of the maximum value as the threshold value can be considered. In the component arrangement as shown in FIG. 2, as shown in FIG. 11, the region of the image closer to the imaging distance corresponding to the upper surface of the stand 114 is determined as the inspection region 307. This inspection area 307 has a pixel value of "1". Further, the region 306 other than the inspection region 307, which has a longer imaging distance, is a region corresponding to the background, and this region 306 has a pixel value of “0”. Before binarizing the distance image, the image data may be filtered with a known filter to reduce noise. The binary image obtained as shown in FIG. 11 as described above is hereinafter referred to as a "mask image". The subsequent inspection processing unit 113 executes image processing for various inspections such as a quality inspection on the inspection area 307 determined as described above.

The inspection processing unit 113 executes an inspection using, for example, a mask image (FIG. 11) generated from a distance image and a shading image (FIG. 9: 301). Here, first, the shading image (FIG. 9: 301) is masked based on the mask image (FIG. 11) generated by the inspection area specifying unit 112. In this masking process, the luminance value of the shading image (FIG. 9: 301) is adopted as it is for the pixels whose luminance value of the mask image (FIG. 11) is "1", and the luminance value of the mask image (FIG. 11) is "0". The brightness value is set to 0 for the pixel of. By this masking process, as shown in FIG. 12, a significant image exists in the inspection area 307 where the image 308 of the component 105 to be inspected exists, and the area 306 corresponding to the other background has a brightness value “0” (black). ) Is obtained.

Next, as shown in FIG. 12, the inspection processing unit 113 obtains, for example, a quality determination result of the component 105 (308) by performing appropriate image processing on the image in which only the component 105 (308) exists. As this inspection image processing, for example, a method of binarizing an image to generate a binary image and determining that the product is non-defective if the area of the binary image is equal to or larger than the threshold value can be considered. Alternatively, a method of calculating a matching score with a component image prepared in advance by template matching and determining that the product is non-defective if the matching score is equal to or higher than the threshold value may be used. In addition, a method of detecting an edge component of an image, calculating a matching score with shape information prepared in advance by shape matching, and determining a non-defective product if the matching score is equal to or higher than a threshold value, a known non-defective product learning method, or the like is used. May be good. The quality determination result obtained by the inspection processing unit 113 is output to the controller 106.

The controller 106 generates a command content to be transmitted to the transfer robot 116 based on the pass / fail determination result. For example, in the case of a quality inspection, if the inspection result of the parts is a good product, the controller 106 issues a command to the transfer robot 116 to perform an operation of grasping the parts and transporting them to a subsequent process on the production line. If the inspection result of the part is defective, the transfer robot 116 is instructed to grasp the part and discharge it from the production line. At this time, a warning sound may be generated by a voice generating means (not shown). Alternatively, in the case of determining a defective product, an error process may be performed so as to stop the production system including the transfer robot 116 and generate a warning sound.

According to the present embodiment, it is possible to acquire a shade image (first image) and a distance image (second image) in an appropriate lighting state suitable for each imaging by one imaging. Then, the inspection area for performing image processing on the grayscale image can be determined by using the distance information of the distance image. For example, even in an inspection in an environment where a complicated background is reflected in a camera, an inspection area excluding the background portion can be set by using the distance information of the distance image, and an image inspection of an object, for example, a part, for example, The pass / fail judgment can be inspected stably and at high speed.

<Embodiment 2>
In the first embodiment, the image pickup device 101 is a stereo camera, and the distance image (second image) is acquired by stereo shooting, but the distance information can also be acquired by the image pickup device 101 having another configuration. In this embodiment, an example of acquiring a distance image (second image) using illumination light (second light) by pattern light is shown. In the following, the same reference numerals will be used for the same components and functions as described above, and detailed description thereof will be omitted unless otherwise required.

In the configuration of the first embodiment, it is desirable that there is a pattern (texture) on the surface of the component like the illustrated gear, and a distance image can be generated after stereo-associating the two images by stereo imaging. .. However, some actual parts have no pattern (texture) on the surface and cannot be associated with stereo even if stereo imaging is performed. Therefore, parallax cannot be measured and a distance image cannot be generated.

In the configuration of the second embodiment, the distance image (second image) can be acquired even when the object to be inspected does not have a pattern (texture).

FIG. 13 shows the configuration of the image inspection system according to the present embodiment. In the image inspection system of FIG. 13, the image pickup device 101 includes an image pickup element 107, a wavelength filter 108, a lens 109, a grayscale image generation unit 110, and a distance image generation unit 111. However, the image pickup device 101 of FIG. 13 is not a stereo configuration but a so-called monocular image pickup device. The configuration of the image sensor 107, the wavelength filter 108, the lens 109, and the like of the image pickup device 101 is the same as the configuration of the monocular of the first embodiment.

The shading image acquisition lighting device 102 is configured to irradiate the illumination light as parallel light substantially parallel to the imaging optical axis of the imaging device 101, as in the case of the first embodiment. The distance image acquisition lighting device 103 is, for example, a projector equipped with a well-known DMD device.

The configuration of the inspection area specifying unit 112, the image processing unit 104 including the inspection processing unit 113, and the controller 106 is the same as that of the first embodiment. Further, the flow of the process of inspecting the parts is the same as that of the first embodiment (FIG. 23), and detailed description thereof will be omitted.

The component 105 is arranged on the pedestal 114 installed on the gantry 115 as in the first embodiment, and is operated by the transfer robot 116. In this embodiment, it is assumed that there is no pattern (texture) on the surface of the component 105.

The distance image acquisition lighting device 103 is, for example, a projector equipped with a known DMD device. This type of projector illuminates a pattern light having a predetermined light-dark pattern capable of performing a known spatial coding method from an angle different from that of the imaging device, for example, from a direction inclined with respect to the imaging optical axis of the imaging device 101. ..

In FIG. 14, 401, 402, 403, 404 (...) show examples of various pattern lights (second light) projected by the distance image acquisition lighting device 103 composed of the projectors of the present embodiment having different patterns. ing. In the present embodiment, for example, the emission wavelength of the illumination light (second light) of the distance image acquisition illumination device 103 is assumed to be red. When inspecting the component 105, the distance image acquisition lighting device 103 is turned on based on a lighting control command from the controller prior to imaging, and illuminates the component with a predetermined brightness.

At the time of imaging by the imaging device 101, the component 105 is illuminated by both the shading image acquisition lighting device 102 (first lighting device) and the distance image acquisition lighting device 103 (second lighting device). To do. It is assumed that the emission wavelength of the illumination light (first light) of the illumination device 102 (first illumination device) for acquiring a shade image is blue.

Also in this embodiment, as shown in FIG. 7, the shading image generation unit 110 ignores the pixels of the R (red) component and only the pixels of the G (green) component and the B (blue) component with respect to the image data. Is extracted. In addition, when the image is output after thinning or binning with 2 × 2 pixels of the color filter as one block by the function of the image sensor, R (red), G (green) to B (blue) are similarly obtained. Only the desired pixels of can be extracted. Then, the extracted image is subjected to a known Bayer development process and then grayscaled to generate a grayscale image. The shade image is generated in the form shown in FIG. 9 as in the first embodiment.

A modified example in which the shading image generation unit 110 is arranged not in the image pickup apparatus 101 but in the image processing unit 104 can also be implemented in this embodiment. In that case, the image captured by the image pickup apparatus 101 can be transferred to the image processing unit 104 as it is, and the image processing unit 104 can generate a shading image. Further, it is desirable that the grayscale image data generated by the grayscale image generation unit 110 is in a format that holds data for 2 channels of G (green) and B (blue) in order to reduce the capacity. However, for the sake of diversion of data processing and the like, a format may be used in which data for 3 channels of R (red) G (green) B (blue) is retained. Further, in order to further reduce the capacity, a format may be used in which data for one channel of B (blue) is held.

On the other hand, as shown in FIG. 8, the distance image generation unit 111 ignores the pixels of the G (green) component and the B (blue) component of the image data obtained by imaging, and only the pixels of the R (red) component. Is extracted. In addition, when the image is output after thinning or binning with 2 × 2 pixels of the color filter as one block by the function of the image sensor, R (red), G (green) to B ( Only the predetermined pixels of blue) can be extracted. After that, the distance image generation unit 111 performs a known Bayer development process on the extracted image, grayscales it, and generates a distance image using a well-known spatial coding method.

FIG. 22 shows an image obtained by imaging an object as in FIG. 2 similar to the first embodiment with the image pickup apparatus 101 and extracting only the pixels of the R (red) component. In FIG. 22, image 2302 is an image of the pedestal 114, image 2303 is an image of the component 105 of FIG. 2 to be inspected, and images 2304 and 2305 are images of objects 105a, 105b ... Around the pedestal 114. Reference numerals 2301 and 2401 indicate bright and dark portions of the striped pattern light emitted from the projector of the distance image acquisition lighting device 103, respectively. The deviation of the bright and dark portions of the striped pattern light depends on the irradiation angle (known) of the illumination light of the distance image acquisition illumination device 103 and the height difference (shape) of the object. Therefore, the distance image generation unit 111 uses the spatial coding method to generate the height of the portion where the object is illuminated by the pattern light, that is, the distance image from the deviation of the bright and dark portions. In this way, also in the configuration of FIG. 13, a distance image including the distance information used by the inspection area specifying unit 112 in the inspection area specifying process can be generated as in the first embodiment.

Similar to the first embodiment, the generated distance image is represented by, for example, a brightness value that is brighter as the distance from the image pickup apparatus 101 is longer and darker as the distance is shorter (FIG. 10: 301). .. The distance image in this embodiment has 32 bits of floating point data as distance data for each pixel. Although the present embodiment shows an example in which the distance image generation unit 111 is provided in the image pickup apparatus 101, a configuration in which the distance image generation unit 111 is provided in the image processing unit 104 may be used. In that case, the image captured by the image pickup apparatus 101 can be transferred to the image processing unit 104 as it is, and the distance image can be generated on the side of the image processing unit 104.

Also in this embodiment, the illumination light of the blue (B) wavelength band is used for the shade image, and the illumination light of the red (R) wavelength band is used for the distance image. However, the assignment of the wavelength of the illumination light (shooting light) (emission color of the illumination) is arbitrary as in the first embodiment. In short, it suffices that the wavelength band (emission color of the illumination) of the illumination light (photographed light) is different between the shade image and the distance image, and the wavelengths of the other illumination light (photographed light) for each of the shade image and the distance image. A band (emission color of lighting) may be assigned.

The transfer robot 116 can be operated based on the command content commanded by the controller based on the quality determination result. Specifically, if the inspection result of the part is non-defective, the transfer robot grabs the part and transfers it to the subsequent process, and if the inspection result of the part is defective, the system is stopped and a warning sound is generated. Such control can be performed.

In this embodiment, a method of actively projecting a texture on the surface of a component when there is no pattern (texture) on the component has been described by a spatial coding method using a projector. However, in addition to the spatial coding method, the lighting device for distance image acquisition is configured as a single or multi-line slit laser or a multi-point point laser, and the distance information of each place is calculated by a known triangulation or optical cutting method. Then, a distance image may be generated. Alternatively, a projector may actively project a random texture onto the surface of the component, utilize it as a feature point of stereo association, and generate a distance image by the stereo method described in the first embodiment.

According to this embodiment, even when there is no pattern (texture) on the part, it is possible to acquire a shade image and a distance image in appropriate lighting modes by one imaging. Therefore, for example, it is possible to stably and quickly inspect the quality of parts in an environment where a complicated background is reflected in the camera.

<Embodiment 3>
In the above-described first embodiment, the wavelength band of the illumination light having a direction suitable for the grayscale image and the distance image is set to any one of R (red), G (green), and B (blue), for example, R. It was separated into (red) and B (blue). Then, a wavelength filter is used to separate the wavelength band of the illumination light between the grayscale image and the distance image to acquire the grayscale image and the distance image. As a result, it was possible to capture a shade image and a distance image with illumination light having an appropriate illumination mode. However, in an actual inspection, there may be an inspection in which all the color information of R (red), G (green), and B (blue) is required for the shade image. Further, the wavelength component of ambient light such as a fluorescent lamp in an inspection environment may include a wavelength band of R (red), G (green), or B (blue), which has an influence on a distance image, for example. There is a possibility that the distance image cannot be generated normally due to disturbance.

Therefore, in the present embodiment, a configuration is considered in which all the color information of R (red), G (green), and B (blue) can be used for the shade image, and the image inspection can be executed stably and at high speed. Therefore, in this embodiment, wavelengths other than R (red), G (green), and B (blue), for example, infrared (Ir) wavelength bands are assigned for the distance image. The configuration of the image processing device according to the present embodiment is the same as that of FIG. 1 of the first embodiment. That is, this configuration is a configuration in which a stereo camera generates a distance image for using the distance information.

The flow of processing for inspecting parts in this embodiment is as follows. First, both the shading image acquisition lighting device 102 and the distance image acquisition lighting device 103 are turned on by the lighting control command of the controller 106 (FIGS. 23 and S10). Then, an image is taken by the image pickup elements 107 and 107 of the image pickup device 101 having a stereo configuration according to the image pickup command of the controller, and two image data having left-right parallax are obtained (FIGS. 23 and S12).

In the present embodiment, the shading image acquisition lighting device 102 irradiates visible light including wavelength bands of red, green, and blue. On the other hand, the distance image acquisition lighting device 103 irradiates infrared light. The wavelength band of these illumination lights can be set by selecting, for example, an LED element to be used as an illumination light source of each illumination device, or by arranging an appropriate filter in front of each illumination light source.

The grayscale image generation unit 110 and the distance image generation unit 111 are stereo-imaged in the same manner as described above, generate a grayscale image and a distance image based on the two image data, and transfer the grayscale image and the distance image to the image processing unit 104. To do. In the generation of the grayscale image, among the wavelength filters 108 of the image pickup apparatus 101, the pixels corresponding to the filters that transmit the light in the wavelength bands of red (R), green (G), and blue (B) are used. Further, in the generation of the distance image, pixels corresponding to the filter through which infrared light (Ir) is transmitted are used (FIGS. 23 and S14). The image processing unit 104 specifies an inspection area for image inspection based on the image information of the distance image by the inspection area specifying unit 112 (FIGS. 23 and S16). In addition, the inspection processing unit 113 performs inspection processing, for example, quality determination of a component imaged as an object, only for the inspection area specified by the inspection area specifying unit in the shade image (FIGS. 23 and S18). This determination result is transferred to the controller 106, and the operation of the production equipment such as the transfer robot 116 in the subsequent process is controlled based on the determination result (FIGS. 23 and S20).

As described above, also in the present embodiment, the shading image and the distance image can be acquired by one imaging, and the quality inspection of the component using the shading image and the distance image can be performed at high speed.

Subsequently, the configuration of the present embodiment will be described in more detail below. As described above, the configuration of the inspection system in the present embodiment is the same as that in FIG. 1, and the configurations of the parts 105, the controller 106, the image processing unit 104, and the like are the same as those in the first embodiment. The difference from the first embodiment is the configuration of the following lighting device and wavelength filter.

The member arrangement of the shading image acquisition lighting device 102 is the same as that of the first embodiment (FIG. 3), but the emission wavelength of the LED element includes, for example, the wavelength bands of blue (B), green (G), and red (R). Make it white. The member arrangement of the distance image acquisition lighting device 103 is the same as that of the first embodiment (FIG. 4), but for example, the emission wavelength of the LED element is in the infrared (Ir) wavelength range.

The image pickup device 101 is composed of two image pickup elements 107, 107, two wavelength filters 108, 108, two lenses 109, 109, a grayscale image generation unit 110, and a distance image generation unit 111. The image pickup apparatus 101 may be manufactured, for example, in the form of an integrated camera. The configuration of the image sensor 107 and the lens 109 may be the same as in the first embodiment.

The wavelength filter 108 of the present embodiment is slightly different from a general color filter composed of a combination of R (red) G (green) B (blue). In the wavelength filter 108 of the present embodiment, for example, as shown in FIG. 15, R (red), G (green), Ir (infrared), and B (blue) are arranged for every 2 × 2 pixels of the image sensor. ing. This configuration corresponds to a configuration in which one of the two arranged G (green) filters of the wavelength filter 108 of FIG. 5 is replaced with Ir (infrared).

In the image data captured by the image sensor 107 through the wavelength filter 108, as shown in FIG. 16, the upper row shows the R (red) component and the G (green) component, and the lower row shows the Ir (infrared) component for each 2 × 2 pixel. , B (blue) component wavelength band pixels are lined up. Note that this color arrangement is an example, and the arrangement of R (red), G (green), B (blue), and Ir (infrared) of the filter may be in any order or combination different from the above. Further, in the present embodiment, a wavelength filter having a different wavelength component for each color pixel is used, but if the wavelengths are different, such as setting 2 × 2 pixels to one wavelength or setting a predetermined size region to one wavelength. The wavelength filter may be configured by changing the unit to be set. The same wavelength filters 108 and 108 are arranged on the two image pickup devices 107 and 107, respectively.

As shown in FIG. 17, the grayscale image generation unit 110 ignores the pixels of the Ir (infrared) component with respect to either one of the two captured image data (1071), and R Only the pixels of the (red), G (green), and B (blue) components are extracted (1076). When the 2x2 pixels of the filter are set as one block by the function of the image sensor and the image is output after thinning or binning, the same applies to R (red) G (green) B (blue) Ir (infrared). Only a predetermined pixel can be extracted. After that, the extracted image can be subjected to a known Bayer development process and then grayscaled to generate a grayscale image. Although this shade image is a color image, it has the configuration shown in FIG. In the present embodiment, the shading image generation unit 110 is provided in the image pickup apparatus 101, but the shading image generation unit 110 may be provided on the side of the image processing unit 104. In that case, the image captured by the image pickup apparatus 101 may be transferred to the image processing unit 104 as it is, and a shading image may be generated on the side of the image processing unit 104.

On the other hand, as shown in FIG. 18, the distance image generation unit 111 ignores the pixels of the R (red), G (green), and B (blue) components for both of the two image data obtained by imaging. , Only the pixels of the Ir (infrared) component are extracted (1071, 1077). In addition, when the image is output after thinning or binning the 2 × 2 pixels of the filter as one block by the function of the image sensor, R (red) G (green) B (blue) Ir (red) is also used. Only predetermined pixels (outside) can be extracted. Then, the extracted image is subjected to a known Bayer development process and then grayscaled to obtain two grayscale images capable of stereo processing. Further, using a known stereo method, two grayscale images are associated with each other in stereo, parallax is calculated for each pixel, and a distance image is generated. This distance image has distance information of the object being imaged.

As shown in FIG. 10, the distance image is generated as an image represented by a brightness value that is brighter as the distance from the image pickup apparatus is longer and darker as the distance is shorter. The distance image in the present embodiment has 32 bits of floating point data as the distance data for each pixel. Although the distance image generation unit 111 is provided in the image pickup apparatus 101 in the present embodiment, the distance image generation unit 111 may be provided in the image processing unit 104. In that case, the image captured by the image pickup apparatus 101 can be transferred to the image processing unit 104 as it is, and the distance image can be generated on the image processing unit side.

In this embodiment, white illumination light is used for capturing a grayscale image, and infrared wavelength illumination light is used for imaging a distance image. However, if the wavelength bands of the illumination light are different between the shade image and the distance image of different illumination modes, the illumination light of another wavelength band is used for each of the shade image and the distance image, and the corresponding wavelength filter is used. Can be used.

The transfer robot 116 is operated based on a command content commanded by the controller 106 based on, for example, a quality determination result. For example, if the inspection result of the part is a non-defective product, the transfer robot grasps the part and conveys it to a subsequent process, and if the inspection result of the part is a defective product, the system is stopped and a warning sound is generated.

As described above, according to the present embodiment, a shade image can be generated by using all the components of R (red), G (green), and B (blue). Therefore, even when a large amount of color information is required for image processing for quality determination, the desired image inspection can be appropriately performed. Then, as described in the first embodiment, it is possible to stably and quickly inspect the quality of the component in an environment where a complicated background is reflected in the camera.

<Embodiment 4>
In the first and third embodiments described above, the image pickup device 101 has a stereo configuration for acquiring a distance image (distance information), and includes two image pickup elements, two wavelength filters, and two lenses. This configuration tends to be more complex and expensive than a so-called monocular camera, eg, a configuration using a single image sensor, wavelength filter and lens. Further, the second embodiment uses a projector, and the system may be complicated and expensive.

Therefore, in the present embodiment, a configuration is considered in which a simpler configuration, a distance image (distance information) for background removal, etc. can be acquired, and an image inspection such as a quality judgment of a part can be performed stably and at high speed. In the present embodiment, for example, a distance image (distance information) is acquired by using an image sensor composed of an image pickup surface phase difference sensor in the image pickup device 101.

FIG. 19 shows the configuration of the image processing apparatus according to the present embodiment. As shown in the figure, this image processing device includes an image pickup device 101, an image processing unit 104, a shading image acquisition lighting device 102, a distance image acquisition lighting device 103, and a controller 106.

The image pickup device 101 includes an image pickup element 107, a wavelength filter 108, a lens 109, a grayscale image generation unit 110, and a distance image generation unit 111. As shown in the figure, the image pickup apparatus 101 has a monocular configuration, not a stereo camera configuration as shown in FIG. Further, the image processing unit 104 includes an inspection area specifying unit 112 and an inspection processing unit 113.

The configuration around the control system of FIG. 19 is the same as that of FIG. The controller 106 controls the operation of the imaging device 101 by the imaging command, and controls the operation of the shading image acquisition lighting device 102 and the distance image acquisition lighting device 103 by the lighting control command. The controller 106 controls the operation of the transfer robot 116 by a robot control command according to the processing result of the image processing unit 104. The transfer robot 116 operates a component 105 placed on or around a stand 114 arranged on the stand 115. By the operation of the transfer robot 116, an article such as an industrial product or a component assembly thereof is manufactured from the component 105.

The overall flow of the process of inspecting the parts is the same as that described so far with reference to FIG. Next, the details of each component will be described.

The image pickup device 101 is composed of, for example, an image pickup element 107, a wavelength filter 108, a lens 109, a grayscale image generation unit 110, and a distance image generation unit 111. The image pickup apparatus 101 may be manufactured, for example, in the form of an integrated camera.

In the image pickup apparatus 101 of the present embodiment, the image pickup device 107 and the lens 109 are configured in the same manner as the camera of the imaging surface phase difference type. As is well known, in the image pickup surface phase difference method, each of a plurality of pixels of the image pickup device 107 includes two photoelectric conversion units (photodiodes). A plurality of pixels provided with these two photoelectric conversion units (photodiodes) are regularly arranged in a constant pattern on the light receiving surface of the image sensor 107, or if possible, all the pixels are two photoelectric conversion units. It may be a configuration including.

In the imaging surface phase difference method, a pair of images (hereinafter, may be referred to as A image and B image) formed by light flux passing through two different regions (partial pupils) in the exit pupil of the camera optical system can be obtained. .. Therefore, the image pickup device 101 of the present embodiment obtains the parallax from the A image and the B image, and is specific on the object based on the principle of triangulation from the parallax, like the stereo cameras of the first and third embodiments. The imaging distance of the measurement point can be measured.

The image sensor 107 of FIG. 19 is, for example, a CMOS image sensor including the above two photoelectric conversion units, and each photoelectric conversion unit can obtain image data having, for example, an 8-bit bit depth. The input / output of the captured image data can be performed by, for example, the LVDS method, and two image data of the A image and the B image are output to the outside of the element. The image pickup optical system in which the image pickup element 107 and the lens 109 are combined is configured to have a shooting field of about twice the size of the component, for example, the diameter in the horizontal direction and the vertical direction of the captured image data. The image sensor 107 reads the image data after writing the set value to the built-in register via communication such as SPI or I2C. Further, the image pickup device 107 can receive a shooting trigger issued inside or outside the device and start capturing an image.

After the light-shade image acquisition lighting device 102 and the distance image acquisition lighting device 103 are turned on, the image pickup device 101 captures an image by an imaging command issued from the controller 106. Further, the image sensor 107 can perform thinning imaging and binning imaging by writing a set value to a register.

The shading image generation unit 110 and the distance image generation unit 111 generate a shading image and a distance image and transfer them to the image processing unit 104. The processing of the grayscale image generation unit 110 is the same as that of the first and third embodiments, and the grayscale image is generated in the same manner as in the first embodiment as shown in FIG. On the other hand, the distance image generation unit 111 can handle the two image data of the A image and the B image in the same manner as in the case of the stereo imaging of the first embodiment, and has the distance information from the parallax of the A image and the B image. Can be generated. Similar to the first embodiment, the distance image generated by the distance image generation unit 111 is represented by a brightness value that is brighter as the distance from the image pickup apparatus 101 is longer and darker as the distance is shorter, for example, as shown in FIG. Can be.

The subsequent image inspection of the component 105 by image processing can be performed in the same manner as described above. For example, the operation of the production equipment arranged on the production line handling the component 105, for example, the transfer robot 116, is controlled according to the quality determination result of the component 105. be able to. For example, the transfer robot 116 can be operated based on the command content commanded by the controller 106 based on the pass / fail determination result. For example, if the inspection result of the part is a non-defective product, the transfer robot grasps the part and conveys it to a subsequent process, and if the inspection result of the part is a defective product, the system is stopped and a warning sound is generated.

According to the present embodiment, an image pickup device can be configured with a set of image pickup elements, a wavelength filter, and a lens, and with a simple and inexpensive configuration, it is possible to acquire a shade image and a distance image in appropriate lighting modes by one imaging. Therefore, for example, it is possible to stably and quickly inspect the quality of parts in an environment where a complicated background is reflected in the camera.

<Embodiment 5>
In the above-described first embodiment, the image pickup device 101, the shading image acquisition lighting device 102, the distance image acquisition lighting device 103, and the image processing unit 104 are separately configured, and the installation procedure tends to be complicated. In the present embodiment, a configuration in which these constituent members are provided as an integrated unit is shown.

FIG. 20 shows the configuration of the image processing apparatus of this embodiment. As shown in the figure, this image processing device includes an imaging device 101, a shading image acquisition lighting device 102, a distance image acquisition lighting device 103, and an inspection unit 117 in which an image processing unit 104 is housed in an integrated housing. The functions of these individual components are equivalent to those of the embodiments described above. An inspection system is configured by connecting the inspection unit 117 to the controller 106.

The positions of the image pickup device 101, the shading image acquisition lighting device 102, the distance image acquisition lighting device 103, and the image processing unit 104 are fixed to the housing of the inspection unit 117. However, the shading image acquisition lighting device 102, the distance image acquisition lighting device 103, and the like may each be provided with an adjustment mechanism for making the position and angle adjustable. Further, the shading image acquisition lighting device 102 and the distance image acquisition lighting device 103 may be detachably attached to each other so that the type, size, and emission wavelength of the lighting device can be changed.

The overall flow of the process of inspecting the parts is the same as that described so far with reference to FIG. Next, the details of each component will be described.

In the present embodiment, the following configuration is adopted especially for the lighting device so that each part can be housed in an integrated housing.

In the present embodiment, the shading image acquisition lighting device 102 shown in the second embodiment in the distance image generation unit 111 is used as the distance image acquisition lighting device 103 in the first and third embodiments, for example, the embodiment shown in FIG. It is an LED lighting device having. In the present embodiment, since the projector as in the second embodiment is used for the distance image acquisition lighting device 103 as described below, the parallel light illumination as in the first and third embodiments is not necessarily required. Conversely, also in the second embodiment, it is conceivable to use, for example, the LED lighting device as shown in FIG. 4 for the light and shade image acquisition lighting device 102. In the present embodiment, the emission wavelength of the shading image acquisition lighting device 102 is, for example, blue (B). When inspecting a component, the shading image acquisition lighting device 102 illuminates the component with a predetermined brightness based on a lighting control command from the controller 106 prior to imaging.

The distance image acquisition lighting device 103 has a projector configuration equipped with, for example, a known DMD device as shown in the second embodiment. The illumination mode of the projector is as described with reference to FIGS. 14 and 22 in the second embodiment, and the feature points of the stereo association can be obtained by projecting a random texture on the surface of the component. In the present embodiment, the emission wavelength of the distance image acquisition lighting device 103 is red (R). When inspecting a component, the distance image acquisition lighting device 103 illuminates the component with a predetermined brightness based on a lighting control command from the controller 106 prior to imaging. At the time of imaging, the parts can be illuminated by both the shading image acquisition lighting device 102 and the distance image acquisition lighting device 103 in order to acquire the shading image and the distance image in one imaging.

Since the actual inspection procedure is the same as that of the second embodiment using the lighting device 103 for acquiring a distance image by a projector, a duplicate description will be omitted here. According to this embodiment, the image processing device can have a simple system configuration, and the quality of parts can be determined based on the shading image and the distance image acquired in the appropriate lighting modes as in the above-described embodiment. Image inspection can be performed stably and at high speed. Further, the imaging device and the lighting device are configured as an integrated inspection unit 117, and the system can be easily installed in a short time simply by connecting the inspection unit 117 to the controller 106.

<Embodiment 6>
In the above-described first embodiment, the final inspection process is performed on the two-dimensional shading image (first image) and on the shading image (first image) by the distance image (second image). An example of determining the inspection area for image inspection is shown.

However, depending on the specific image inspection method or the form of the parts of the object, the depth information of the object being imaged may be important information for inspection, and the distance image (second image). There may also be a demand for performing image inspection processing using. In addition, there may be a demand for using both a shading image (first image) and a distance image (second image) for the image inspection process.

Generally speaking, the image processing dealt with in the present specification uses the distance information of the object acquired from the grayscale image (first image) and / or the distance image (second image), and the grayscale image (first image). Image) and / or distance image (second image) may determine the inspection area.

Therefore, in the following, as an example, an example will be described in which the inspection area specifying unit 112 uses a shading image and then the inspection processing unit 113 performs image inspection processing on the specified inspection area using a distance image.

The configuration of the image inspection system according to the present embodiment may be the same as that shown in FIG. 1, for example. The flow of the process of actually inspecting the parts in the present embodiment (FIG. 23) is as follows.

First, both the shading image acquisition lighting device 102 and the distance image acquisition lighting device 103 are turned on by the lighting control command of the controller (S10 in FIG. 23). In this state, an image is imaged by the two image pickup devices 101 according to the image pickup command of the controller, and two image data are obtained (S12 in FIG. 23). At this time, for example, the shading image acquisition illuminating device 102 irradiates the blue (B) illumination light, and the distance image acquisition illuminating device 103 irradiates the red (R) illumination light.

Similar to the first embodiment, the image pickup apparatus 101 generates a shade image and a distance image based on the two image data captured in stereo (S14 in FIG. 23), and transfers the shade image and the distance image to the image processing unit 104. .. In the generation of the grayscale image (first image generation step), the grayscale image is generated by using the pixels corresponding to the filter through which the blue color (B) is transmitted among the wavelength filters 108 of the image pickup apparatus 101. In the generation of the distance image (second image generation step), the distance image is generated using the pixels corresponding to the filter through which the red color (R) is transmitted.

In the present embodiment, in the image processing unit 104, the inspection area specifying unit 112 specifies the inspection area based on the image information of the shading image (S16 in FIG. 23). Further, the inspection processing unit 113 performs an image inspection only on the inspection area of the distance image specified by the inspection area specifying unit 112 (S18 in FIG. 23), and determines the quality of the component. The result of the image inspection is transferred to the controller 106, and the controller 106 controls the production equipment such as the transfer robot 116 based on the determination result (S20 in FIG. 23).

By the above processing, the shading distance image and the distance image can be acquired by one imaging, and the image inspection, for example, the quality judgment of the part can be performed stably and at high speed by using the shading image and the distance image. can do.

Subsequently, the operation of each component of the image processing apparatus will be described in detail. The component 105, the controller 106, the shading image acquisition lighting device 102, the distance image acquisition lighting device 103, and the image pickup device 101 in this embodiment have the same configurations as those in the first embodiment.

The image processing unit 104 is composed of an inspection area specifying unit 112 and an inspection processing unit 113, and executes the inspection area identification and inspection processing using the grayscale image and the distance image input from the image pickup apparatus 101. As described above, the actual image processing unit 104 can be mounted by, for example, a PC equipped with a CPU, a board computer, an arithmetic unit such as a GPU or FPGA.

The inspection area specifying unit 112 of the present embodiment specifies an image area having a rectangular shape as an inspection area based on the shade image generated by the shade image generation unit 110. Here, it is assumed that an image inspection of the component 105 is performed on an object composed of the component 105 and an object around the component 105 as shown in FIG. In that case, the inspection area can be determined by detecting or specifying the rectangular shape of the upper surface of the stand 114 on which the component 105 is mounted from the shade image.

If the rectangular shape of the upper surface of the stand 114 is to be detected, for example, first, a plurality of linear edge components of a shading image (FIG. 9: 301) are detected. Next, based on the detected linear edge component, a known rectangular shape detection process is performed to detect the rectangular shape in the grayscale image. As a result, it is possible to generate a binary image as shown in FIG. 11, in which the inside of the rectangle is "1" and the outside of the rectangle is "0". Before detecting the linear edge component from the shading image, a known filter for reducing noise may be applied. The binary image obtained in this way is hereinafter referred to as a "mask image" and is used as an inspection area for image processing in the subsequent stage.

The inspection processing unit 113 can perform an image inspection using the mask image and the distance image generated by the inspection area specifying unit 112 from the shading image as described above. First, the distance image is masked based on the mask image generated by the inspection area specifying unit 112. Specifically, the brightness value of the distance image is adopted as it is for the pixel having the brightness value of "1" in the mask image, and the brightness value is set to 0 in the pixel having the brightness value of "0" in the mask image.

By such a masking process, a distance image in which only the parts are present can be obtained as shown in FIG. In FIG. 21, 1301 is a region corresponding to the brightness value “0” of the mask image, and is a portion corresponding to the background around the stand 114. Reference numeral 1303 is a region corresponding to the brightness value “1” of the mask image, and is a portion corresponding to a rectangular shape on the upper surface of the stand 114. 1302 corresponds to the distance image of the component 105.

An image inspection can be performed on a distance image (FIG. 21) in which only the component 105 thus generated exists. As an image inspection using a distance image, for example, as a quality inspection of a part 105, a process of determining whether or not the height of the part 105, that is, the thickness corresponding to the depth direction of the distance image is a certain size or more. And so on.

In such an inspection of the height of the component 105, for example, a threshold value process based on the distance information is performed on the distance image (FIG. 21) in which only the component 105 exists, and the distance image is binarized.

Here, for example, the height of the stand 114 from the gantry 115 is 100 mm, the height (thickness) of the specific portion on the upper surface of the component 105 is 20 mm, and the height (thickness) of the specific portion is 10 mm or less. Part 105 shall be determined to be defective. In that case, for example, the threshold value is set to an imaging distance corresponding to a height of 110 mm from the gantry 115, and the imaging distance corresponding to the height of the upper surface of the component 105 being imaged is compared with the threshold value to finish the component 105. The dimensions can be determined.

Then, for example, by the above threshold value determination, a binary image is acquired in which the brightness value of the pixel having a height of more than 110 mm from the gantry 115 is "1" and the brightness value of the pixel having a height of 110 mm or less is "0". In this case, for example, the area of the pixel value of "1" (for example, the number of pixel values of "1") is obtained from the generated binary image, and if the area is larger than the area threshold value, it is determined as a non-defective product, and the area threshold value is determined. If it is as follows, it is possible to make a quality judgment such that it is judged as a defective product. The quality determination result obtained in this way is output to the controller 106.

The transfer robot 116 can be operated based on the command content commanded by the controller based on the quality determination result. Specifically, if the inspection result of the part is non-defective, the transfer robot grabs the part and transfers it to the subsequent process, and if the inspection result of the part is defective, the system is stopped and a warning sound is generated. It is possible to control devices such as.

As described above, according to the present embodiment, even when performing an image inspection using distance information, it is possible to acquire a shade image and a distance image in an appropriate lighting environment by one imaging. Therefore, even in an environment where a complicated background is reflected in the camera, the background can be reliably separated, and image inspection such as quality judgment of parts can be inspected stably and at high speed.

Although the case where the robot device uses an articulated robot arm having a plurality of joints has been described, the drive source of the joints is not limited to the servomotor, and may be, for example, an artificial muscle. Further, the various embodiments described above are for machines capable of automatically performing expansion / contraction, bending / stretching, up / down movement, left / right movement, turning operation, or a combined operation thereof based on information of a storage device provided in the control device. Applicable.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. But it is feasible. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101 ... image pickup device, 102 ... shading image acquisition lighting device, 103 ... distance image acquisition lighting device, 104 ... image processing unit, 105 ... parts, 106 ... controller, 107 ... image sensor, 108 ... wavelength filter, 109 ... lens , 110 ... shade image generation unit, 111 ... distance image generation unit, 112 ... inspection area identification unit, 113 ... inspection processing unit, 114 ... stand, 115 ... stand, 116 ... transfer robot, 117 ... inspection unit.

Claims (22)

A first illuminating device that irradiates a first light including a first wavelength,
A second illuminating device that irradiates a second light containing a second wavelength different from the first wavelength,
An imaging device that captures an object and
A controller that uses the image pickup device to image the object in a state where the object is irradiated with the first light and the second light from the first lighting device and the second lighting device. An image processing device equipped. In the image processing apparatus according to claim 1,
A first image generation unit that generates a first image captured through a first optical filter that transmits the first wavelength from the output of the image pickup apparatus.
An image processing device including a second image generation unit that generates a second image captured through a second optical filter that transmits the second wavelength from the output of the image pickup device. In the image processing apparatus according to claim 2,
The inspection area of the first image and / or the second image is determined using the distance information of the object obtained from the second image and / or the first image, and the inspection area is determined. An image processing device including an image processing unit that inspects the object by performing image processing. In the image processing device according to claim 2 or 3, the image pickup device has passed through the first pixel data captured by the image pickup light that has passed through the first optical filter or the second optical filter. An image processing device including a readout mechanism that outputs second pixel data captured by the imaging light. In the image processing apparatus according to any one of claims 2 to 4, the first optical filter and the second optical filter are a filter that transmits red, a filter that transmits green, and a filter that transmits blue. An image processing device including one of a filter and a filter that transmits infrared rays, which are different from each other. In the image processing device according to any one of claims 1 to 3, the image pickup device is a stereo camera, and the distance information of the object is obtained from the image data stereo-captured by the image pickup device as the second image. Image processing device to be acquired. In the image processing apparatus according to any one of claims 2 to 6, the image pickup device of the image pickup device includes an image pickup surface phase difference sensor including two photoelectric conversion units, and the image pickup light including the second wavelength. An image processing device that acquires distance information of the object from image data captured by the image pickup surface phase difference sensor as the second image. In the image processing apparatus according to claim 6 or 7, the image processing in which the first illumination device irradiates the object with the first light by illumination light parallel to the imaging optical axis of the imaging apparatus. apparatus. In the image processing device according to any one of claims 6 to 8, the second lighting device is a lighting device that irradiates the object with the second light from the surroundings of the object. Image processing device. The image processing device according to any one of claims 1 to 5, wherein the second lighting device is a projector that irradiates the object with pattern light having a predetermined light / dark pattern. The image processing device according to claim 10, wherein the distance information of the object is acquired from the image of the pattern light of the image data captured by the image pickup device as the second image. The image processing device according to any one of claims 1 to 11, wherein the lighting light source of the first lighting device and / or the second lighting device is an LED element. The image processing device according to any one of claims 1 to 11, wherein the lighting light source of the first lighting device and / or the second lighting device is a laser light source. In the image processing apparatus according to any one of claims 1 to 13, the image processing unit uses the first image and the second image in image processing for the inspection area to perform the object. Image processing device to inspect. The image processing device according to any one of claims 1 to 14, a robot device, and a control device that controls the robot device and causes the robot device to operate parts to execute manufacturing of an article. In the robot system
A robot system in which the control device controls the robot device to manufacture an article according to an inspection result in which the image processing device inspects the component as an object. In the robot system according to claim 15,
When the robot device operates the component as the object, the controller uses the first lighting device and the second lighting device to emit the first light and the first light to the component. A robot system that irradiates the light of 2 and images the component using the image pickup device. The image processing device according to any one of claims 1 to 14, a robot device, and a control device that controls the robot device and causes the robot device to operate parts to execute manufacturing of an article. In the manufacturing method of goods, which manufactures goods by a robot system
A method for manufacturing an article in which the control device controls the robot device to manufacture the article according to an inspection result in which the image processing device inspects the part as the object. In a control method of an image processing device that performs image processing on an image obtained by capturing an object with an image pickup device,
A state in which the first illuminating device and the second illuminating device are used to irradiate the first light including the first wavelength and the second light including the second wavelength different from the first wavelength. A control method comprising an imaging step of imaging the object with the imaging apparatus. In the control method according to claim 18,
A first image generation step of generating a first image captured through a first optical filter that transmits the first wavelength from the output of the image pickup apparatus.
A control method including a second image generation step of generating a second image captured through a second optical filter that transmits the second wavelength from the output of the image pickup apparatus. In the control method according to claim 19,
The inspection area of the first image and / or the second image is determined by using the distance information of the object obtained from the second image and / or the first image, and the inspection area is determined. A control method including an inspection step of inspecting the object by performing the image processing. A control program for causing a computer to execute each step in the method for manufacturing an article according to claim 17 or the control method according to any one of claims 18 to 20. A computer-readable recording medium containing the control program according to claim 21.

Images