JP7474636B2 - Image Inspection System - Google Patents

Description

The present invention relates to an image inspection system that inspects, for example, imaged workpieces.

Conventionally, as disclosed in, for example, Patent Document 1, image processing devices that use a pattern projection method to obtain height information of an object to be inspected, such as a workpiece, are known. In the image processing device disclosed in Patent Document 1, a camera obtains multiple luminance images while sequentially irradiating the object to be inspected with multiple pattern projection lights, and the multiple luminance images obtained by the camera are transferred to a controller, which generates height data. The controller performs a process of converting height data within a predetermined height range from the generated height data into a height image with fewer gradations than the height data, i.e., a height extraction process. After the height extraction process, the controller performs image processing using the height images to determine the quality of the object to be inspected, and outputs a determination signal to an external device such as a PLC.

JP 2019-168285 A

It is also possible to connect a setting device consisting of a general-purpose personal computer or the like to the controller, and to perform various settings on the setting device for the height data output from the controller. In this case, the setting device first converts the height data output from the controller into an image with fewer gradations than the height data. During image inspection, the user sets the target area (line area or surface area) to be the subject of image inspection on the height image, but this height image at the time of setting is an image of the inspection target viewed from directly above with the viewpoint fixed at the camera position, so there is a problem in terms of usability in that it is difficult for the user to grasp the three-dimensional effect and height difference of the inspection target.

Furthermore, in height images displayed on a monitor, it is common to express height differences in the object being inspected by applying an artificially generated surface color that differs from the actual surface color of the object being inspected. When this type of expression is adopted, large changes in height can be recognized by the change in color, but in areas such as small steps, there is almost no difference in surface color, making it difficult to grasp the height difference.

In some cases, a two-dimensional image of the object being inspected, captured simultaneously with the height image, is synthesized and displayed with the artificially colored height image. While this makes it easier to grasp the relationship between the parts of the object being inspected and their height, it also makes it more difficult to grasp the height difference, since the color indicating the height difference is mixed with the actual surface color of the object being inspected.

As described above, when it becomes difficult to grasp the height difference on the height image, when a user sets a target area for image inspection, it becomes difficult for the user to grasp on the height image what kind of target area will be set for the object to be inspected.

The present invention was made in consideration of these points, and its purpose is to make it easier to grasp the target area that is the subject of image inspection, thereby improving usability.

To achieve the above object, the first disclosure can be premised on an image inspection system including an image inspection imaging device having an acquisition unit that acquires height data in which height information of an object to be inspected is arranged on a two-dimensional coordinate system, and an inspection setting unit that performs various inspection settings based on user operations in a setting device connected to the image inspection imaging device via a network.

The inspection setting unit may have an object generating means for generating a three-dimensional object that stereoscopically represents the object to be inspected based on the height data acquired by the acquisition unit, a display means for displaying the three-dimensional object generated by the object generating means on a monitor of the setting device, an attitude adjusting means for adjusting the attitude of the three-dimensional object displayed by the display means, an area designation receiving means for receiving designation of a target area to be subject to image inspection on the three-dimensional object whose attitude has been adjusted by the attitude adjusting means, and an area setting means for setting the target area designated by the area designation receiving means to the height data acquired by the acquisition unit.

According to this configuration, when the acquisition unit acquires height data in which height information of the inspection object is arranged on two-dimensional coordinates, the object generation means generates a three-dimensional object based on the height data. The three-dimensional object can be displayed on the monitor of the setting device in a form such as a three-dimensional polygon display or a three-dimensional point cloud display. Since the posture of the three-dimensional object can be adjusted while it is displayed on the monitor, a three-dimensional object in which the inspection object is viewed from a desired direction can be displayed on the monitor. The user can specify a target area on the three-dimensional object after the posture has been adjusted.

In other words, the user can view the three-dimensional object that represents the object to be inspected in a three-dimensional manner from any direction, so that the user can easily and reliably grasp the height difference even in areas such as small steps. Furthermore, since the target area can be specified while grasping the height difference of the object to be inspected, the user can easily grasp where on the object to be inspected and what size or shape the target area has been specified. The target area can be, for example, a linear area or an area enclosed by a figure, and as long as it is an area with a certain range, its shape and size can be freely specified.

When a target area is specified, the target area is set to the height data acquired by the acquisition unit. In other words, the information on the target area specified on the three-dimensional object can be reflected in the height data acquired during operation, making it possible to perform image inspection on the desired area.

In a second disclosure, the acquisition unit may include an imaging element that captures an image of an object to be inspected. The inspection setting unit may include an element generation means that generates a viewpoint identification element for identifying a viewpoint at the position of the imaging element when the area designation receiving means accepts a designation of an object area. The display means can display the viewpoint identification element generated by the element generation means on the monitor by superimposing it on the three-dimensional object.

With this configuration, when a user specifies a target area to be inspected using images, a viewpoint-specifying element is displayed superimposed on the three-dimensional object. This allows the user to specify the position of the imaging element that captured the object to be inspected, i.e., the position of the viewpoint.

In a third disclosure, the viewpoint identification element may include a plurality of line segments extending a predetermined length along the imaging axis of the imaging element from each of the plurality of points designated by the area designation receiving means, and a graphic element consisting of a plane or curved surface formed based on the plurality of line segments.

With this configuration, when the user specifies multiple points when specifying a target area, a surface formed by multiple line segments extending along the imaging axis is displayed superimposed on the three-dimensional object. This makes it easy to identify the position of the viewpoint.

In the fourth disclosure, the graphic element may be semi-transparent. This configuration makes it possible to visually recognize three-dimensional objects behind the graphic element, making it easier to specify a target area.

In the fifth disclosure, the predetermined length of the multiple line segments forming the graphic element can be determined according to the maximum height and minimum height in the height image.

In other words, if the height dimension of a graphic element or line segment is unnecessarily long, it may be difficult to see the height image, and if the height dimension of a graphic element or line segment is too short, it may be impossible to confirm the graphic element or line segment. With this disclosure, the height dimension of a graphic element or line segment will not be unnecessarily long or too short, making it easier to identify the position of the viewpoint.

In a sixth disclosure, the viewpoint specifying element may be capable of being adjusted in position on the three-dimensional object after being generated by the element generating means.

With this configuration, if you want to change the position of a viewpoint-specific element after it has been generated, you can place it in the desired position by moving the viewpoint-specific element. As the viewpoint-specific element moves, the target area also moves to a different area.

In the seventh disclosure, the viewpoint-specific element can have its shape changed on the three-dimensional object after it is generated by the element generation means.

With this configuration, after the viewpoint-specific element is generated, its shape can be changed to any shape. The shape of the target region is also changed in conjunction with the change in the shape of the viewpoint-specific element.

In the eighth disclosure, the viewpoint determining element may be prohibited from moving in the height direction of the three-dimensional object. For example, if the Z direction is the height direction in an XYZ coordinate system, the Z coordinate of the viewpoint determining element can be fixed, and the viewpoint determining element can be made movable only in the XY directions.

In other words, because the viewpoint identification element is an element that is displayed when setting the target area to be subject to image inspection to height data arranged on two-dimensional coordinates, it only needs to be moved in the X or Y direction of the three-dimensional object, and there is no need to move it in the direction in which the viewpoint is located (the height direction of the three-dimensional object); if the viewpoint identification element were movable in the height direction of the three-dimensional object, it is thought that the operation of setting the target area would become cumbersome. In the present disclosure, the viewpoint identification element can be moved only in the X or Y direction of the three-dimensional object, making the operation of setting the target area easier.

In the ninth disclosure, when the area designation receiving means receives designation of a start point and an end point on the three-dimensional object, it can set the target area at coordinates corresponding to the position of a line segment extending from the start point to the end point in the two-dimensional coordinates of the height data.

With this configuration, the user can simply specify the start and end points on a three-dimensional object, and the target area including the start and end points can be automatically set as height data.

In the tenth disclosure, the acquisition unit can generate a height image based on the acquired height data. The display means is configured to be able to switch between displaying on the monitor the height image generated by the acquisition unit and the three-dimensional object generated by the object generation means. The area designation receiving means can receive designation of a target area to be subject to image inspection on the height image generated by the acquisition unit.

For example, it may be easier to specify the target area for image inspection using a height image than using a three-dimensional object. In this case, the height image can be displayed on a monitor and the target area for image inspection can be specified on the height image as well.

In the eleventh disclosure, the setting device includes an image processing unit that performs image processing on the target area set by the area setting means for the height data, and an inspection unit that performs an appearance inspection based on the image processed by the image processing unit. This makes it possible to perform various appearance inspections, such as measuring steps and scratches.

In the twelfth disclosure, the setting device is configured to acquire from the outside a file describing a setting item consisting of a target area accepted by the area designation accepting means and register information indicating a location where the setting value of the setting item is stored, and transmit the setting value of the setting item set by the user and the register information corresponding to the setting item contained in the file to the image inspection imaging device. The image inspection imaging device can store the setting value of the setting item received from the setting device in a location indicated by the register information corresponding to the setting item.

According to this configuration, if both the imaging device and the setting device comply with a common standard, the setting device acquires a file that describes the setting items and register information indicating the location where the setting values of the setting items are stored. For example, if the standard is the GenICam standard, this file is a Device XML file. This file may be acquired by the setting device from the imaging device, or may be downloaded from a website or the like and specified as a separate file to be referenced by GenApi.

When a user sets a target area for image inspection, the coordinates and size of the target area for image inspection become the setting values, and the setting item of the target area, its coordinates and size, together with register information, are sent to the imaging device. This allows the settings to be made in the imaging device, so that if the imaging device complies with standardization specifications, the above settings can be freely changed in the setting device, improving the freedom of selection of imaging device models.

As described above, according to the present disclosure, it is possible to specify the target area to be subjected to image inspection while grasping the height difference of the inspection object by adjusting the posture of the three-dimensional object that stereoscopically represents the inspection object, thereby enabling the user to easily grasp the target area, thereby improving usability.

1 is a diagram illustrating a schematic configuration and an operational state of an image inspection system according to a first embodiment of the present invention; FIG. 10 is a view corresponding to FIG. 1 according to a second embodiment. FIG. 10 is a view corresponding to FIG. 1 according to a third embodiment. FIG. 10 is a view corresponding to FIG. 1 according to a fourth embodiment. 1 is a flowchart showing a procedure for generating a height image based on the principle of deflectometry. FIG. 11 is a view corresponding to FIG. 1 according to a fifth embodiment in which a height image is generated using a photometric stereo method. FIG. 2 is a diagram illustrating a connection interface between an image inspection application and a camera. 10 is a flowchart conceptually illustrating an example of internal processing of the imaging device. FIG. 13 is a diagram showing a parameter set for generating an inspection image using the principle of deflectometry. FIG. 13 is a diagram showing a parameter set when generating an inspection image by multispectral imaging. FIG. 4 is a block diagram of an inspection setting unit. 13 is a flowchart illustrating an example of a procedure for creating a new target region. FIG. 13 is a conceptual diagram of a 3D hit determination. 13A and 13B are diagrams illustrating a case where a new linear target region is created on a three-dimensional display or a two-dimensional display. 13A and 13B are diagrams illustrating a case where a new rectangular target region is created on a three-dimensional display or a two-dimensional display. 13 is a flow chart showing a first stage of a procedure for modifying a region of interest. 13 is a flow chart showing a second stage of the target region modification procedure. FIG. 13 is a conceptual diagram of a 3D hit determination. FIG. 13 is a conceptual diagram of 2D hit determination. FIG. 13 is a diagram showing a specific example 1 of adjusting the position of a linear target region. FIG. 11 is a diagram showing a specific example 2 of adjusting the position of a linear target region. 13A and 13B are diagrams showing a specific example of adjusting the position of a rectangular target area. 13 is a flowchart showing an example of a procedure for measuring a step in a linear target area during operation. 13 is a flowchart showing an example of a procedure for inspecting a rectangular target area for flaws during operation. 13 is a flowchart illustrating an example of a procedure for inspecting height in a rectangular target area during operation. 13 is a flowchart illustrating an example of a procedure for inspecting an expiration date in an arc-shaped target area during operation. 13 is a flowchart showing an example of a procedure for determining engraving in an arc-shaped target area during operation.

The following describes in detail an embodiment of the present invention with reference to the drawings. Note that the following description of the preferred embodiment is essentially merely an example and is not intended to limit the present invention, its applications, or its uses.

(Embodiment 1)
FIG. 1 is a schematic diagram showing an operation state of an image inspection system 1 according to an embodiment of the present invention. The image inspection system 1 is configured to inspect the presence or absence of defects in a workpiece W, which is an inspection object, using an image of the workpiece W captured, and includes an illumination device 2 for illuminating the workpiece W, an imaging unit 3 for capturing an image of the workpiece W, and a setting device 4 configured by a personal computer (PC) or the like for setting the imaging unit 3. The workpiece W can also be called an imaging object, since it is an object to be captured by the imaging unit 3. The illumination device 2 and the imaging unit 3 constitute an acquisition unit A for acquiring height data. The imaging device 300 for image inspection, which constitutes a part of the image inspection system 1, is a device including the acquisition unit A. The imaging device 300 and the setting device 4 may be separate, and the imaging device 300 alone may be an embodiment of the present invention.

The setting device 4 includes a display unit 5, a keyboard 6, and a mouse 7. The display unit 5 includes a monitor such as an organic EL display or a liquid crystal display, and is a portion capable of displaying images captured by the imaging device 300, images after various processing of the images captured by the imaging device 300, various user interface images (GUI), etc. The various user interface images, etc. are generated in the main body of the setting device 4. The horizontal direction of the display unit 5 can be the X direction of the display unit 5, and the vertical direction of the display unit 5 can be the Y direction of the display unit 5.

The keyboard 6 and mouse 7 are conventionally known devices for operating a computer. By operating the keyboard 6 or mouse 7, various information can be input to the setting device 4, various settings can be made, and images and the like displayed on the display unit 5 can be selected. Note that instead of or in addition to the keyboard 6 and mouse 7, devices for operating a computer, such as a voice input device or a pressure-sensitive touch panel, can also be used.

Figure 1 shows a case where multiple workpieces W are placed on the upper surface of the transport belt conveyor B and transported in the direction indicated by the white arrow in Figure 1. An external control device 8 consisting of a programmable logic controller (PLC) or the like is connected to the setting device 4 as a device for sequence control of the transport belt conveyor B and the image inspection system 1. This external control device 8 may be configured as part of the image inspection system 1. In addition, the external control device 8 does not have to be a component of the image inspection system 1.

In the description of this embodiment, the direction in which the workpiece W is transported by the transport belt conveyor B (the direction in which the workpiece W moves) is defined as the Y direction, the direction perpendicular to the Y direction in a plan view of the transport belt conveyor B is defined as the X direction, and the direction perpendicular to the X direction and the Y direction (the direction perpendicular to the upper surface of the transport belt conveyor B) is defined as the Z direction, but this definition is provided merely for the convenience of description.

The image inspection system 1 can be used for visual inspection of the workpiece W, that is, for inspecting the surface of the workpiece W for defects such as scratches, dirt, and dents, and can also determine the quality of the workpiece W from the inspection results. During operation, the image inspection system 1 receives an inspection start trigger signal that specifies the start timing of a defect inspection (quality determination inspection) from the external control device 8 via a signal line. The image inspection system 1 performs imaging and illumination of the workpiece W based on this inspection start trigger signal, and obtains an inspection image after predetermined processing. Then, an appearance inspection is performed based on the inspection image, and the inspection result is transmitted to the external control device 8 via a signal line. In this way, during operation of the image inspection system 1, the input of the inspection start trigger signal and the output of the inspection result are repeatedly performed between the image inspection system 1 and the external control device 8 via a signal line. Note that the input of the inspection start trigger signal and the output of the inspection result may be performed via a signal line between the image inspection system 1 and the external control device 8 as described above, or may be performed via other signal lines not shown. For example, a sensor for detecting the arrival of the workpiece W may be directly connected to the image inspection system 1, and an inspection start trigger signal may be input from the sensor to the image inspection system 1. The image inspection system 1 may also be configured to automatically generate a trigger signal internally to perform the inspection.

In addition to being configured with dedicated hardware, the image inspection system 1 may also be configured with software installed on general-purpose equipment, such as a general-purpose or dedicated computer with an image inspection program installed. For example, it may be configured with an image inspection program installed on a dedicated computer with hardware such as a graphics board specialized for image inspection processing.

(Configuration of Acquisition Unit A)
The acquisition unit A is a part configured to acquire a luminance image according to the amount of reflected light received from the workpiece W, and to acquire height data in which height information of the workpiece W is arranged on a two-dimensional coordinate system. The luminance image can be acquired by imaging the workpiece W illuminated by the lighting device 2 with the imaging unit 3. When illumination of the workpiece W is not required, the luminance image can be acquired by imaging the workpiece W with the imaging unit 3 without illumination by the lighting device 2.

The method of acquiring height data in which the height information of the workpiece W is arranged on two-dimensional coordinates is not particularly limited, and any method may be used, and hardware and software can be appropriately selected according to the method. The acquisition unit A can generate a three-dimensional image by acquiring the height data.

That is, there are broadly two methods for acquiring height data: one is a passive method (passive measurement method) in which height data is acquired using an image captured under lighting conditions for obtaining a normal luminance image, and the other is an active method (active measurement method) in which height data is acquired by actively irradiating light to measure the height direction. A typical passive method is the stereo measurement method. In this method, two imaging units 3 are prepared, these two imaging units 3 are arranged in a predetermined positional relationship, and each performs imaging, and height data can be acquired by performing well-known calculations based on the two generated images.

Typical active methods are the light section method and the pattern projection method. In the light section method, in the stereo measurement method described above, one of the imaging units 3 is replaced with a laser projector, a line of laser light is projected onto the workpiece W, and the three-dimensional shape of the workpiece W is restored based on the degree of distortion of the image of the line of light according to the shape of the surface of the workpiece W. The light section method allows for stable measurement because it does not require the determination of corresponding points.

The pattern projection method is a method in which multiple images are captured by shifting the shape, phase, etc. of a specified projection pattern projected onto the workpiece W, and the three-dimensional shape of the workpiece W is restored by analyzing the multiple captured images. There are several types of pattern projection methods, including a phase shift method in which the phase of a sine wave stripe pattern is shifted to capture multiple images (at least three or more), the phase of the sine wave is calculated for each pixel from the multiple images, and the three-dimensional coordinates on the work surface are calculated using the phase obtained; a moire projection method in which a three-dimensional shape is restored using a type of spatial frequency undulation phenomenon that occurs when two regular patterns are combined; a spatial coding method in which the pattern itself projected onto the work W is changed for each shot, for example, a stripe pattern with a black-and-white duty ratio of 50% and a stripe width that becomes thinner from half the screen, one-quarter, one-eighth, etc. is projected sequentially, and the pattern projection image is taken with each pattern to determine the absolute phase of the height of the work W; and a multi-slit method in which multiple thin line pattern illuminations (multi-slits) are projected onto the work, the pattern is moved at a pitch narrower than the slit period, and multiple shots are taken.

Although the height data can be obtained by combining the above-mentioned phase shift method and spatial coding method, this is not limiting and the height data can be obtained by other methods. In addition, any conceivable method for obtaining height data can be adopted, such as a method other than the above-mentioned methods, such as an optical radar method (time of flight), a focusing method, a confocal method, or a white light interferometry method.

The lighting device 2 and image capture unit 3 are configured to realize the pattern projection method. The specific configurations of the lighting device 2 and image capture unit 3 are described below.

(Configuration of Illumination Device 2)
First, the illumination device 2 will be described. The illumination device 2 is a light projection unit that projects structured illumination having a predetermined projection pattern onto the workpiece W, and includes a light emitting unit 2a and an illumination control unit 2b that controls the light emitting unit 2a. The light emitting unit 2a and the illumination control unit 2b may be separate or integrated. The illumination control unit 2b may be incorporated into the setting device 4. The light emitting unit 2a may be configured, for example, with a light emitting diode, a projector using a liquid crystal panel, an organic EL panel, a digital micromirror device (DMD), or the like, and may also be called an illumination unit. Although the light emitting diode, the liquid crystal panel, the organic EL panel, and the DMD are not shown, those having a conventionally known structure may be used. The illumination device 2 is connected to the setting device 4 via a signal line 100a, and can be installed away from the imaging unit 3 and the setting device 4.

The lighting device 2 of the first embodiment is configured to be capable of uniform surface emission. The lighting device 2 is also configured to be capable of performing illumination capable of performing deflectometry processing, as an example of structured illumination having a predetermined projection pattern, and therefore has a light-emitting unit 2a that irradiates the workpiece W with pattern light having a periodic illuminance distribution. In other words, the lighting device 2 can be a pattern light illumination unit that performs pattern light illumination that sequentially irradiates the workpiece W with a plurality of different pattern lights. Below, we will explain the lighting device 2 used when acquiring an inspection image including height data by performing deflectometry processing.

When multiple light-emitting diodes are used, multiple light-emitting diodes can be arranged in a dot matrix and current value control can be used to generate a pattern light with a periodic illuminance distribution. For example, in the case of a Y-direction pattern light in which light and dark change in the Y direction, it can be expressed as a pattern light in which stripes are repeated in the Y direction. When generating this Y-direction pattern light, the phase of the illuminance distribution can be shifted in the Y direction to generate multiple Y-direction pattern lights with different phases of illuminance distribution. The illuminance distribution of the Y-direction pattern light can also be expressed as a waveform that approximates a sine wave. In this case, for example, by changing the phase in increments of 90°, it is possible to generate a Y-direction pattern light at 0°, a Y-direction pattern light at 90°, a Y-direction pattern light at 180°, and a Y-direction pattern light at 270°.

In addition, in the case of X-direction pattern light in which light and darkness change in the X direction, it can also be expressed as a pattern light in which stripes are repeated in the X direction, and when generating this X-direction pattern light, the phase of the illuminance distribution can be shifted in the X direction to generate multiple X-direction pattern lights with different phases of illuminance distribution. The illuminance distribution of the X-direction pattern light can also be expressed as a waveform that approximates a sine wave, and in this case, for example, by changing the phase by 90 degrees, it is possible to generate X-direction pattern light for 0 degrees, X-direction pattern light for 90 degrees, X-direction pattern light for 180 degrees, and X-direction pattern light for 270 degrees. In other words, the lighting device 2 can illuminate the workpiece W in different lighting modes. When performing deflectometry processing, the pattern light irradiated to the workpiece W can be not only a sine wave but also a triangular wave pattern light or the like.

Furthermore, by passing the same current value through all light-emitting diodes, it is possible to emit light with a uniform illuminance distribution across the surface. By passing the same current value through all light-emitting diodes and changing it, it is possible to change the light emission state from a dark surface emission state to a bright surface emission state.

In addition, in the case of liquid crystal panels and organic EL panels, each panel can be controlled so that the light emitted from each panel becomes pattern light having a periodic illuminance distribution. In the case of a digital micromirror device, a pattern light having a periodic illuminance distribution can be generated and emitted by controlling the built-in micromirror surface. Note that the configuration of the lighting device 2 is not limited to that described above, and any device or apparatus that can generate pattern light having a periodic illuminance distribution can be used.

As will be described later, the lighting device 2 used when generating an inspection image using the photometric stereo method can be a lighting device capable of individually irradiating light from at least three different directions. The lighting device 2 may also be a lighting device configured to be capable of multispectral lighting, and the configuration of the lighting device 2 is not particularly limited.

(Configuration of imaging unit 3)
Next, the imaging unit 3 will be described. The imaging unit 3 has a camera 31, a light collecting optical system 32, and an imaging control unit 33. The imaging unit 3 is connected to the setting device 4 via a signal line 100b, and can be installed away from the lighting device 2 and the setting device 4.

The camera 31 has an image sensor consisting of an image sensor 31d such as a CCD or CMOS that converts the intensity of light obtained through the light collecting optical system 32 into an electrical signal. The camera 31 is an image generating unit that obtains a luminance image according to the amount of reflected light received from the workpiece W, and receives the light reflected by the workpiece W from the structured illumination projected by the illumination device 2 to generate multiple pattern projection images. The imaging control unit 33 has a storage device and a signal processing device, and is a part that controls the start and end of exposure of the camera 31, controls the exposure time, adjusts the analog gain, etc. An internal logic circuit can control the driving of the image sensor and the transfer of image data. The imaging control unit 33 can also perform various image processing, filter processing, image generation, etc., and the imaging unit 3 is a device that has a filter processing function.

The focusing optical system 32 is an optical system for focusing light incident from the outside, and typically has one or more optical lenses. The focusing optical system 32 is configured to be capable of performing autofocus. The positional relationship between the imaging unit 3 and the illumination device 2 can be set so that the light irradiated from the illumination device 2 toward the surface of the workpiece W is reflected on the surface of the workpiece W and enters the focusing optical system 32. When the workpiece W is a member having translucency such as a transparent film or sheet, the positional relationship between the imaging unit 3 and the illumination device 2 can be set so that the pattern light irradiated from the illumination device 2 toward the surface of the workpiece W passes through the workpiece W and enters the focusing optical system 32 of the imaging unit 3. In any of the above cases, the imaging unit 3 and the illumination device 2 are arranged so that the specular reflection component and the diffuse reflection component reflected on the surface of the workpiece W enter the focusing optical system 32 of the imaging unit 3. Additionally, the imaging unit 3 can use a line camera in which light receiving elements are arranged in a straight line in the Y direction, but instead of a line camera, an area camera (a camera in which light receiving elements are arranged in a line in the X and Y directions) can also be used, and in the case of this area camera, a lighting form known as coaxial lighting is also possible.

(Embodiment 2)
2 is a diagram showing a schematic configuration and an operating state of an image inspection system 1 according to a second embodiment of the present invention. In the second embodiment, an imaging control unit 33 is separated from a camera 31 having an imaging element and a light collecting optical system 32, and an illumination control unit 2b is incorporated and integrated into the imaging control unit 33. The imaging control unit 33 is connected to the setting device 4 via a signal line 100b. The imaging control unit 33 and the illumination control unit 2b may be separate entities.

(Embodiment 3)
3 is a diagram showing a schematic configuration and an operating state of an image inspection system 1 according to a third embodiment of the present invention. In the third embodiment, the illumination control unit 2b is incorporated and integrated into the imaging control unit 33, and a display unit 5 and a mouse 7 are connected to the imaging control unit 33. The setting device 4 is formed of, for example, a notebook computer, and is connected to the imaging control unit 33. The setting device 4 has a keyboard 6. A mouse may also be connected to the setting device 4. Various user interfaces and the like generated by the setting device 4 can be displayed on the display unit 5 via the imaging control unit 33.

(Embodiment 4)
4 is a diagram showing a schematic configuration and an operating state of an image inspection system 1 according to a fourth embodiment of the present invention. In the fourth embodiment, the lighting device 2 and the imaging unit 3 are integrated, and the lighting control unit 2b and the imaging control unit 33 are also integrated. In the second to fourth embodiments, the same parts as those in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted.

(Deflectometry Processing)
In the first to fourth embodiments, the height data is acquired based on a plurality of pattern projection images generated by the imaging unit 3. Specifically, the imaging unit 3 generates phase data of the surface of the workpiece W based on the principle of deflectometry from a plurality of luminance images captured by the imaging unit 3, and performs deflectometry processing to generate height data in which the height information of the workpiece W is arranged on two-dimensional coordinates, i.e., a height image showing the shape of the workpiece W, based on the phase data.

As shown in an example in FIG. 5, the light-emitting unit 2a of the lighting device 2 includes a first light-emitting unit 21, a second light-emitting unit 22, a third light-emitting unit 23, and a fourth light-emitting unit 24, and each of the first to fourth light-emitting units 21 to 24 is capable of projecting a pattern.

Hereinafter, the generation of the height image will be described in detail with reference to FIG. 5. When the first light-emitting unit 21 of the lighting device 2 irradiates the workpiece W with four pattern lights according to the spatial code method, the imaging unit 3 generates a gray code pattern image set consisting of four different images. When the first light-emitting unit 21 of the lighting device 2 irradiates the workpiece W with eight pattern lights according to the phase shift method, the imaging unit 3 generates a phase shift pattern image set consisting of eight different images.

Similarly, when the second light-emitting unit 22 of the lighting device 2 irradiates the workpiece W with pattern light according to the spatial code method, a Gray code pattern image set is generated, and when the workpiece W is irradiated with pattern light according to the phase shift method, a phase shift pattern image set is generated.

Similarly, when the third light-emitting unit 23 of the lighting device 2 irradiates the workpiece W with pattern light according to the spatial code method, a gray code pattern image set is generated, and when the workpiece W is irradiated with pattern light according to the phase shift method, a phase shift pattern image set is generated.

Similarly, when the fourth light-emitting unit 24 of the lighting device 2 irradiates the workpiece W with pattern light according to the spatial code method, a gray code pattern image set is generated, and when the workpiece W is irradiated with pattern light according to the phase shift method, a phase shift pattern image set is generated.

After generating a phase shift pattern image set, each image data of the phase shift pattern image set can be acquired and a relative phase calculation process can be performed by using the phase shift method. This allows a phase image to be acquired. This phase is the relative phase (phase before unwrapping).

After generating the Gray code pattern image set, each image data of the Gray code pattern image set is acquired, and a spatial code calculation process is performed using the spatial code method to obtain a stripe number image. The stripe number image is an image in which a space irradiated with light is divided into many small spaces, and a series of spatial code numbers are assigned to the small spaces to make them identifiable.

Then, absolute phase phasing processing is performed. In absolute phase phasing processing, the phase image and the fringe number image are synthesized (unwrapped) to generate an absolute phase image (intermediate image). In other words, the spatial code number obtained by the spatial code method can be used to correct the phase jump by the phase shift method (phase unwrap), so high-resolution measurement results can be obtained while maintaining a wide dynamic range of height.

Height measurement may be performed using only the phase shift method. In this case, the dynamic range of height measurement is narrowed, so in the case of a workpiece W with a large difference in height that causes the phase to shift by one period or more, the height cannot be measured correctly. Conversely, in the case of a workpiece W with a small change in height, the fringe images are not captured or synthesized using the spatial code method, which has the advantage of speeding up processing accordingly. For example, when measuring a workpiece W with a small difference in height, there is no need to take a large dynamic range, so the processing time can be shortened while maintaining high-precision height measurement performance using only the phase shift method. Also, since the absolute height can be determined, the height may be measured using only the spatial code method. In this case, the accuracy can be improved by increasing the number of codes.

The absolute phase image can also be said to be an angle image in which each pixel has irradiation angle information of the pattern light on the workpiece W. That is, since the first pattern image set (phase shift pattern image set) includes eight first pattern images captured by shifting the phase of the sine wave stripe pattern, by using the phase shift method, each pixel has irradiation angle information of the pattern light on the workpiece W. In other words, based on multiple first pattern images, a first angle image can be generated in which each pixel has irradiation angle information of the first measurement pattern light on the workpiece W. The first angle image is an image that visualizes the angle of light irradiated from the first light-emitting unit 21 to the workpiece W.

Similarly, a second angle image can be generated in which each pixel has irradiation angle information of the second measurement pattern light onto the workpiece W based on a plurality of second pattern images, a third angle image in which each pixel has irradiation angle information of the third measurement pattern light onto the workpiece W based on a plurality of third pattern images, and a fourth angle image in which each pixel has irradiation angle information of the fourth measurement pattern light onto the workpiece W based on a plurality of fourth pattern images.

The top intermediate image in Figure 5 is the first angle image, the second image from the top is the second angle image, the third image from the top is the third angle image, and the bottom image is the fourth angle image. The areas that appear to be painted completely black in each angle image are areas that are in the shadow of the lighting, and are invalid pixels with no angle information.

The first height image can be generated from the first angle image and the second angle image, and the second height image can be generated from the third angle image and the fourth angle image. The first height image and the second height image can then be synthesized to generate a synthesized height image. This makes it possible to reduce the number of invalid pixels in the synthesized height image.

(Embodiment 5)
6 is a diagram showing a schematic configuration and an operating state of an image inspection system 1 according to a fifth embodiment of the present invention. While the first to fourth embodiments generate an inspection image based on the principle of deflectometry, the fifth embodiment is configured to generate an inspection image using a photometric stereo method.

The specific method for generating an inspection image using the photometric stereo method will be described below with reference to FIG. 6. The same parts as in embodiment 1 are designated by the same reference numerals and their explanations are omitted, and the differences will be described in detail.

The image inspection system 1 according to the fifth embodiment can be configured similarly to the image inspection system disclosed in, for example, JP 2015-232486 A. That is, the image inspection system 1 includes an imaging unit 3 that images the workpiece W from a certain direction, and an illumination device 200 that illuminates the workpiece W from three or more different illumination directions, and also includes at least a display unit 5, keyboard 6, and mouse 7 similar to those of the first embodiment. The imaging unit 3 and illumination device 200 constitute an acquisition unit A.

The lighting device 200 is configured to irradiate the workpiece W with light from different directions, and has first to fourth light-emitting units 201 to 204, and a lighting control unit 205 that controls the first to fourth light-emitting units 201 to 204. This lighting device 200 is a part that performs multi-directional lighting that irradiates the workpiece W with light from different directions, and is configured to be able to project structured lighting having a predetermined projection pattern. The first to fourth light-emitting units 201 to 204 are arranged to surround the workpiece W with a gap between them. The first to fourth light-emitting units 201 to 204 can be light-emitting diodes, incandescent bulbs, fluorescent lamps, etc. Also, the first to fourth light-emitting units 201 to 204 may be separate or integrated.

In this embodiment, the first to fourth light-emitting units 201 to 204 are sequentially turned on, and the imaging unit 3 captures the workpiece W when any of the first to fourth light-emitting units 201 to 204 is turned on. For example, when the lighting device 200 receives the first lighting trigger signal, the lighting control unit 205 turns on only the first light-emitting unit 201. At this time, the imaging unit 3 captures the workpiece W at the timing when the imaging trigger signal is received and light is irradiated. When the lighting device 200 receives the second lighting trigger signal, the lighting control unit 205 turns on only the second light-emitting unit 202, and at this time the imaging unit 3 captures the workpiece W. In this way, four luminance images can be obtained. Note that the number of lights is not limited to four, and can be any number as long as it is three or more and can illuminate the workpiece W from different directions.

Then, the normal vector of each pixel relative to the surface of the workpiece W is calculated using the pixel values of each pixel that correspond to each other in the multiple luminance images captured by the imaging unit 3. Differentiation processing is performed in the X and Y directions on the calculated normal vector of each pixel to generate a contour image that shows the contour of the inclination of the surface of the workpiece W. In addition, the albedo of each pixel, the same number of pixels as the normal vector, is calculated from the calculated normal vector of each pixel, and a texture rendering image is generated from the albedo to show a pattern with the inclination of the surface of the workpiece W removed. Since this method is well known, a detailed description is omitted. A shape image (height image) showing the shape of the workpiece W can be generated by synthesizing multiple luminance images based on the principle of photometric stereo.

(Multispectral lighting)
As another embodiment, the lighting device 2 may be capable of multispectral lighting as structured lighting having a predetermined projection pattern. Multispectral lighting is a method of irradiating the workpiece W with light of different wavelengths at different times, and is suitable for inspecting color unevenness, stains, etc. of printed matter (inspection target). For example, the lighting device 2 can be configured to be able to irradiate the workpiece W with yellow, blue, and red in sequence. Specifically, the lighting device 2 may have LEDs of multiple colors, or the lighting device 2 may be configured with a liquid crystal panel, an organic EL panel, or the like.

The imaging unit 3 captures images of the workpiece W at the timing when light is irradiated, obtaining multiple brightness images. The multiple brightness images can then be synthesized to obtain an image for inspection. This is called multispectral imaging. The irradiated light can also include ultraviolet and infrared rays.

(Connection interface between camera 31 and image inspection application)
The camera 31 in the first to fifth embodiments is a GenIcam-compliant camera that complies with the GenIcam standard. The GenIcam standard is a standard that standardizes the connection interface between a PC application and the camera 31, and as shown in FIG. 7, it is a standard that mainly controls the camera 31 from an image inspection application 40 constructed on a personal computer that is the main body of the setting device 4, and standardizes an interface that acquires an image captured by the camera 31 with the image inspection application 40 of the setting device 4. If both the imaging unit 3 and the image inspection application 40 of the setting device 4 are compatible with the GenIcam standard, it is possible to connect the camera 31 and the image inspection application 40 of the setting device 4. The image inspection application 40 is composed of software installed on a personal computer. In this embodiment, a case will be described in which the camera 31 and the setting device 4 are compatible with the GenIcam standard as a standardized standard, but the standardized standard is not limited to the GenIcam standard and may be another standardized standard.

If we consider the case where the image inspection application 40 is decomposed into a hierarchical structure, the GenIcam layer 41 is positioned as an intermediate layer located between the upper layer (inspection unit 4A, image processing unit 4B) that actually performs image inspection, defect inspection, etc. in the image inspection application 40, and layer 42 that performs control based on a specific network communication standard. The GenIcam layer 41 can be roughly classified into two parts, GenApi 41a and GenTL (TL: Transport Layer) 41b. GenApi 41a is a part that converts the setting items of the camera 31 and the register addresses inside the camera 31. With this GenApi 41a, the image inspection application 40 can abstractly specify a setting item as an argument, such as "ExposureTime" for exposure time or "AnalogGain" for analog gain, without specifying a specific address of the camera 31, and when connected to the camera 31, it can derive the register address corresponding to that string (called Feature) by analyzing a file called DeviceXML (details of which will be described later) obtained from the camera 31.

GenTL41b specifies the interface (API) that controls the transfer of data between the image inspection application 40 and the camera 31. Specifically, it specifies the API specifications for writing to and reading from the registers of the camera 31, and the API specifications for passing image data transferred from the camera 31 to a higher layer of the image inspection application 40.

The physical standard for connecting the image inspection application 40 and the camera 31 may be any standard that uses a high-speed network, such as the GigE Vision standard 3A that uses an Ethernet cable 31a and the USB3Vision standard 3B that uses a USB3.0-compatible cable 31b. Thus, the imaging unit 3 and the setting device 4 are connected via a network, which may be a wired network using a high-speed network cable or a wireless network. The network may also be a network using a cable other than the Ethernet cable 31a or the USB3.0-compatible cable 31b, and is not particularly limited.

The GenIcam standard does not specify a specific standard to be used as a physical communication standard, but merely specifies an abstract specification in the form of GenTL 41b. As a lower layer 42 of this GenTL 41b, communication standards are specified using specific communication networks, such as the GigE Vision standard 3A and the USB3Vision standard 3B. The specific communication standard is not limited to the GigE Vision standard 3A and the USB3Vision standard 3B, but it is sufficient that it is compatible with the GenIcam standard. In order to explain the concepts of the GigE Vision standard 3A and the USB3Vision standard 3B, FIG. 7 shows a state in which cameras 31 and 31 compatible with each standard are connected to the setting device 4, but only one of the cameras 31 can be connected to the setting device 4 and used.

A file called DeviceXML (DeviceXML file) 31c is stored in the GenIcam-compliant camera 31. The DeviceXML file 31c is stored in a storage device (internal memory) built into the camera 31. When an image inspection application 40 that complies with the GenIcam standard connects to a camera 31 that is to be configured by the image inspection application 40, the DeviceXML file 31c is read from the camera 31. The DeviceXML file 31c can be read, for example, by downloading the DeviceXML file 31c from the camera 31. The downloaded DeviceXML file 31c is held in GenApi 41a. The DeviceXML file 31c can be downloaded in response to a request from the image inspection application 40 or automatically when connected to the camera 31. The DeviceXML file 31c can be downloaded to the setting device 4, allowing the setting device 4 to acquire the DeviceXML file 31c. It is also possible to obtain the DeviceXML file 31c corresponding to the camera 31 by other means (e.g., downloading from a website) without downloading it from the camera 31 to be connected, and specify this to the GenApi 41a when connecting.

The DeviceXML file 31c describes all of the setting items stored inside the camera 31 in association with the register addresses (register information) where the setting values of each setting item are stored. The setting items are called Features, and each Feature is assigned a character string to identify the individual Feature. The node for each Feature contains a reference to a node that describes a specific register address. The register information also includes the register address and a character string that identifies the register.

In the DeviceXML file 31c, for example, there is a Feature named "ExposureTime" (exposure time setting), whose attribute is instructed to refer to a register address called "ExposureTimeReg", and a certain value is entered as the specific address. Different cameras 31 may have different address values, but the name of the Feature called "ExposureTime" is the same for all cameras 31. In this way, by managing setting items that are common to many cameras 31 under a unified name, it becomes possible for the image inspection application 40 to set and control the cameras 31 without being aware of differences in the models or manufacturers of the cameras 31.

As described above, the GenApi 41a layer analyzes the contents of the DeviceXML file 31c to convert the Feature string passed as an argument from the image inspection application 40, which is a higher layer, into a register address. For example, when writing a value (value: 100.5) to a register corresponding to a Feature called "ExposureTime", it is possible to set the register value of the camera 31 via a physical communication standard such as the GigE Vision standard 3A or the USB3Vision standard 3B by executing a function with two arguments, the address value and the value to be written, such as WriteRegister(address value, 100.5).

The DeviceXML file 31c specifies what should be included as common Features for each manufacturer, but it is also possible for a vendor to define its own Features. In the case of a camera 31 that has highly unique functions, it is possible to use special functions that do not exist in general-purpose cameras by accessing the camera 31 through a dedicated Feature.

(Internal processing unit of imaging unit 3)
8 is a flowchart conceptually illustrating an example of the internal processing of the imaging unit 3. In this flowchart, the position correction 1 group is shown to be composed of a pattern search unit and a position correction unit contained in the group, but the position correction 2 to 5 groups are not shown. This is because the position correction 2 to 5 groups are folded to simplify the drawing.

As shown in this flowchart, it is composed of a combination of multiple units. A unit is a unit that controls image capture and image processing, and by combining units on the flowchart, it is possible to cause the imaging section 3 to perform the desired operation. In the flowchart shown in Figure 8, a unit is described as one step.

For example, if a function that performs a certain process is enabled by the user's settings, the process can be made executable by adding a unit corresponding to that function. A unit can be defined as a combination of a program for executing the process, parameters required for executing the process, and a memory area for storing data resulting from the process. Each process can be performed by the imaging control unit 33 shown in FIG. 1, and the memory area can be secured in the memory device of the imaging control unit 33. Note that the concept of a unit itself is not essential for realizing the external specifications of the camera 31 that are compliant with the GenICam standard.

The flowchart shown in FIG. 8 has multiple units arranged vertically and horizontally in a flowchart format, and can simply be called a flow. It may also be a flowchart with multiple units arranged only vertically. Processing is executed in order from the start to the end of the flowchart shown in FIG. 8, but branching is also possible by providing a branching step SB1 along the way. If branching occurs, a merging step SB2 can be provided on the way to the end, which allows merging before proceeding to the end.

(Parameter set)
When the image inspection application 40 is operated in an actual inspection environment, the setting parameters of the imaging unit 3 may be dynamically changed when the workpiece W is switched or a change in the surrounding environment, such as brightness, is detected. When only a very limited number of parameters, such as the exposure time, are changed, the image inspection application 40 can directly write the corresponding Feature value.

On the other hand, in the case of a highly functional imaging unit 3, the number of configurable items increases, and the number of parameters that must be changed at once, such as when switching the workpiece W, also increases. In this case, the time required to change the settings increases in correlation with the number of parameters. In an actual image inspection line, when the workpiece W is switched, the time between when the new workpiece W arrives at the imaging field of view of the imaging unit 3 and when it leaves that field of view is often short, and there are cases where it is necessary to perform a series of setting changes quickly. In such cases, a function called a parameter set may be used.

A parameter set is a set of multiple combinations of parameters to be used when capturing images with the imaging unit 3, each of which is stored in advance and can be managed by a parameter set number. For example, if there are three types of workpieces W and it is desired to capture images and process each of them using different parameters, three parameter sets are prepared.

By using a parameter set, a series of setting changes can be completed in a short time by simply specifying the parameter set number corresponding to the type of workpiece W to be imaged next before imaging the workpiece W. From the perspective of the image inspection application 40, the Feature that specifies the parameter set number can be considered a type of selector, which will be described later.

The selector that switches the target to be set from the image inspection application 40 side and the register that switches the parameters that the imaging unit 3 internally references during operation can be independent or the same. The upper limit on the number of parameter sets that can be set is limited by the parameter storage space in the internal memory provided inside the imaging unit 3.

The concept of parameter sets can also be applied to the imaging unit 3 having a filter processing function. For example, suppose that parameters corresponding to the parameter set index are set so that when the parameter set index is 1, a binarization filter is executed, when the parameter set index is 2, a dilation filter is executed, and when the parameter set index is 3, the filter is not executed. In this way, it is possible to dynamically switch between imaging and the contents to be executed as filter processing depending on the parameter set index.

The GenIcam standard supports a function for dynamically switching imaging parameters. In accordance with the GenIcam standard, the imaging unit 3 allows all setting parameters to be accessed by Features. The imaging function has multiple illumination-imaging control modes, such as a function for synchronizing the illumination device 2 and the imaging unit 3 to synthesize images captured in multiple patterns using a dedicated algorithm (generation of an inspection image using the principle of deflectometry described above), and a multispectral imaging function for acquiring multiple images by irradiating light of different wavelengths. These can be set for each of the above parameter sets, and imaging processing, filter processing, synthesis processing, and image output can be performed while dynamically switching according to conditions. The image generated by the imaging function may be the image itself acquired by the image sensor while switching the illumination pattern, or may include multiple images synthesized by the above-mentioned dedicated algorithm.

It is possible to set multiple patterns of filter processing within the same parameter set. For example, in the above-mentioned imaging function, a series of imaging executions results in the generation of multiple patterns of images, and it is possible to apply filter processing to the different images generated therein individually. In another pattern, it is also possible to set a specific region of interest (ROI) for the same image and then filter only the inside of each region. The range of the specific region can be set, for example, by operating the keyboard 6 or mouse 7. There may be one specific region or multiple specific regions. The size of the specific region can be set as desired.

The filter process may be a multi-stage filter that can repeatedly set multiple types of filters in multiple stages for the same image. For example, after performing a dilation filter process on an image, a binarization filter process can be performed on the image. The filter process is not limited to a multi-stage filter, and may be a single filter process.

By utilizing the function of dynamically switching imaging parameters of the GenICam standard, multiple parameter sets can be combined into one flowchart as shown in FIG. 8. In the internal processing flowchart of the imaging unit 3, a group of units forming the flowchart from branching step SB1 to merging step SB2 are collectively called a parameter set. In the example shown in FIG. 8, there are four parameter sets, namely, the first to fourth parameter sets. In other words, there are multiple patterns of combinations of values of multiple selectors to realize the processing set by the user. The user can set which of the first to fourth parameter sets to select; for example, when parameter set number 1 is selected, the first parameter set is automatically selected.

After starting the flow chart shown in FIG. 8, in branching step SB1, the process passes through each unit constituting one of the first to fourth parameter sets, then merges at merge step SB2, and can proceed to the end. If parameter set number 1 is selected, the branch number becomes 1 in branching step SB1, and each process of the first parameter set is executed. If parameter set number 2 is selected, the branch number becomes 2 in branching step SB1, and each process of the second parameter set is executed. If parameter set number 3 is selected, the branch number becomes 3 in branching step SB1, and each process of the third parameter set is executed. If parameter set number 4 is selected, the branch number becomes 4 in branching step SB1, and each process of the fourth parameter set is executed. The number of parameter sets is not limited to four, and can be set arbitrarily.

Specific examples of parameter sets are shown in FIG. 9 and FIG. 10. FIG. 9 shows a parameter set for generating an inspection image using the principle of deflectometry. In order to generate an inspection image using the principle of deflectometry, the unit UA1 executes multiple imaging processes with one trigger signal, the unit UA2 executes expansion filter processing, the unit UA3 executes averaging filter processing, and the unit UA4 executes gray-scale inversion processing. In other words, the multi-stage processing defined by the parameter set including filter processing is sequentially executed on the image captured by the unit UA1. Thereafter, the unit UA5 transfers the image data on which the multi-stage processing has been sequentially executed to the PC, that is, outputs it to the setting device 4, which is an external device. The image data may be stored internally without being transferred. Depending on the parameter set, it may be set so that the multi-stage processing is not performed. When transferred to the setting device 4, the inspection unit 4A of the image inspection application 40 shown in FIG. 7 can perform defect inspection and pass/fail judgment. The algorithm for defect inspection and pass/fail judgment can be a conventionally known one.

On the other hand, FIG. 10 shows a parameter set for generating an inspection image by multispectral imaging. In order to generate an inspection image by multispectral imaging, unit UB1 executes multiple imaging processes with one trigger signal, unit UB2 executes binarization filter processing on a certain region (region 0), and unit UB3 executes dilation filter processing on a region (region 1) different from region 0. In other words, in this example, multi-stage processing defined by a parameter set including filter processing is sequentially executed on the captured image captured by unit UB1. After that, unit UB4 does not execute processing. Unit UB5 transfers image data to a PC in the same way as in the case shown in FIG. 9. As in this example, the parameter set may include a unit that does not perform processing, i.e., a disabled unit, or the parameter set may include a mixture of enabled and disabled units.

(Tracking moving objects)
When generating an inspection image based on the principles of photometric stereo or multispectral, multiple image capture processes are performed with one trigger signal, and the resulting multiple images are synthesized. In this case, if the workpiece W moves between the multiple image capture processes, the synthesis process after imaging cannot be performed correctly, so it is necessary to detect the amount of misalignment for each image. A search unit can perform a search for detecting the amount of misalignment, and the synthesis process of the multiple images is performed after correcting the amount of misalignment based on the detected amount of misalignment.

The compositing process can be performed by the image calculation unit. The DeviceXML file includes, as a setting item, a compositing setting for compositing multiple captured images. Therefore, before an image captured by the imaging unit 3 is output to the outside, the DeviceXML file includes, as a setting item related to the processing applied to the captured image, a compositing setting for compositing multiple captured images.

(Unit type)
There are several types of units, including, for example, a pattern search unit that performs pattern search processing to determine an inspection area, a position correction unit, an image calculation unit that performs internal position correction of an image, color extraction, filter processing, etc., and a highly functional combination of these units that perform relatively simple processing. Each unit is a unit that performs processing applied to the captured image captured by the imaging section 3 before outputting the image to the outside.

The pattern search unit is a unit for searching for the workpiece W or the part of the workpiece W to be inspected from the image including the workpiece W captured by the imaging unit 3, and for correcting the position of the workpiece W in the captured image. For example, when setting up the image inspection system 1, edge detection can be performed on the image of the workpiece W using a known edge detection method, and the area specified by the detected edge can be stored in the storage device of the imaging unit 3 as a model of the workpiece W or the part of the workpiece W to be inspected (search model image). The edge detection process itself can use a conventionally known method, for example, the pixel value of each pixel on the luminance image is obtained, and if there is an area where the change in pixel value on the luminance image is equal to or greater than the threshold value for edge detection, the boundary part is extracted as an edge. The threshold value for edge extraction can be adjusted as desired by the user.

After the image inspection system 1 is set up, when the image inspection system 1 is in operation, the workpieces W that are transported in sequence are imaged to obtain an inspection image, and the pattern search unit performs a search process based on the stored model to determine whether or not the workpiece W or the part of the workpiece W to be inspected is present in the obtained inspection image, and measures the position and angle of the workpiece W through the search process. The position of the workpiece W can be specified by the X coordinate and the Y coordinate. The angle of the workpiece W can be the angle around the imaging axis of the imaging unit 3, or the angle around the Z axis shown in FIG. 1.

The position correction unit is a unit that searches for the workpiece W or the part of the workpiece W to be inspected in an image including the workpiece W captured by the imaging unit 3 using the pattern search unit, measures the position and angle of the workpiece W, and then corrects the position of the workpiece W in the image. When the image inspection system 1 is in operation, multiple workpieces W are not always transported in the same position and orientation, and workpieces W in various positions and orientations may be transported. Since the pattern search unit searches for the reference part of the workpiece W and then the position correction unit can correct the position, the image is rotated or moved vertically or horizontally so that the reference part of the workpiece W is always in a fixed position and the workpiece W is in a specified orientation, and position correction is performed. As a position correction tool for performing position correction, multiple types of tools, such as a pattern search, can be prepared.

There are several types of image calculation units, such as a unit that performs filtering, a unit that performs synthesis processing to synthesize multiple images captured by the imaging unit 3, a unit that performs deflectometry processing, a unit that generates an inspection image using a photometric stereo method, and a unit that performs multispectral imaging. Since there are several types of filtering, multiple types of units that perform filtering can be provided, such as a binarization filter and an expansion filter. Since the generation of an inspection image by deflectometry processing involves multiple processes as described above, a unit can be provided for each process, and a unit that generates a specular reflection component image, a unit that generates a diffuse reflection component image, a unit that generates a reference phase difference image, and the like can be provided.

(Configuration of Setting Device 4)
The setting device 4 is a device for transmitting to the imaging unit 3 data indicating the setting values of each setting item set by the user and register information corresponding to each setting item contained in the DeviceXML file 31c, and configuring the imaging unit 3.

As shown in FIG. 7, the setting device 4 includes a UI generation unit 4a. The UI generation unit 4a is a part that generates various user interface images. The various user interface images generated by the UI generation unit 4a are displayed on the display unit 5.

On the user interface image, it is possible to switch the image to be edited, display and change the setting items, and display the image to be edited. Setting items include, for example, position correction settings and filter processing settings, and as Features corresponding to position correction settings, a section for selecting whether or not to enable position correction and a section for selecting the type of position correction tool are assigned. In addition, as Features corresponding to filter processing settings, a section for selecting and adjusting the type of selected filter and parameters related to filter settings such as extraction size and extraction color are assigned.

For example, when a user selects an image, the value of the selector that specifies the setting target internally switches to the index value that corresponds to the pre-processing module used to generate that image, and the configurable contents switch according to the image. When the value of the selector that specifies the setting target is specified, one or more Features of the pre-processing module pointed to by the selector that corresponds to that value are read, and an image that reflects the setting items is generated and displayed, while the respective parameter values are displayed as position correction and filter settings. When the user operates the parameter values of position correction and filter settings, the operation is accepted and the setting items of the unit corresponding to the pre-processing module that corresponds to the value of the selector that specifies the setting target are changed.

The following method can be used to identify the unit to be accessed from the selector value. That is, a pre-processing module is made up of multiple units, and a selector that switches the index of the pre-processing module can be used that is common to these units, and it is possible to name the multiple units that make up the pre-processing module so that which Feature belongs to which unit is uniquely determined by the Feature name. This makes it possible to identify the unit to be accessed from the combination of the selector value and the Feature selected to be edited.

Therefore, when the user changes the setting values for the setting items of the imaging unit 3, the setting values for each setting item set by the user and data indicating the register information corresponding to each setting item contained in the DeviceXML file are transmitted to the imaging unit 3, and the imaging unit 3 can be configured. Therefore, if the imaging unit 3 complies with the standardization specification, the setting values can be changed from the setting device 4, improving the freedom of selection of the imaging unit 3 model.

In addition, a combination of selector values can uniquely identify a part of the multi-stage processing that is executed sequentially, so that an imaging unit 3 that complies with standardization standards can perform multi-stage processing such as generating inspection images using the principles of photometric stereo or deflectometry, multispectral imaging, and applying filter processing to the generated inspection images.

(Specific configuration example of the test setting unit 400)
A specific configuration example of the setting device 4 will be described below. As shown in FIG. 7, the setting device 4 includes an inspection setting unit 400. The inspection setting unit 400 is a part that performs various inspection settings based on user operations in the setting device 4, and includes an object generating means 401, a displaying means 402, a posture adjusting means 403, an area designation receiving means 404, an element generating means 405, and an area setting means 406, as shown in FIG. 11. The inspection setting unit 400 is constructed in the setting device 4 by executing a program installed in the setting device 4, and each of the means 401 to 406 functions as shown in the figure by executing the program on the setting device 4. In FIG. 11, the means 401 to 406 are described separately for convenience, but one means may have multiple functions, and FIG. 11 is conceptual. Also, a part of each of the means 401 to 406 may be configured by hardware.

(Configuration of Object Generation Means 401)
The object generating means 401 is a means for generating a three-dimensional object that stereoscopically represents the workpiece W based on the height data acquired by the acquisition unit A. The three-dimensional object may be a three-dimensional polygon display that displays the height data by connecting it with a triangular plane, or a three-dimensional point cloud display that displays each height data for each pixel.

Now, we will explain 2.5D (two and a half dimensional) data. Generally, when displaying a three-dimensional object on a monitor or the like, 3D (three dimensional) display is used, and in this three dimensional display, data is handled by holding three dimensional coordinates of X, Y, and Z. In contrast, 2.5D has some restrictions that are close to 2D within 3D, and one height datum is assigned to each pixel. Although 2.5D has a lower degree of data freedom than 3D, it has the characteristic that measurement processing is easier to perform, and in this example, 2.5D data may be used for measurement processing. In 2.5D, only the part visible from the camera is measured, so coordinate data for the back side of the object does not exist, but it is possible to display it three-dimensionally using the above-mentioned three-dimensional polygon display or three-dimensional point cloud display.

The use of 2D data makes it possible to perform measurements such as shading inspection, pattern search, edge extraction, blobs, OCR, etc. on the target area (inspection area) that is the subject of image inspection on the workpiece W. Furthermore, the use of 2.5D data makes it possible to perform measurements such as height measurement, 3D search, profile measurement, 3D blobs, etc. on the target area. Although the types of data differ between 2D data and 2.5D data, they are the same in that the data is handled in pixel units by the camera 31, so the target area data itself does not change.

Various types of filtering may be performed on the 2D data or 2.5D data. For example, blurring may be performed on the 2D data, smoothing may be performed on the 2.5D data, or 2D image conversion may be performed on the 2.5D data to convert it to 2D data. The filtering may be varied depending on the type of measurement being performed.

(Configuration of display means 402)
The display means 402 is a means for controlling the display unit 5 which is a monitor of the setting device 4, and for example, causes the display unit 5 to display a three-dimensional object generated by the object generating means 401. The display means 402 can also cause the display unit 5 to display various viewpoint specifying elements generated by the element generating means 405 described later, superimposed on the three-dimensional object.

(Configuration of the attitude adjustment means 403)
The attitude adjustment means 403 is a means for adjusting the attitude of the three-dimensional object displayed on the display unit 5 by the display means 402. The attitude of the three-dimensional object can be adjusted by rotating the three-dimensional object around each of the X-axis, Y-axis, and Z-axis, or by moving the object in any of the X-direction, Y-direction, and Z-direction.

Specifically, the attitude adjustment means 403 is capable of detecting the operating state of, for example, the keyboard 6 or the mouse 7, and by clicking or dragging the mouse 7 on a three-dimensional object displayed on the display unit 5, the three-dimensional object can be rotated by any angle around any of the X, Y, and Z axes. In addition, by operating the mouse 7 on the three-dimensional object, the three-dimensional object can be moved by any amount in any of the X, Y, and Z directions.

In addition, by operating the keyboard 6 or mouse 7, the rotation angle around each axis of the three-dimensional object can be set numerically, and the amount of movement in any direction can be set numerically. The set numerical value is accepted by the attitude adjustment means 403. The attitude adjustment means 403 adjusts the attitude of the three-dimensional object by rotating or moving the three-dimensional object so as to correspond to the accepted numerical value. The above-mentioned method for adjusting the attitude of the three-dimensional object is an example, and the attitude of the three-dimensional object may be adjusted by other methods.

By adjusting the orientation of the three-dimensional object, the user can display the three-dimensional object on the display unit 5 as viewed from a desired direction. This makes it easier to specify the target area, as described below.

(Specifying the target area when setting)
The area designation receiving means 404 is a means for receiving designation of a target area to be subjected to image inspection on the three-dimensional object whose orientation has been adjusted by the orientation adjustment means 403. The target area is created by a user's operation as described later, and may be, for example, a linear area or an area surrounded by a figure, and may have any shape or size as long as it has a certain range. The linear area may be a straight line area or a curved area. The curved area may be, for example, a circle or an arc, or may be a free curve. Examples of areas surrounded by figures include areas surrounded by a circle, an ellipse, an arc, a polygon, etc.

The user can view the three-dimensional object that represents the workpiece W in a three-dimensional manner from any direction, so that the user can easily and reliably grasp the height difference even in small steps that exist on the workpiece W. Furthermore, since the target area can be specified while grasping the height difference of the workpiece W, the user can easily grasp where on the workpiece W the target area is specified and what size or shape it is.

The procedure for creating a new target area will be described with reference to the flowchart shown in FIG. 12. This flowchart starts when the user performs an operation to start creating a new target area. An example of an operation to start creating a new target area is pressing a start button (not shown).

After starting, in step SC1, a click operation of the mouse 7 by the user is detected, and the coordinates of the clicked point are obtained. In step SC1, the XY coordinates of the clicked point are obtained.

Then, proceed to step SC2 to perform a hit determination with the three-dimensional image data forming the three-dimensional object. Here, a 3D hit determination is performed to determine whether the XY coordinates acquired in step SC1 hit the three-dimensional image data. The 3D hit determination will be explained based on FIG. 13. When point P1 is clicked on the display unit 5 to obtain the XY coordinates of point P1, an extension line L1 is virtually generated extending from point P1 in the Z direction in three-dimensional space. The Z direction coincides with the imaging axis direction of the camera 31. If the extension line L1 intersects with the workpiece W1, a hit point PH exists and it is determined that a hit has occurred. On the other hand, if the extension line L1 does not intersect with the workpiece W1, it is determined that there is no hit.

After step SC2 in FIG. 12, the process proceeds to step SC3. In step SC3, it is determined whether or not a hit point was found in step SC2. If no hit point was found, the process proceeds to the end, where the user clicks on another location. If the determination in step SC3 is YES and a hit point was found, the process proceeds to step SC4, where the XY coordinates of the hit point PH are obtained and the display is treated in the same way as a two-dimensional display. In other words, point P1 is displayed at a location in the display area of the display unit 5 that corresponds to the XY coordinates of the hit point PH.

After step SC4, proceed to step SC5 to determine whether the number of points is greater than or equal to a predetermined number. For example, when creating a linear target area, at least two points are required to determine the line, in which case step SC5 determines whether the number of points is two. Also, for example, when creating a triangular target area, an arc, or a circular target area, if three points are required, step SC5 determines whether the number of points is three. The user can select the shape of the target area to be created before step SC1. The determination in step SC5 changes depending on the shape of the selected target area.

If step SC5 returns NO and the number of points is not the predetermined number or more, the process proceeds to step SC1 to accept a click operation for the next point, whereas if step SC5 returns YES and the number of points is the predetermined number or more, the target area can be identified and the creation of the target area is completed. The created target area is assumed to have been specified by the user and the process proceeds to step SC6. In step SC6, the target area created by the user as described above is added. In step SC7, the target area data added in step SC6 is saved. Target area data is data regarding the shape, position, size, etc. of the target area.

(Creating a Linear Region of Interest)
Next, a specific example of creating a target region will be described using an example of a three-dimensional object. First, a specific example of creating a linear target region will be described based on Fig. 14. The upper side of Fig. 14 shows a three-dimensional object 200 representing a workpiece W as displayed on the display unit 5 by the display means 402, and the lower side of Fig. 14 shows a two-dimensional image (height image) 201 of the workpiece W as displayed on the display unit 5 by the display means 402. Such a two-dimensional image 201 can be generated by the acquisition unit A.

The display means 402 is configured to be able to switch between displaying the three-dimensional object 200 and the two-dimensional image 201 on the display unit 5. For example, a switching operation or a selection operation by the user can cause only the three-dimensional object 200 to be displayed on the display unit 5, or only the two-dimensional image 201 to be displayed on the display unit 5. It is also possible to display both the three-dimensional object 200 and the two-dimensional image 201 on the display unit 5. Examples of the switching operation or selection operation include operation of a switching button or selection button.

When creating a linear target area, the user first selects "linear" from the options for the shape of the target area. Then, when the pointer 7a of the mouse 7 is placed on the three-dimensional object 200, a line segment L2 is drawn extending upward (in the Z direction) from the tip of the pointer 7a. When the pointer 7a is moved, the line segment L2 moves in the same manner. The extension direction of the line segment L2 is the direction in which the imaging element 31d of the camera 31 is located, and it extends a predetermined length along the imaging axis of the imaging element 31d.

When the mouse 7 is pressed, a hit determination is made in step SC2 shown in FIG. 12 based on the XY coordinates of point P1 indicated by the pointer 7a at that time. As a result, if point P1 hits the three-dimensional object 200, point P1 is set, but if it does not hit, point P1 is not set. This point P1 becomes the starting point. In addition, when the mouse 7 is pressed, the color of line segment L2 changes and line segment L2 is fixed at the position of point P1, so that it can be visually understood that point P1 has been set. Note that a press operation means pressing the button on the mouse 7.

While the user presses the mouse 7, the user drags the mouse 7 to move the pointer 7a away from point P1. While the user is dragging, a line segment L4 extending upward from the tip of the pointer 7a is drawn separately from the line segment L2. When the user then releases the mouse 7, a hit determination is made in step SC2 shown in FIG. 12 based on the XY coordinates of point P3 indicated by the pointer 7a at the time of the release. If point P3 hits the three-dimensional object 200, point P3 is set as the end point, the color of the line segment L4 changes, and the line segment L4 is fixed at the position of point P3. If the line segment does not hit, point P3 is not set. The top end of the line segment L4 and the top end of the line segment L2 are at the same height.

When point P3 is set, a line segment (upper XY direction line) L5 is drawn connecting the upper end of line segment L4 and the upper end of line segment L2. A projection line L6 extending from point P1 to point P3 is drawn on the three-dimensional object 200 at the position where line segment L5 is projected in the height direction. The projection line L6 is displayed superimposed on the three-dimensional object 200, and has a shape that follows the three-dimensional object 200. The portion of the surface of the three-dimensional object 200 where the projection line L6 is drawn becomes the target area.

In other words, when the area designation receiving means 404 receives the designation of a start point (point P1) and an end point (point P3) on the three-dimensional object 200, it sets the target area at the coordinates corresponding to the position of the projection line L6 extending from point P1 to point P3 in the two-dimensional coordinates of the height data. By performing the target area designation operation on the three-dimensional object 200 in this way, it is possible to easily designate as the target area the side of the wall of the workpiece W, which is difficult to designate on the two-dimensional image 201.

Line segment L2 starting from point P1 and line segment L4 starting from point P3 are parallel to each other and extend along the imaging axis of image sensor 31d toward image sensor 31d. By referring to line segment L2 and line segment L4, the user can identify the viewpoint at the position of image sensor 31d. Line segment L2 and line segment L4 are included in the viewpoint identification element generated by element generation means 405. Element generation means 405 is a means for generating line segment L2 and line segment L4 as viewpoint identification elements for identifying the viewpoint at the position of image sensor 31d when the designation of the target area is accepted by area designation acceptance means 404. The viewpoint identification element generated by element generation means 405 is displayed on display unit 5 by display means 402, so that the user can grasp the direction of image sensor 31d while creating the target area.

When point P3 is set, graphic element D1 is displayed superimposed on the three-dimensional object 200. Graphic element D1 is included in the viewpoint identification element, and this graphic element D1 is also generated by area designation receiving means 404 and displayed on the display unit 5 by display means 402. Graphic element D1 has a larger area than a line, and in this example, it is configured as a plane extending from projection line L6 in the same direction as line segments L2 and L4, that is, a plane extending along the imaging axis of the imaging element 31d. Graphic element D1 is formed between line segments L2 and L4, and both edges of graphic element D1 are determined by line segments L2 and L4, so graphic element D1 is an element formed based on line segments L2 and L4. Graphic element D1 is also drawn downward from projection line L6, although this is not essential.

The height of the upper end of the graphic element D1 is the same as the height of the upper ends of the line segments L2 and L4. The lengths of the line segments L2 and L4 are determined according to the maximum height and minimum height in the height image. That is, when determining the length of the line segments L2 and L4, the element generating means 405 first acquires the maximum height and minimum height in the height image based on the height data. Next, the element generating means 405 increases the length of the line segments L2 and L4 the greater the difference between the acquired maximum height and minimum height, and decreases the length of the line segments L2 and L4 the smaller the difference between the acquired maximum height and minimum height. Since the length of the line segments L2 and L4 varies depending on the height of the upper ends of the line segments L2 and L4, it can also be said that the element generating means 405 determines the height of the upper ends of the line segments L2 and L4 according to the maximum height in the height image. Similarly, it can also be said that the element generating means 405 determines the height of the upper end of the graphic element D1 according to the maximum height in the height image.

The lengths of line segments L2 and L4 can be determined according to the maximum and minimum heights in the height image, so that line segments L2 and L4 are not too short or too long relative to the height of the workpiece W, making line segments L2 and L4 easier to see. Similarly, the vertical dimension of graphic element D1 is also determined, making graphic element D1 easier to see. In addition, the upper and lower limits of the lengths of line segments L2 and L4 may be set in advance.

It is also possible to create a curved target area, such as a circular arc. In this case, the graphic element D1 will be composed of a curved surface. The graphic element D1 may also be composed of a flat surface and a curved surface.

The graphic element D1 can be made semi-transparent. Semi-transparent means that the other side of the graphic element D1 can be seen through the graphic element D1. The transparency of the graphic element D1 can be low or high, as long as the user can see the outline of the workpiece W located on the other side of the graphic element D1 displayed on the display unit 5. The graphic element D1 can be colored in a color different from the color of the workpiece W, which makes it easy to distinguish between the graphic element D1 and the workpiece W. The graphic element D1 and the line segments L2, L4, and L5 are displayed in a distinguishable manner. In this example, the graphic element D1 is semi-transparent, and the line segments L2, L4, and L5 are opaque, which makes it easy to distinguish between the two. The distinguishable display manner may be a form in which the colors of the two are changed or the color intensity is changed.

The viewpoint identification element may be anything that displays the viewpoint from which the workpiece W is viewed from the camera 31, and may include, for example, an arrow or a mesh-like graphic element. The graphic element D1 may be, for example, an element such as a transparent white wall.

Graphic element D1 does not have to be translucent. In this case, it will be impossible to see through graphic element D1, but if graphic element D1 is colored a different color from workpiece W, it will be possible to distinguish it from workpiece W.

When points P1 and P3 are specified on the three-dimensional object 200, points P1 and P3 are displayed on the two-dimensional image 201, and a line segment L5 connecting points P1 and P3 is drawn, as shown in the lower part of Figure 14. A projection line L6 is positioned directly below line segment L5. This allows the target area to be grasped not only on the three-dimensional object 200, but also on the two-dimensional image 201.

The system may also be configured to accept the specification of a target area on the two-dimensional image 201. That is, when points P1 and P3 are specified on the two-dimensional image 201, a line segment L5 is drawn on the two-dimensional image 201, and points P1 and P3 are displayed on the three-dimensional object 200, and a projection line L6 and a graphic element D1 are also drawn. By specifying a target area on the two-dimensional image 201 in this way, it is possible to easily specify as a target area the center of a hole in the workpiece W or the back of a step, which are difficult to specify on the three-dimensional object 200.

(Creating a rectangular target area)
Next, a specific example of creating a rectangular target region will be described with reference to Fig. 15. First, the user selects "rectangle" from among the options for the shape of the target region. After that, when the pointer 7a of the mouse 7 is placed on the three-dimensional object 200, a line segment L2 extending upward from the tip of the pointer 7a is drawn, similar to the case of a straight line.

When the mouse 7 is pressed, a hit determination is made in step SC2 shown in FIG. 12 based on the XY coordinates of point P1 indicated by the pointer 7a at that time. As a result, if point P1 hits the three-dimensional object 200, point P1 is set.

While the mouse 7 is still pressed, the mouse 7 is dragged to move the pointer 7a away from point P1. While the mouse 7 is being dragged, a line segment L4 extending upward from the tip of the pointer 7a is drawn. When the mouse 7 is then released, a hit determination is made in step SC2 shown in FIG. 12 based on the XY coordinates of point P3 indicated by the pointer 7a at the time of the release. If point P3 hits the three-dimensional object 200, point P3 is set.

When point P3 is set, a rectangular frame L7 is drawn with a virtual line (not shown) extending from point P1 to point P3 as its diagonal. In addition, line segments L10 and L11 are drawn extending from two other vertices of rectangular frame L7. Line segments L10 and L11 are parallel to line segments L2 and L4, and line segments L2, L4, L10, and L11 are each positioned at the vertices of rectangular frame L7.

In addition, a projection line L8 is drawn on the three-dimensional object 200 at a position where the rectangular frame L7 is projected in the Z direction (height direction). The projection line L8 is displayed superimposed on the three-dimensional object 200, and has a shape that follows the three-dimensional object 200. The part on the surface of the three-dimensional object 200 that is surrounded by the projection line L8 becomes the target area.

When point P3 is set, graphic element D2 is displayed superimposed on the three-dimensional object 200. Graphic element D2 is included in the viewpoint identification element. Graphic element D2 is composed of four planes that extend from projection line L8 in the same direction as line segments L2 and L4. The four planes are a plane between line segments L2 and L10, a plane between line segments L2 and L11, a plane between line segments L4 and L10, and a plane between line segments L4 and L11, and graphic element D2 is an element formed based on line segments L2, L4, L10, and L11. Graphic element D2 is also drawn downward from projection line L8, although this is not essential.

The target area may be highlighted. For example, the target area may be colored a different color than the other areas or may have a predetermined pattern.

(Change of target area)
Next, a case where the position, size, and shape of the target area specified as described above are changed will be described with reference to Fig. 16 and Fig. 17. Fig. 16 shows the first stage of the procedure for changing the target area, and after this first stage, the procedure proceeds to the second stage shown in Fig. 17. In the example described below, a case where the user changes the target area using the mouse 7 will be described, but the target area can also be changed by operating a touch operation panel other than the mouse 7.

In step SD1 of the flowchart shown in FIG. 16, the coordinates (XY coordinates) of the pointer at the time when the user clicks the mouse 7 are obtained. Then, in step SD2, a 3D hit determination is performed for a line extending in the Z direction from the point specified by the coordinates obtained in step SD1, with the XY coordinates fixed. The 3D hit determination method is the same as step SC2 of the flowchart in FIG. 12, and it is determined whether or not the line extending in the Z direction hits the three-dimensional image data.

After step SD2, the process proceeds to step SD3. In step SD3, it is determined whether or not a hit occurred in step SD2. If there is no hit, the process proceeds to the end and the user clicks on another location. If the determination in step SD3 is YES and there is a hit, the process proceeds to step SD4. In step SD4, a two-stage hit determination is performed, that is, a 3D hit determination is followed by a 2D hit determination.

Figure 18A is a conceptual diagram of 3D hit determination. Straight line L20 is a line extending in the Z direction from the point specified by the coordinates acquired in step SD1 with the XY coordinates fixed, plane D3 is a Z-direction surface, and plane D4 is an XY-direction surface. First, a 3D hit determination is performed with the upper end of plane D4. That is, although there are various types of lines in the target area and they are complex, they are limited to XY-direction lines, and the X coordinates to be drawn are also fixed. Using this, a 3D hit determination is performed with plane D3 for each Z coordinate, and the XY coordinates of the hit points obtained by the hit determination are obtained. Then, as shown in Figure 18B, a 2D hit determination is performed using the XY coordinates acquired by the 3D hit determination. This makes it easy to implement 3D hit determination.

After step SD4 shown in FIG. 16, the process proceeds to step SD5. In step SD5, it is determined whether or not a hit was made in step SD4. If there was no hit, the process proceeds to the end and the user clicks on another location. If the determination in step SD5 is YES and there was a hit, the process proceeds to step SD6.

In step SD6, a 3D hit determination is performed on the bottom end of the XY plane in the same manner as in step SD4, and then a 2D hit determination is performed. After step SD6, the process proceeds to step SD7. In step SD7, it is determined whether or not a hit occurred in step SD6. If there is no hit, the process proceeds to the end and the user clicks on another location. If the determination in step SD6 is YES and there is a hit, the process proceeds to step SD8. In step SD8, the XYZ coordinates of the hit point are saved. The above steps are executed automatically in response to a click by the user.

The user can move the target area by moving the mouse 7 (dragging) while still performing the above-mentioned click operation. When a drag operation is performed, the process proceeds to step SE1 of the flowchart shown in FIG. 17. In step SE1, the coordinates (XY coordinates) of the pointer moved by the drag operation are obtained. After that, the process proceeds to step SE2, where a 3D hit determination is performed with the Z coordinate plane of the XYZ coordinates of the hit point saved in step SD8 of the flowchart shown in FIG. 16.

After step SE2, the process proceeds to step SE3. In step SE3, it is determined whether or not a hit was made in step SE2. If there was no hit, the process proceeds to the end, and the user performs a drag operation to another location. If the determination in step SE2 is YES and there was a hit, the process proceeds to step SE4.

In step SE4, the viewpoint-specific element and the target area are translated in the XY directions to the hit coordinates. In other words, if a hit occurs at the coordinates of the pointer after the drag operation, the target area and viewpoint-specific element (line segments L2, L4, L10, L11, and graphic elements D1, D2 in FIG. 14) are translated to those coordinates, so that the target area and viewpoint-specific element move in almost real time to follow the pointer. This allows the user to adjust the position of the target area on the three-dimensional object so that it is placed in the desired position.

In step SE4, the viewpoint identification element is only permitted to move in parallel in the X or Y direction, and movement in the Z direction (height direction) is prohibited. In other words, the Z-direction position of the viewpoint identification element is made to not change before and after the movement. The viewpoint identification element is an element that is displayed when the target area to be inspected by the image is set to height data arranged on two-dimensional coordinates, so it is sufficient to move it in the X or Y direction of the three-dimensional object, and there is no need to move it in the direction in which the viewpoint is located (the height direction of the three-dimensional object). If the viewpoint identification element were movable in the height direction of the three-dimensional object, it is considered that the setting operation of the target area would become complicated. In this embodiment, the viewpoint identification element can be moved only in the X or Y direction of the three-dimensional object, which makes it easier to set the target area.

In step SE4, the target area is merely translated, and the position of the target area has not yet been finalized. After step SE4, the process proceeds to step SE5, where it is determined whether or not a confirmation operation has been performed by the user. An example of a confirmation operation is the operation of releasing the button of the mouse 7 (release operation), but it may be another operation.

If step SE5 returns NO and the confirmation operation has not been performed, the process returns to step SE1. If step SE5 returns YES and the confirmation operation has been performed, the process proceeds to step SE6, where the target area is changed to the one after translation. In step SE7, the target area data changed in step SE6 is saved.

Figures 19 and 20 show specific examples 1 and 2 when adjusting the position of a linear target area. The left image in Figure 19 for specific example 1 shows the target area before the position is adjusted, and the right image shows the target area after the position is adjusted. In the left image in Figure 19, the pointer 7a of the mouse 7 is moved onto line segment L5 (after mouse hover) and then pressed. This causes line segments L2 and L4 to change color, indicating that they have been selected by the user.

After that, when the mouse 7 is dragged, the line segments L2, L4, L5, graphic element D1, and projection line L6 move together along the movement path of the pointer 7a, but this movement is prohibited in the Z direction and is an XY parallel movement. The user may place the pointer 7a on any of the line segments L2, L4, L5, graphic element D1, and projection line L6 and drag them. When the line segment L5 is dragged, the line segments L2, L4, L5, graphic element D1, and projection line L6 are moved in the XY parallel direction so that the line segment L5 is located on the pointer 7a at the destination. When the projection line L6 is dragged, the line segments L2, L4, L5, graphic element D1, and projection line L6 are moved in the XY parallel direction so that the projection line L6 is located on the pointer 7a at the destination. Furthermore, when line segment L2 or line segment L4 is dragged, line segment L2, line segment L4, line segment L5, graphic element D1, and projection line L6 are translated in the XY direction so that line segment L2 or line segment L4 is positioned on pointer 7a at the destination. Furthermore, when a point on graphic element D1 is dragged, line segment L2, line segment L4, line segment L5, graphic element D1, and projection line L6 are translated in the XY direction so that the same point on graphic element D1 is positioned on pointer 7a at the destination.

Even during movement, line segment L2, line segment L4, line segment L5, graphic element D1, and projection line L6 can be drawn. When the user then performs a release operation, line segment L2, line segment L4, line segment L5, graphic element D1, and projection line L6 are fixed in the state after the XY parallel movement.

The left image in Fig. 20 for specific example 2 shows the target area before the position is adjusted, and the right image shows the target area after the position is adjusted. In the left image in Fig. 20, the pointer 7a of the mouse 7 is moved onto line segment L4 (after mouse hover) and then pressed. This causes the color of line segments L2 and L4 to change, making it clear that they have been selected by the user.

When the mouse 7 is then dragged, line segment L4 moves parallel in the XY direction while line segment L2 remains fixed. Line segment L5 and projection line L6 are drawn extending from line segment L2 to the moved line segment L4. Graphic element D1 is also drawn in accordance with the movement of line segment L4. Next, when the user performs a release operation, line segment L4 is fixed in the state after it has moved parallel in the XY direction. This makes it possible to change not only the position of the target area but also the length of the target area.

Figure 21 is a diagram showing a specific example of adjusting the position of a rectangular target area. The diagram on the left side of Figure 21 shows the target area before the position adjustment, and the diagram on the right side shows the target area after the position adjustment. In the diagram on the left side of Figure 21, the pointer 7a of the mouse 7 is moved onto the rectangular frame L7 (after mouse hover), and then pressed. As a result, the colors of the lines L2, L4, L10, and L11 change, making it clear that they have been selected by the user. In addition, the press operation also draws the middle line L9. The middle line L9 is drawn between the lines L2 and L11, between the lines L2 and L10, between the lines L4 and L10, and between the lines L4 and L11, and extends in the Z direction. By dragging the handle (middle point handle) of the middle line L9 with the mouse 7, only one of the width and height of the rectangular frame L7 can be changed. There are also handles at the vertices of rectangular frame L7, and by dragging these vertex handles with mouse 7, both the width and height of rectangular frame L7 can be changed simultaneously. When the user performs a release operation, the width and height of rectangular frame L7 are fixed in the changed state. The handle positions can be set as desired.

(Drawing of effective measurement area)
There may be cases where a part of the workpiece W is an area that does not want to be measured. The process of excluding that area from the measurement target can be called masking, and the area that has been masked can be called a mask area. The area other than the mask area can be called an effective measurement area.

By drawing the effective measurement area on a three-dimensional object, the user can easily distinguish between the mask area and the effective measurement area. When drawing the effective measurement area on a three-dimensional object, the following drawing flow can be applied. First, a binary image is created internally and the normal area is filled with "1" or the like. After that, the mask area is filled with "0" to erase the parts that overlap with the normal area. Next, a boundary image is generated from the binary image. The wall surface is drawn semi-transparently in three dimensions in the boundary part of the boundary image. This reduces the number of semi-transparent objects drawn and makes it easier to see as it can be expressed in a single color.

(Procedure for measuring steps during operation)
FIG. 22 is a flowchart showing an example of a procedure for measuring a step in a linear target area during operation of the image inspection system 1. Before step measurement, the user sets a threshold value for the step height in advance. In step SF1 after starting, 2.5D data of the workpiece W is acquired based on the height data newly acquired by the acquisition unit A. At this time, 2D data may be acquired simultaneously. In step SF2, linear target area data saved in step SC6 shown in FIG. 12 or step SF7 shown in FIG. 7 is set to the 2.5D data acquired in step SF1. This step SF2 is a step executed by the area setting means 406 shown in FIG. 11, and by going through this step, the target area specified by the area designation receiving means 404 can be set for the height data acquired by the acquisition unit A.

In step SF3, a height profile on the target area set by the area setting means 406 in step SF2 is obtained. In step SF4, the height of the upper side and the lower side of the step are measured in the height profile obtained in step SF3. In step SF5, the step height is calculated from the difference between the upper side height and the lower side height. Steps SF3 to SF5 are steps in which the image processing unit 4B shown in FIG. 7 performs image processing on the target area set by the area setting means 406. In step SF6, the step height calculated in step SF5 is compared with a threshold range set in advance, and if the step height calculated in step SF5 falls within the threshold range, it is judged as "good", and if it does not fall within the threshold range, it is judged as "no good". This step SF6 is a step executed by the inspection unit 4A shown in FIG. 7.

In step SF7, the setting device 4 outputs the judgment value of step SF6 to the external device. If the external device is a defective product ejection mechanism, in step SF8, the defective product is ejected from the conveyor belt B by the ejection mechanism.

(Procedure for inspection of defects during operation)
Fig. 23 is a flow chart showing an example of a procedure for inspecting a rectangular target area for defects when the image inspection system 1 is in operation. Prior to the defect inspection, the user sets a threshold value for the amount of defects in advance. In step SG1 after the start, new 2D data is acquired by acquisition unit A. At this time, 2.5D data may also be acquired. In step SG2, rectangular target area data is set in the 2D data acquired in step SG1. This step SG2 is a step executed by area setting means 406 shown in Fig. 11.

In step SG3, the amount of damage within the target area set by the area setting means 406 in step SG2 is obtained using a damage measurement algorithm. The damage measurement algorithm may be a conventional one. This step SG3 is a step in which the image processing unit 4B shown in FIG. 7 performs image processing on the target area set by the area setting means 406.

In step SG4, the amount of damage acquired in step SG3 is compared with a preset threshold range, and if the amount of damage acquired in step SG3 falls within the threshold range, it is judged as "good", and if it does not fall within the threshold range, it is judged as "bad". This step SG4 is a step executed by the inspection unit 4A shown in FIG. 7.

In step SG5, the setting device 4 outputs the judgment value of step SG4 to an external device. If the external device is a defective product ejection mechanism, in step SG6, the defective product is ejected from the conveyor belt B by the ejection mechanism.

(Height inspection procedure during operation)
Fig. 24 is a flow chart showing an example of a procedure for inspecting height in a rectangular target area during operation of the image inspection system 1. Prior to height inspection, the user sets a height threshold in advance. In step SH1 after starting, 2.5D data is acquired similarly to step SF1 in Fig. 22. In step SH2, rectangular target area data is set in the 2.5D data acquired in step SH1. This step SH2 is a step executed by the area setting means 406 shown in Fig. 11.

In step SH3, the average height within the target area set by the area setting means 406 in step SH2 is calculated. This step SH3 is a step in which the image processing unit 4B shown in FIG. 7 performs image processing on the target area set by the area setting means 406. In step SH4, the average height acquired in step SH3 is compared with a threshold range set in advance, and if the average height acquired in step SH3 falls within the threshold range, it is judged as "good", and if it does not fall within the threshold range, it is judged as "bad". This step SH4 is a step executed by the inspection unit 4A shown in FIG. 7.

In step SH5, the setting device 4 outputs the judgment value of step SH4 to the external device. If the external device is a defective product ejection mechanism, in step SH6, the defective product is ejected from the conveyor belt B by the ejection mechanism.

(Expiration date inspection procedure during operation)
25 is a flow chart showing an example of a procedure for inspecting a best-before date in an arc-shaped target area during operation of the image inspection system 1. Before the best-before date inspection, the user sets a date and time threshold in advance. In step SI1 after starting, 2D data is acquired by the acquisition unit A. At this time, 2.5D data may be acquired. In step SI2, arc-shaped target area data is set in the 2D data acquired in step SI1. This step SI2 is a step executed by the area setting means 406 shown in FIG. 11.

In step SI3, the characters within the target area set by the area setting means 406 in step SI2 are extracted. In step SI4, the characters extracted in step SI3 are identified using an optical character recognition (OCR) algorithm. In step SI5, the characters are converted into a date and time. These steps SI3 to SI5 are steps in which the image processing unit 4B shown in FIG. 7 performs image processing on the target area set by the area setting means 406.

In step SI6, the date and time converted in step SI5 is compared with a preset threshold range, and if the date and time converted in step SI5 falls within the threshold range, it is judged as "good", and if it does not fall within the threshold range, it is judged as "bad". This step SI6 is a step executed by the inspection unit 4A shown in FIG. 7.

In step SI7, the setting device 4 outputs the judgment value of step SI6 to an external device. If the external device is a defective product ejection mechanism, in step SI8, the defective product is ejected from the conveyor belt B by the ejection mechanism.

Note that the expiration date is just one example, and other items such as the serial number and manufacturing date may also be subject to inspection.

(Procedure for identifying engravings during operation)
Fig. 26 is a flow chart showing an example of the procedure for identifying markings in an arc-shaped target area when the image inspection system 1 is in operation. Before identifying the markings, the user sets the characters to be engraved on the workpiece W in advance. In step SJ1 after starting, 2.5D data is acquired as in step SF1 in Fig. 22. In step SJ2, the 2.5D data acquired in step SJ1 is converted into 2D data using a height extraction algorithm. In step SJ3, arc-shaped target area data is set in the 2D generated in step SJ2. This step SJ3 is a step executed by the area setting means 406 shown in Fig. 11.

In step SJ4, the characters are extracted from within the target area set by the area setting means 406 in step SJ3. In step SJ5, the characters extracted in step SJ4 are determined using an optical character recognition (OCR) algorithm. These steps SJ4 and SJ5 are steps in which the image processing unit 4B shown in FIG. 7 performs image processing on the target area set by the area setting means 406.

In step SJ6, the character determined in step SJ5 is compared with the character set in advance, and if the character converted in step SJ5 is the same as the measured character, it is determined to be "good", and if they are different, it is determined to be "bad". This step SJ6 is a step executed by the inspection unit 4A shown in FIG. 7.

In step SJ7, the setting device 4 outputs the determination value of step SJ6 to the external device. If the external device is a discharge mechanism for defective products, the discharge mechanism discharges the defective products from the transport belt conveyor B in step SJ8.
(Effects of the embodiment)
As described above, according to this embodiment, when the acquisition unit A acquires height data in which the height information of the workpiece W is arranged on a two-dimensional coordinate system during setting, the object generation means 401 generates a three-dimensional object based on the height data. The generated three-dimensional object can have its posture adjusted while displayed on the display unit 5 of the setting device 4 in a form such as a three-dimensional polygon display or a three-dimensional point cloud display, so that a three-dimensional object in which the workpiece W is viewed from a desired direction can be displayed on the display unit 5. The user can specify a target area on the three-dimensional object after the posture adjustment.

In this way, when specifying the target area, the user can view the three-dimensional object that represents the workpiece W in a three-dimensional manner from the desired direction, so that the user can easily and reliably grasp the height difference even in areas such as small steps. And because the target area can be specified while grasping the height difference of the workpiece W, the user can easily grasp where on the workpiece W the target area has been specified and what size or shape it has.

When a target area is specified, that target area is set to the newly acquired height data during operation. In other words, the information about the target area specified on the three-dimensional object is reflected in the height data during operation, so image inspection is performed on the desired area.

The above-described embodiments are merely illustrative in all respects and should not be interpreted as limiting. Furthermore, all modifications and variations within the scope of the claims are within the scope of the present invention.

As explained above, the present invention can be used, for example, when inspecting workpieces, etc.

1 Image inspection system 2 Lighting device (projector)
4A Inspection unit 4B Image processing unit 5 Display unit (monitor)
31 Camera (image generating unit)
31d Imaging element 300 Imaging device 400 Inspection setting unit 401 Object generating means 402 Display means 403 Attitude adjustment means 404 Area designation receiving means 405 Element generating means 406 Area setting means A Acquisition unit D1 Graphic elements L2, L4 Line segment (viewpoint identification element)
W Work (inspection object)

Claims (11)

An image inspection system including: an image pickup device for image inspection, which has an image pickup element for picking up an image of an object to be inspected, and an acquisition unit for acquiring height data of the object to be inspected, the height data being arranged on a two-dimensional coordinate system using the image pickup element ; and an inspection setting unit for setting various inspection settings based on user operations in a setting device connected to the image pickup device for image inspection via a network,
The test setting unit includes:
an object generating means for generating a three-dimensional object that stereoscopically represents an object to be inspected based on the height data acquired by the acquiring unit;
a display means for displaying the three-dimensional object generated by the object generating means on a monitor of the setting device;
an attitude adjustment means for adjusting an attitude of the three-dimensional object displayed by the display means;
an area designation receiving means for receiving designation of a target area to be subjected to image inspection , the target area being formed along a surface of the three-dimensional object whose orientation has been adjusted by the orientation adjusting means;
an element generating means for generating a viewpoint specifying element for specifying a viewpoint at a position of the imaging device with respect to a three-dimensional object whose surface has a target area specified by the area specification receiving means;
a region setting means for setting the target region designated by the region designation receiving means for the height data acquired by the acquisition unit;
having
The image inspection system according to the present invention, wherein the display means displays the viewpoint-specific elements generated by the element generating means on the monitor . 2. The image inspection system according to claim 1 ,
An image inspection system in which the viewpoint identification element includes a plurality of line segments extending a predetermined length along the imaging axis of the imaging element from each of a plurality of points whose designation is accepted by the area designation accepting means, and a graphical element consisting of a plane or curved surface formed based on the plurality of line segments. 3. The image inspection system according to claim 2 ,
The graphics element is semi-transparent. 4. The image inspection system according to claim 2 ,
An image inspection system, wherein the predetermined length of the plurality of line segments forming the graphic element is determined according to a maximum height or a minimum height in the height image. 5. The image inspection system according to claim 1 ,
An image inspection system, wherein the viewpoint identification element is capable of being adjusted in position on the three-dimensional object after being generated by the element generation means. 6. The image inspection system according to claim 1 ,
An image inspection system, wherein the viewpoint identifying element is capable of changing its shape on the three-dimensional object after being generated by the element generating means. 6. The image inspection system according to claim 5 ,
An image inspection system in which the viewpoint identification element is prohibited from moving in the height direction of the three-dimensional object. 8. The image inspection system according to claim 1,
The area designation receiving means of the image inspection system, when it receives a designation of a start point and an end point on the three-dimensional object, sets a target area at coordinates in the two-dimensional coordinates of the height data that correspond to the position of a line segment extending from the start point to the end point. 9. The image inspection system according to claim 1,
The acquisition unit generates a height image based on the acquired height data,
the display means is configured to be able to switch between displaying the height image generated by the acquisition unit and the three-dimensional object generated by the object generation means on the monitor;
The area designation receiving means is an image inspection system that receives a designation of a target area to be subjected to image inspection on the height image generated by the acquisition unit. 10. The image inspection system according to claim 1 ,
The setting device is an image inspection system that includes an image processing unit that performs image processing on the target area set for height data by the area setting means, and an inspection unit that performs appearance inspection based on the image processed by the image processing unit. The image inspection system according to any one of claims 1 to 10 ,
the setting device is configured to acquire from an external device a file in which a setting item consisting of a target area accepted by the area designation accepting means and register information indicating a location where a setting value of the setting item is stored, and transmit to the image inspection imaging device the setting value of the setting item set by a user and the register information corresponding to the setting item included in the file;
The image inspection system, wherein the imaging device for image inspection stores the setting value of the setting item received from the setting device in a location indicated by register information corresponding to the setting item.

Images