JP2023032090A - Image processing device, image processing method and program - Google Patents

Abstract

To improve efficiency of the confirmation work for obtaining the camera/workpiece positional relation considering a reflection light amount.SOLUTION: An image processing device comprises an estimation unit, an attribute addition unit and an output unit. The estimation unit, on the basis of first model information including information indicating the outline of a first three-dimensional model of a subject, position information indicating a relative positional relation between the imaging unit that targets the subject for imaging and the subject and illumination information indicating characteristics of irradiation light irradiated to the subject, obtains estimation information being information indicating an estimation image obtained by estimating an image of the subject imaged by the imaging unit. The attribute addition unit adds reflection information being information indicating the state of reflection light being light obtained by reflecting the irradiation light by the subject and incident on the imaging unit to first model information on the basis of the estimation information and the first model information, and generates second model information including information indicating the outline of a second three-dimensional model of the subject. The output unit visually outputs the second three-dimensional model on the basis of the observation position for observing the subject.SELECTED DRAWING: Figure 14

Description

The present disclosure relates to image processing.

2. Description of the Related Art An inspection for defects in a component having a three-dimensional shape is performed, for example, by visually inspecting an object to be inspected (hereinafter referred to as an "inspection object" or "workpiece") from various angles. For example, an inspection apparatus that automatically inspects an object to be inspected has been proposed for the purpose of reducing the number of personnel and ensuring quality standards.

For example, an image captured by a camera may be displayed on the screen, and an area in which a portion of the workpiece to be inspected is captured in the image may be specified (for example, Patent Document 1).

For example, by attaching an optical device to the tip of an industrial robot arm having a structure resembling a human arm, a defect inspection apparatus capable of freely imaging a desired portion of an object to be inspected is being considered (for example, , Patent Document 2). According to such a defect inspection apparatus, it is possible to inspect, for example, even a large-sized workpiece that is difficult to be inspected visually by a person.

When the outer shape of the workpiece includes holes or recesses, or depending on the surface condition of the workpiece, even if the workpiece is irradiated with light from the lighting and reflected by the workpiece, the amount of light entering the camera from the workpiece (hereinafter (tentatively referred to as the "reflected light amount") may be insufficient for defect inspection.

In order to increase the amount of reflected light in such a case, it is conceivable to change the relative positional relationship between the camera and the workpiece (hereinafter also referred to as "camera/work positional relationship"). Patent Document 2 discloses a technique for changing the camera/work positional relationship. In Patent Document 3, by specifying an angle that indicates variation in the direction of travel of reflected light with an intensity necessary for inspection and setting a value based on that angle as an allowable value, recognition accuracy is ensured within a range where inspection is possible. Disclosed is a technique for

JP 2015-21764 A JP 2021-52247 A JP-A-2009-8502

In order to obtain a camera/work positional relationship in which the amount of reflected light satisfies the amount of light required for defect inspection, however, in many cases it is necessary to check the amount of reflected light for each camera/work positional relationship by trial and error. Performing such confirmation every defect inspection poses a problem of increasing the number of man-hours required for the defect inspection.

The present disclosure has been made in view of the above problems, and aims to improve the efficiency of confirmation work for obtaining the camera/work positional relationship in consideration of the amount of reflected light.

An image processing apparatus according to a first aspect includes an estimating section, an attribute adding section, and an output section. The estimating unit includes first model information including information indicating an outline of a first three-dimensional model of a subject, and a position indicating a relative positional relationship between the subject and an imaging unit that captures the subject. Based on the information and illumination information indicating the characteristics of the illumination light illuminating the subject, estimation information is obtained, which is information indicating an estimated image obtained by estimating the image of the subject captured by the imaging unit. The attribute addition unit is information indicating a state of reflected light, which is light incident on the imaging unit after the irradiation light is reflected by the subject, based on the estimation information and the first model information. Reflection information is added to the first model information to generate second model information including information indicating an outline of a second three-dimensional model of the subject. The output unit visually outputs the second three-dimensional model based on an observation position at which the subject is observed.

An image processing apparatus according to a second aspect is the image processing apparatus according to the first aspect, wherein the estimation unit obtains a plurality of pieces of estimation information respectively corresponding to a plurality of pieces of position information, and the attribute adding unit adds the reflection information corresponding to each of the plurality of position information to one of the first model information based on the plurality of the estimated information to generate one of the second model information The output unit visually outputs the second three-dimensional model by making the reflection information corresponding to each of the plurality of position information different from each other.

An image processing method according to a third aspect includes first model information including information indicating an outline of a first three-dimensional model of a subject, and relative information indicating an estimated image obtained by estimating the image of the subject captured by the imaging unit based on position information indicating the positional relationship and illumination information indicating the characteristics of the illumination light illuminating the subject a step of obtaining certain estimated information; and information indicating a state of reflected light, which is light incident on the imaging unit after the illumination light is reflected by the subject, based on the estimated information and the first model information. to the first model information to generate a second three-dimensional model of the subject; and based on an observation position for observing the subject, the second three-dimensional model and visually outputting the model.

The image processing apparatus according to the first aspect improves the efficiency of confirmation work for obtaining the camera/work positional relationship in consideration of the amount of reflected light.

The image processing apparatus according to the second aspect contributes to selection of the position of the imaging unit newly set by the operator.

The image processing method according to the third aspect improves the efficiency of confirmation work for obtaining the camera/work positional relationship in consideration of the amount of reflected light.

1 is a perspective view schematically showing the appearance of an inspection system; FIG. 1 is a diagram showing a schematic configuration of an inspection system; FIG. It is a figure showing an example of the main physical composition of an imaging device. FIG. 2 is a diagram showing an example of a physical configuration of an imaging unit; FIG. It is a block diagram showing an example of functional composition of an inspection system. It is a block diagram which shows an example of the functional structure of an injection device. 2 is a block diagram showing an example of a functional configuration of an imaging device; FIG. 3 is a block diagram showing an example of a functional configuration of an imaging unit; FIG. It is a block diagram which shows an example of a functional structure of an inversion apparatus. It is a block diagram showing an example of the functional configuration of the ejection device. 1 is a block diagram showing an example of an electrical configuration of an information processing device according to the present disclosure; FIG. It is a schematic diagram which shows a three-dimensional model. It is a schematic diagram explaining the position of the imaging part in an imaging device. 3 is a block diagram illustrating a functional configuration realized by an arithmetic processing unit; FIG. FIG. 4 is a diagram schematically showing an estimated image; FIG. FIG. 4 is a diagram schematically showing an estimated image; FIG. It is a figure which shows the image with reflection information. It is a figure which shows the image with reflection information. 4 is a flow chart showing an example of the flow of image processing; FIG. 4 is a diagram schematically showing another example of the physical configuration of the imaging unit; It is a figure which shows an example of an illumination part typically. It is a figure which shows an example of an illumination part typically. It is a figure which shows an example of an illumination part typically.

Each embodiment of the present disclosure will be described below with reference to the accompanying drawings. Components described in each embodiment are merely examples, and the scope of the present disclosure is not intended to be limited to them. The drawings are only schematic representations. In the drawings, the dimensions and numbers of each part may be exaggerated or simplified as necessary for easy understanding. In the drawings, parts having similar configurations and functions are denoted by the same reference numerals, and overlapping descriptions are omitted as appropriate.

In this specification, expressions indicating relative or absolute positional relationships (e.g., "parallel", "perpendicular", "center", etc.), unless otherwise specified, not only strictly represent the positional relationship, but also include tolerances. In addition to representing states, we shall also represent states that are displaced relative to each other in terms of angle or distance to the extent that equivalent functionality is obtained. Expressions indicating that two or more things are equal (e.g., "same," "equal," "homogeneous," etc.), unless otherwise specified, not only express quantitatively exact equality, but also It shall also represent the state in which there is a difference in which the degree of function is obtained.

Expressions indicating shapes (e.g., “square shape” or “cylindrical shape”), unless otherwise specified, not only represent the shape strictly geometrically, but also within a range where the same degree of effect can be obtained, such as unevenness or Shapes with chamfers and the like are also represented.

References to "comprise," "comprise," "comprise," "include," or "have" an element are not exclusive expressions that exclude the presence of other elements.

Unless otherwise specified, the expression "connected" includes not only the state in which two elements are in contact, but also the state in which two elements are separated with another element interposed therebetween.

<1. Configuration of inspection system>
An inspection system 1 according to the present disclosure is described with reference to FIGS. 1-10. FIG. 1 is a perspective view schematically showing the appearance of an inspection system 1 according to the present disclosure. FIG. 2 is a diagram showing a schematic configuration of an inspection system 1 according to the present disclosure.

As shown in FIGS. 1 and 2, the inspection system 1 includes, for example, an input device 11, four imaging devices 12, an inverting device 13, and an ejecting device . More specifically, in the inspection system 1, for example, an input device 11, a first imaging device (also referred to as a first imaging device) 121, and a second imaging device (also referred to as a second imaging device) 122 , the reversing device 13, the third imaging device (also referred to as the third imaging device) 123, the fourth imaging device (also referred to as the fourth imaging device) 124, and the ejection device 14 are arranged in the +X direction. They are positioned in a connected state in the order of this description. The input device 11, the imaging device 12, the reversing device 13, and the ejection device 14 are appropriately abbreviated as "devices". Here, for example, the inspection system 1 is manufactured by interconnecting a plurality of separately manufactured devices 11, 12, 13, 14 in the +X direction. For example, an inspection system is manufactured by appropriately combining two or more devices including one or more imaging devices 12 . The devices 11 , 12 , 13 , 14 are interconnected using, for example, connecting members and fastening members such as screws.

Each of the devices 11, 12, 13, 14 has, for example, a tubular portion (also referred to as tubular portion) having an internal space in the upper portion in which the workpiece W0 is placed and transported. This tubular portion is positioned, for example, so as to penetrate in the +X direction. Specifically, for example, the loading device 11 has a tubular portion 11tb, the imaging device 12 has a tubular portion 12tb, the reversing device 13 has a tubular portion 13tb, and the discharging device 14 , and a cylindrical portion 14tb.

For example, one cylindrical portion (cylindrical portion) 1tb forming a route (also referred to as a transfer route) Rt1 through which the workpiece W0 can be transferred among the plurality of apparatuses 11, 12, 13, and 14 is formed. A plurality of devices 11, 12, 13, 14 are interconnected. In FIG. 2, a two-dot chain line arrow is drawn along the transport route Rt1 along the +X direction.

The XYZ coordinate system displayed in the present disclosure is a right-handed system, the +X direction is the direction in which the work W0 is transported along the horizontal plane, and the +Y direction is the direction perpendicular to the direction in which the work W0 is transported along the horizontal plane. , and the direction of gravity orthogonal to both the +X direction and the +Y direction is the -Z direction.

For example, the tubular portion 1tb includes the tubular portion 11tb of the loading device 11, the tubular portion 12tb of the first imaging device 121, the tubular portion 12tb of the second imaging device 122, and the tubular portion 13tb of the reversing device 13. , the tubular portion 12tb of the third imaging device 123, the tubular portion 12tb of the fourth imaging device 124, and the tubular portion 14tb of the ejection device 14 are connected in this order in the +X direction. ing.

For example, the inspection system 1 conveys the work W0 from the loading device 11 to the first imaging device 121, the second imaging device 122, the reversing device 13, the third imaging device 123, the fourth imaging device 124, and the discharging device 14 in this order. Then, the workpiece W0 can be inspected. Here, the upper surface portion positioned in the +Z direction and the side surface portion positioned in the ±Y direction of each of the tubular portions 11tb, 12tb, 13tb, and 14tb may or may not be transparent, for example.

<1-1. Configuration of input device>
The loading device 11 is a device into which the work W0 is loaded from outside the inspection system 1 . This loading device 11 is, for example, positioned first among the plurality of devices included in the inspection system 1 in the transport route Rt1 of the workpiece W0.

In the example of FIGS. 1 and 2, the loading device 11 has a belt conveyor that is a transport unit Cv1 capable of transporting the work W0, and each imaging device 12 has a belt conveyor that is a transport unit Cv2 capable of transporting the work W0. , the reversing device 13 has a belt conveyor that is a transport unit Cv3 capable of transporting the work W0, and the discharge device 14 has a belt conveyor that is a transport unit Cv4 capable of transporting the work W0.

The transport unit Cv1 of the loading device 11 can transport the work W0 between the loading device 11 and the outside of the loading device 11, for example. The transport unit Cv2 of the imaging device 12 can transport the workpiece W0 between the imaging device 12 and the outside of the imaging device 12, for example. The transport section Cv3 of the reversing device 13 can transport the work W0 between the reversing device 13 and the outside of the reversing device 13, for example. The transport unit Cv4 of the discharge device 14 may be capable of transporting the work W0 between, for example, the discharge device 14 and the outside of the discharge device 14, or may transport the work W0 to a predetermined position within the discharge device 14. It may be transportable.

For example, the loading device 11 has an openable/closable portion (also referred to as an opening/closing portion) 11oc at the end of the cylindrical portion 11tb in the -X direction opposite to the imaging device 12 . The opening/closing unit 11oc has, for example, an openable/closable door or shutter. For example, the workpiece W0 can be loaded into the loading device 11 via the opening/closing portion 11oc.

For example, it is conceivable that the operator Op0 loads the workpiece W0 into the loading device 11. FIG. In this case, for example, it is conceivable that the operator Op0 places the workpiece W0 on the belt of the belt conveyor as the transport unit Cv1 in accordance with the mark drawn or projected on the belt of the belt conveyor. In the loading device 11, for example, a sensor that detects the workpiece W0 placed on the belt may detect that the workpiece W0 has been loaded into the loading device 11. FIG.

For example, a robot or the like provided outside the inspection system 1 may place the work W0 on the belt of a belt conveyor, thereby loading the work W0 into the loading device 11 . For example, the transport unit Cv1 transfers the workpiece W0 placed on the belt of the belt conveyor serving as the transport unit Cv1 to the transport unit Cv2 (transport Cv21).

<1-2. Configuration of Imaging Device>
The imaging device 12 performs, for example, imaging as inspection processing using the workpiece W0 as an imaging target (also referred to as an imaging target).

The first imaging device 121 can, for example, image the workpiece W0 transferred from the conveying unit Cv1 of the loading device 11 to the conveying unit Cv21 of the first imaging device 121 as an inspection process. The workpiece W0 imaged by the first imaging device 121 is transported from the first imaging device 121 by the transporting part Cv21 to the transporting part Cv2 of the second imaging device 122 located outside the first imaging device 121 in the +X direction. (also referred to as transport unit Cv22).

For example, the second imaging device 122 can perform imaging as an inspection process for the workpiece W0 transferred from the transportation unit Cv21 of the first imaging device 121 to the transportation unit Cv22 of the second imaging device 122. can. For example, the workpiece W0 imaged by the second imaging device 122 is received from the second imaging device 122 by the transportation section Cv22 to the transportation section Cv3 of the reversing device 13 located outside the second imaging device 122 in the +X direction. Passed.

The third imaging device 123 performs, for example, the work W0 transferred from the transporting unit Cv3 of the reversing device 13 to the transporting unit Cv2 (also referred to as the transporting unit Cv23) of the third imaging device 123 as an inspection process. Imaging can be performed. The workpiece W0 imaged by the third imaging device 123 is transported from the third imaging device 123 by the transporting part Cv23 to the transporting part Cv2 of the fourth imaging device 124 located outside the third imaging device 123 in the +X direction. (also referred to as transport unit Cv24).

For example, the fourth imaging device 124 can perform imaging as an inspection process for the workpiece W0 transferred from the transportation unit Cv23 of the third imaging device 123 to the transportation unit Cv24 of the fourth imaging device 124. can. The workpiece W0 imaged by the fourth imaging device 124 is received by the transportation section Cv24 from the fourth imaging device 124 to the transportation section Cv4 of the discharging device 14 located outside the fourth imaging device 124 in the +X direction, for example. Passed.

FIG. 3 is a diagram illustrating an example of the main physical configuration of the imaging device 12 according to the present disclosure. FIG. 4 is a diagram illustrating an example physical configuration of an imaging unit 12s according to the present disclosure. Here, the first imaging device 121, the second imaging device 122, the third imaging device 123, and the fourth imaging device 124 each have the same configuration.

As shown in FIG. 3, the imaging device 12 has, for example, a transport section Cv2, an imaging unit 12s, and a moving mechanism 12t.

The transport unit Cv2 has a function as a portion (also referred to as a placement unit) Sg2 for placing a workpiece W0 as an object to be imaged, for example.

The imaging unit 12s can, for example, perform imaging of a workpiece W0 as an object to be imaged. In the present disclosure, imaging device 12 has one imaging unit 12s. Specifically, as shown in FIG. 4, the imaging unit 12s has, for example, an imaging section I1 and an illumination section F1.

The imaging unit I1 has, for example, an imaging device such as a charge coupled device (CCD), and a lens unit Lz1 as an optical system for forming an optical image of the workpiece W0 on the imaging device.

For example, planar lighting in which a plurality of light emitting diodes (LEDs) are two-dimensionally arranged is applied to the lighting unit F1. In this case, for example, the work W0 can be illuminated over a wide range by the illumination unit F1.

In the example of FIG. 4, the lens portion Lz1 is positioned while being inserted through the hole portion H1 of the illumination portion F1. From another point of view, the optical axis Pi1 of the lens portion Lz1 is set to pass through the hole portion H1. Specifically, the lens portion Lz1 of the imaging portion I1 is positioned while being inserted through the hole portion H1 of the illumination portion F1. From another point of view, the optical axis Pi1 of the lens portion Lz1 is set to pass through the hole portion H1. As a result, for example, the imaging unit I1 can capture an image of at least a portion of the work W0 illuminated by the illumination unit F1 as a subject.

The moving mechanism 12t can, for example, move the imaging unit 12s relative to the workpiece W0 placed on the placement part Sg2. From another point of view, the moving mechanism 12t can, for example, move the imaging unit 12s relative to the placement section Sg2 for placing the work W0.

In this disclosure, the imaging device 12 has one moving mechanism 12t. Specifically, as shown in FIG. 3, the moving mechanism 12t can, for example, move the imaging unit 12s1 relative to the workpiece W0 placed on the placement part Sg2. From another point of view, the moving mechanism 12t can, for example, move the imaging unit 12s relative to the placement section Sg2 for placing the work W0.

If such a configuration is adopted, for example, the workpiece W0 illuminated by the illumination unit F1 can be imaged by the imaging unit I1.

For example, a robot arm or the like is applied to the moving mechanism 12t. The robot arm is, for example, a robot arm that can be driven on the basis of many axes (also referred to as a multi-axis robot arm). Here, for example, the moving mechanism 12t may be a multi-axis robot arm. Thus, for example, if a multi-axis robot arm is applied to the moving mechanism 12t, each part of the workpiece W0 can be imaged from multiple angles.

The multi-axis robot arm includes, for example, a reference portion Pt0, a first movable portion Pt1, a second movable portion Pt2, a third movable portion Pt3, a fourth movable portion Pt4, a fifth movable portion Pt5, and a third movable portion Pt5. A robot arm (also referred to as a 6-axis robot arm) having 6 movable parts Pt6 and capable of rotating about 6 axes is applied. In this case, the reference part Pt0 is fixed to the base part Bs12 of the imaging device 12, for example. For example, the belt conveyor of the transport section Cv2 may be fixed to the base section Bs12.

The reference portion Pt0 has, for example, a rotating portion Pr1 that rotatably holds the first movable portion Pt1 about the first axis Pl1 along the +Z direction. The first movable part Pt1 has, for example, a second rotating part Pr2 that rotatably holds the second movable part Pt2 around a second axis Pl2 along the horizontal direction. The second movable part Pt2 has, for example, a third rotating part Pr3 that rotatably holds the third movable part Pt3 around a third axis Pl3 along the horizontal direction. The third movable part Pt3 has, for example, a fourth rotating part Pr4 that rotatably holds the fourth movable part Pt4 around a fourth axis Pl4 perpendicular to the third axis Pl3. The fourth movable part Pt4 has, for example, a fifth rotating part Pr5 that rotatably holds the fifth movable part Pt5 about a fifth axis Pl5 perpendicular to the fourth axis Pl4. The fifth movable part Pt5 has, for example, a sixth rotating part Pr6 that rotatably holds the sixth movable part Pt6 around a sixth axis Pl6 perpendicular to the fifth axis Pl5. The imaging unit 12s is fixed to the sixth movable portion Pt6.

For example, the imaging unit 12s may be capable of imaging the entire circumference of the workpiece W0 placed on the placement part Sg2. In this case, for example, each portion of the work W0 can be easily imaged from a plurality of angles with respect to a wider area of the surface of the work W0. Here, in a mode in which the image of the entire circumference of the work W0 can be captured, for example, the work W0 placed on the mounting part Sg2 is set to a virtual axis along the vertical direction (Z direction) passing through the work W0. A mode in which imaging is possible from any angle of 360 degrees in the circumferential direction around the center is included.

For example, the work W0 placed on the mounting part Sg2 can be imaged from any angle in the circumference except for the side of the mounting part Sg2. good.

A plurality of moving mechanisms 12t and imaging units 12s, for example, a pair of each, may be provided (see Patent Document 2, for example). In that case, for example, a pair of moving mechanisms 12t are provided so as to sandwich the transport section Cv2 as the mounting section Sg2. As a result, for example, for a wider area of the surface of the work W0, each portion of the work W0 can be easily imaged under a plurality of illumination conditions. For example, of the light emitted from the illumination section F1 of one imaging unit 12s, the light reflected on the workpiece W0 may be imaged by the other imaging unit 12s.

<1-3. Configuration of reversing device>
The reversing device 13 can, for example, reverse the work W0. Reversing the work W0 includes, for example, reversing the work W0 upside down. Here, the reversing device 13 can, for example, reverse the work W0 transferred from the conveying unit Cv22 to the conveying unit Cv3 of the second imaging device 122 .

The reversing device 13 includes, for example, a holding portion 13h (see FIG. 9) capable of holding the work W0 in order to invert the work W0, and a holding portion 13h in a state where the work W0 is held by the holding portion 13h. and a moving mechanism 13t (see FIG. 9) capable of reversing the work W0 by moving the work W0.

For example, a hand having two or more fingers capable of holding the workpiece W0 is applied to the holding portion 13h. A robot arm or the like is applied to the moving mechanism 13t of the reversing device 13, for example, like the moving mechanism 12t of the imaging device 12.

The reversing device 13 may have, for example, one moving mechanism 13t for one holding portion 13h. More specifically, the reversing device 13 may have one moving mechanism 13t if, for example, one holding portion 13h exists, or if two or more holding portions 13h exist. may have two or more moving mechanisms 13t.

<1-4. Configuration of discharge device>
The discharging device 14 is, for example, a device for discharging the work W0 from the inside of the inspection system 1 to the outside of the inspection system 1 . This discharging device 14 is located at the last part of the transport path Rt1 of the workpiece W0 among two or more modules including one or two or more imaging devices 12 included in the inspection system 1, for example. The discharging device 14 has, for example, an openable/closable portion (opening/closing portion) 14oc at the +X direction end of the cylindrical portion 14tb on the side opposite to the imaging device 12 (see FIG. 1). The opening/closing unit 14oc has, for example, an openable/closable door or shutter. The work W0 is discharged to the outside of the discharging device 14 via the opening/closing part 14oc, for example. For example, a mode in which the operator Op0 discharges the work W0 to the outside of the discharge device 14 is conceivable. For example, a robot or the like provided outside the inspection system 1 may eject the workpiece W0 from the inside of the ejection device 14 to the outside.

<1-5. Functional Configuration of Inspection System>
5 to 10 are block diagrams showing examples of functional configurations of the inspection system 1 according to the present disclosure. FIG. 5 is a block diagram schematically showing an example of the overall functional configuration of the inspection system 1. As shown in FIG. FIG. 6 is a block diagram showing an example of the functional configuration of the loading device 11 in the overall functional configuration of the inspection system 1. As shown in FIG. FIG. 7 is a block diagram showing an example of the functional configuration of the imaging device 12 in the overall functional configuration of the inspection system 1. As shown in FIG. FIG. 8 is a block diagram showing an example of the functional configuration of the imaging unit 12s in the functional configuration of the imaging device 12. As shown in FIG. FIG. 9 is a block diagram showing an example of the functional configuration of the reversing device 13 in the overall functional configuration of the inspection system 1. As shown in FIG. FIG. 10 is a block diagram showing an example of the functional configuration of the ejection device 14 in the overall functional configuration of the inspection system 1. As shown in FIG.

<1-6. Functional Configuration of Inserting Device>
As shown in FIG. 6, the loading device 11 includes, for example, an integrated control unit C0, an input unit 11i, a transport control unit Cc1, and a connection unit 11h, which are electrically connected via a wiring Wr1. have. The loading device 11 has, for example, an output unit 11d connected to the integrated control unit C0, and a transport unit Cv1 connected to the transport control unit Cc1.

The general control unit C0 can, for example, centrally control the operation of the inspection system 1 . The integrated control unit C0 has, for example, a calculation unit, a memory, a storage unit, and the like. The computing unit is configured by, for example, one or more Central Processing Units (CPUs). The memory is composed of, for example, a volatile storage medium such as RAM (Random Access Memory). The storage unit is configured by, for example, a non-volatile storage medium such as a hard disk drive (HDD) or a solid state drive (SSD). The storage unit can store, for example, programs and various types of information. The arithmetic unit can realize various functions by reading and executing a program stored in the storage unit, for example. At this time, the RAM is used as, for example, a workspace, and stores information that is temporarily generated or acquired. At least part of the functional configuration realized by the general control unit C0 may be realized by hardware such as a dedicated electronic circuit, for example.

The input unit 11i can input various information, for example, in response to the operation of the operator Op0. For the input unit 11i, for example, an operation unit such as buttons or a touch panel, or a microphone unit capable of voice input, or the like is applied. The actions of the operator Op0 include, for example, actions such as manipulation and vocalization.

The output unit 11d can output information in a manner recognizable by the operator Op0, for example, based on the information from the integrated control unit C0. For the output unit 11d, for example, a display unit or a lamp for visually outputting information, a speaker for audibly outputting information, or the like is applied.

The transport controller Cc1 can, for example, control the operation of the transport unit Cv1. The transport control unit Cc1 has, for example, a configuration similar to that of a computer including an arithmetic unit, a memory and a storage unit. The transport control unit Cc1 can implement the functions of the transport control unit Cc1 by, for example, executing a program in the storage unit in the arithmetic unit. The transport control unit Cc1 can control the operation of the transport unit Cv1 by controlling the operation of a driving unit such as a motor that rotates at least one pulley in the belt conveyor. At least part of the functional configuration implemented by the transport control unit Cc1 may be implemented by hardware such as a dedicated electronic circuit, for example.

The connection part 11h is a part electrically connected to, for example, a device other than the loading device 11 among the plurality of devices constituting the inspection system 1 . The connecting part 11h may, for example, be of a hub type in which the wirings Wr2 of a plurality of devices are electrically connected separately, or of a type in which the wirings Wr2 of the plurality of devices are electrically connected in series. There may be.

<1-7. Functional Configuration of Imaging Device>
Here, a case where the first imaging device 121, the second imaging device 122, the third imaging device 123, and the fourth imaging device 124 have the same functional configuration is illustrated.

As shown in FIG. 7, the imaging device 12 has, for example, a transport controller Cc2, an imaging controller Cs2, and a movement controller Ct2, which are electrically connected to each other via wiring Wr2. The imaging device 12 has, for example, a transportation section Cv2 connected to the transportation control section Cc2, an imaging unit 12s connected to the imaging control section Cs2, and a movement mechanism 12t connected to the movement control section Ct2.

In the example of FIG. 7, the imaging device 12 has an imaging unit 12s, an imaging controller Cs2, a movement controller Ct2, and a movement mechanism 12t. The imaging unit 12s is connected to the imaging control section Cs2, and the movement mechanism 12t is connected to the movement control section Ct2. The imaging unit 12s, as shown in FIG. 8, includes an imaging section I1 and an illumination section F1.

Each of the transport control unit Cc2, the imaging control unit Cs2, and the movement control unit Ct2 has a configuration similar to that of a computer including, for example, a calculation unit, a memory, and a storage unit.

The transport control unit Cc2 can realize the functions of the transport control unit Cc2, for example, by executing the program in the storage unit in the arithmetic unit. The transport control unit Cc2 can control the operation of the transport unit Cv2 by controlling the operation of a driving unit such as a motor that rotates at least one pulley in the belt conveyor. At least part of the functional configuration realized by the transport control section Cc2 may be configured by hardware such as a dedicated electronic circuit, for example.

The image capturing control unit Cs2 can implement the functions of the image capturing control unit Cs2 by, for example, executing a program in the storage unit in the arithmetic unit. The imaging control section Cs2 can, for example, control the operation of the imaging unit 12s and acquire information (also referred to as imaging information) obtained by imaging the workpiece W0 with the imaging unit 12s. Specifically, the imaging control unit Cs2 can, for example, control timings of light emission by the lighting unit F1 and imaging by the imaging unit I1. For example, the imaging unit 12s can be operated by a control signal from the overall control section C0 (see FIG. 6) to the imaging control section Cs2.

The imaging control unit Cs2 can, for example, output imaging information related to the workpiece W0 as it is or after performing various types of information processing to the overall control unit C0 via the wiring Wr2, the wiring Wr1, and the like. Thereby, imaging information as a result of imaging the work W0 as a target can be obtained. Here, for example, the overall control unit C0 may cause the output unit 11d to display an image based on the imaging information, and the operator Op0 may visually inspect the image to inspect the appearance of the work W0. may perform an operation for inspecting the appearance of the work W0 by comparing an image related to the imaging information with a standard image related to at least a part of the work W0. Here, for example, the imaging control unit Cs2 may perform calculations for inspecting the appearance of the workpiece W0 and send information indicating the results of the calculations to the overall control unit C0. At least part of the functional configuration realized by the imaging control unit Cs2 may be configured by hardware such as a dedicated electronic circuit, for example.

The movement control unit Ct2 can realize the function of the movement control unit Ct2 by, for example, executing the program in the storage unit in the calculation unit. The movement controller Ct2 can control the movement of the movement mechanism 12t, for example. Here, for example, the movement control unit Ct2 moves the relative position of the imaging unit 12s with respect to the workpiece W0 placed on the placement unit Sg2, thereby capturing images of a plurality of locations of the workpiece W0 with the imaging unit 12s. can be controlled to do so. At least part of the functional configuration realized by the movement control unit Ct2 may be configured by hardware such as a dedicated electronic circuit, for example.

<1-8. Functional configuration of reversing device>
As shown in FIG. 9, the reversing device 13 has, for example, a transport control section Cc3 and a reversing control section Cr3 electrically connected to each other via a wiring Wr2. The reversing device 13 has, for example, a conveying section Cv3 connected to the conveying control section Cc3, and a holding section 13h and a moving mechanism 13t connected to the reversing control section Cr3. In the example of FIG. 6, the reversing device 13 has one reversing controller Cr3. Here, for example, the reversing device 13 may have two holding portions 13h, two moving mechanisms 13t, and two reversing control portions Cr3.

Each of the transport control section Cc3 and the reverse control section Cr3 has a configuration similar to that of a computer including, for example, a calculation section, a memory, and a storage section.

The transport control unit Cc3 can realize the functions of the transport control unit Cc3 by, for example, executing the program in the storage unit by the calculation unit. The transport control unit Cc3 can control the operation of the transport unit Cv3, for example, by controlling the operation of a driving unit such as a motor that rotates at least one pulley in the belt conveyor. At least part of the functional configuration realized by the transport control section Cc3 may be configured by hardware such as a dedicated electronic circuit, for example.

The inversion control section Cr3 can realize the function of the inversion control section Cr3 by, for example, executing the program in the storage section in the calculation section. The reversal control section Cr3 can, for example, control the operations of the holding section 13h and the moving mechanism 13t. Specifically, for example, the reversing control section Cr3 causes the holding section 13h to hold the work W0 by pinching or the like, and moves the holding section 13h holding the work W0 by the moving mechanism 13t so that the work W0 is held by the holding section 13h. can be flipped upside down. As a result, for example, the first imaging device 121 and the second imaging device 122 perform imaging of the front surface of the work W0, and the third imaging device 123 and the fourth imaging device 124 perform imaging of the work W0. It is possible to perform imaging targeting the back surface.

<1-9. Functional Configuration of Ejection Device>
As shown in FIG. 10, the discharging device 14 has, for example, a transport control section Cc4 electrically connected to the wiring Wr2 and a transport section Cv4 connected to the transport control section Cc4. The transport control unit Cc4 has, for example, a configuration similar to that of a computer including a calculation unit, a memory, and a storage unit. The transport control unit Cc4 can implement the functions of the transport control unit Cc4 by, for example, executing the program in the storage unit in the arithmetic unit. The transport control unit Cc4 can control the operation of the transport unit Cv4, for example, by controlling the operation of a driving unit such as a motor that rotates at least one pulley in the belt conveyor. At least part of the functional configuration realized by the transport control section Cc4 may be configured by hardware such as a dedicated electronic circuit, for example.

<1-10. Summary of Configuration of Inspection System 1>
As described above, according to the imaging device 12, for example, the imaging unit I1 can image the workpiece W0 illuminated by the illumination unit F1 while moving the imaging unit 12s with respect to the workpiece W0. Thereby, for example, each portion of the work W0 can be imaged from a plurality of angles. Therefore, for example, it is possible to easily capture an image that sufficiently captures the work W0 having various shapes.

<2. Information processing device>
<2-1. Schematic Configuration of Information Processing Apparatus>
FIG. 11 is a block diagram showing an example of the electrical configuration of the information processing device 2 according to the present disclosure. As shown in FIG. 11, the information processing device 2 is implemented by, for example, a computer. The information processing device 2 includes, for example, a communication section 21, an input section 22, an output section 23, a storage section 24, a control section 25 and a drive 26, which are connected via a bus line 2b.

The communication unit 21 has a function of performing data communication with an external device, for example, via a communication line. This communication unit 21 can receive, for example, a program 24p and various data 24d.

The input unit 22 has, for example, a function of accepting input of information in response to actions of an operator who uses the information processing device 2 . The input unit 22 may include, for example, an operation unit, a microphone, and various sensors. The operation unit can include, for example, a mouse and keyboard that can input signals according to the operator's operation. The microphone can input, for example, a signal corresponding to the operator's voice. Various sensors can input, for example, signals corresponding to the movements of the operator.

The output unit 23 has, for example, a function of outputting various types of information in a manner recognizable by the operator. The output unit 23 can include, for example, a display unit, a projector, and a speaker. The display unit can, for example, visibly output various types of information in a manner recognizable by the operator. For example, a liquid crystal display or an organic EL display can be applied to the display section. The display section may have the form of a touch panel integrated with the input section 22 . The projector can, for example, visibly output various types of information onto a projection object exemplified on a screen in a manner recognizable by the operator. The projector and the object to be projected can work together to function as a display section that visibly outputs various types of information in a manner recognizable by the operator. The speaker can, for example, audibly output various types of information in a manner recognizable by the operator.

The storage unit 24 has, for example, a function of storing various kinds of information. This storage unit 24 may be configured by a non-volatile storage medium such as a hard disk or flash memory, for example. In the storage unit 24, for example, any of a configuration having one storage medium, a configuration having two or more storage media integrally, and a configuration having two or more storage media divided into two or more parts is adopted. may be The storage unit 24 can store, for example, a program 24p and various data 24d.

The various data 24d may include model information, position and orientation information, and lighting information. The model information is, for example, information relating to a three-dimensional model (also referred to as a three-dimensional model) Wm of the inspection object W0. The position and orientation information is, for example, information relating to the positions and orientations of the imaging unit I1 and the inspection object W0 in the imaging device 12 . The information includes the camera/work positional relationship. The illumination information is, for example, information relating to the characteristics of the light emitted by the illumination unit F1, such as its intensity distribution. For example, the intensity distribution is based on the optical axis Pi1.

The various data 24d may include, for example, information (also referred to as imaging parameter information) relating to parameters exemplified by the angle of view and focal length that define the area that can be imaged by each imaging unit I1.

Design data (also referred to as object design data) on the three-dimensional shape of the inspection object W0 is applied to the model information, for example. For the object design data, for example, data expressing the three-dimensional shape of the inspection object W0 by a plurality of planes exemplified by a plurality of polygons is applied.

This data includes, for example, data defining the position and orientation of each plane and the reflection coefficient for light. As the plurality of planes, for example, rectangular planes and triangular planes are applied. For the data defining the position of each plane, for example, coordinate data of three or more vertices defining the outer shape of the plane is applied. The data defining the orientation of each plane is, for example, data of a vector (also referred to as normal vector) indicating the direction in which the normal to the plane extends (also referred to as normal direction).

FIG. 12 is a schematic diagram showing the three-dimensional model Wm. It can be said that the contour of the three-dimensional model Wm is a model of the contour of the workpiece W0. In the model information, as shown in FIG. 12, the position and orientation of the three-dimensional model Wm are set, for example, with the position corresponding to the reference position of the area in which the inspection object W0 is arranged in the imaging device 12 as the origin. It can be shown using an xyz coordinate system (three-dimensional model coordinate system).

Specifically, for example, the position of the three-dimensional model Wm is indicated by x-coordinates, y-coordinates, and z-coordinates, and the orientation of the three-dimensional model Wm is indicated by one position vector M set for the three-dimensional model Wm. . In FIG. 12 and the following description, the case where the direction of the position vector M is parallel to the y-axis and its starting point coincides with the origin O of the xyz coordinate system is exemplified.

The position and orientation information includes, for example, the relative position and orientation relationship between the imaging unit I1 of the imaging device 12 and the inspection object W0 placed on the imaging device 12 (hereinafter referred to as "relative position and orientation relationship"). ) may apply.

FIG. 13 is a schematic diagram for explaining the position of the imaging section I1 in the imaging device 12. As shown in FIG. In FIG. 13, the imaging unit I1 is positioned such that the x'-axis, y'-axis, and z'-axis parallel to the X-axis, Y-axis, and Z-axis shown in FIGS. It can be represented by an x'y'z' coordinate system (camera coordinate system) with the origin Q as the reference position of the region to be captured. For example, the position of the imaging unit I1 at the position P1 is represented by (x1′, y1′, z1′), and the position of the imaging unit I1 at the position P2 is represented by (x2′, y2′, z2′). expressed.

To simplify the explanation, the x-axis, y-axis, z-axis and the origin O of the three-dimensional model coordinate system coincide with the x'-axis, y'-axis, z'-axis and the origin Q of the camera coordinate system, respectively. A case is exemplified. Even if the origins O and Q deviate, the x-axis and x'-axis are non-parallel, the y-axis and y'-axis are non-parallel, and the z-axis and z'-axis are non-parallel. Even if there is, it can be reduced to coincidence between the coordinate systems described above by appropriate coordinate transformation.

The position and orientation information includes, for example, information on the position of the imaging unit I1 in the imaging device 12 (x', y', and z' coordinates with respect to the origin Q in the above example), and the position of each part in the three-dimensional model Wm (the above (x-, y-, and z-coordinates relative to the origin O) may be included.

The position and orientation information includes, for example, information on the position of the origin Q of the x'y'z' coordinate system in the xyz coordinate system, the x'-axis direction, the y'-axis direction, and the z'-axis direction, or x'y' Information on the position of the origin O of the xyz coordinate system in the z' coordinate system and the directions of the x, y, and z axes can be included.

For example, the optical axis Pi1 of the lens unit Lz1 of the imaging unit I1 is set in a direction toward the origin Q from the position of the imaging unit. Specifically, when the imaging unit I1 is arranged at a position Pk (k=1, 2, . , is set to a straight line connecting the position Pk and the origin Q.

Referring to FIG. 11, control unit 25 includes, for example, arithmetic processing unit 25a that functions as a processor and memory 25b that can temporarily store information. An electric circuit such as a central processing unit (CPU), for example, is applied to the arithmetic processing unit 25a. In this case, the arithmetic processing unit 25a has, for example, one or more processors. A random access memory (RAM), for example, is applied to the memory 25b.

In the arithmetic processing unit 25a, for example, a program 24p stored in the storage unit 24 is read and executed. Thereby, the information processing device 2 can function as a device (also referred to as an image processing device) 200 that performs various image processing, for example. In other words, the information processing device 2 can function as the image processing device 200 by executing the program 24p by the arithmetic processing unit 25a included in the information processing device 2, for example.

For example, the image processing apparatus 200 captures an image (also referred to as a captured image) that can be obtained by imaging the inspection object W0 at a predetermined angle with the imaging device 12 of the inspection system 1, and performs inspection of the inspection object W0. Information (also referred to as area specification information) that specifies an area (also referred to as an inspection image area) in which the target portion is expected to be captured can be created.

For example, in the image processing apparatus 200, area designation information is created before continuous inspection is performed on a plurality of inspection objects W0 based on the same design, or at the initial stage of continuous inspection. Alternatively, area designation information is created before or during inspection of one or more inspection objects W0.

Various information temporarily obtained by various information processing in the control unit 25 can be appropriately stored in the memory 25b or the like.

The drive 26 is, for example, a removable part of the portable storage medium 26m. In the drive 26, for example, data can be exchanged between the storage medium 26m and the controller 25 while the storage medium 26m is mounted. Here, for example, the program 24p may be read from the storage medium 26m into the storage unit 24 and stored by loading the storage medium 26m storing the program 24p into the drive 26 .

For example, when the storage medium 26m storing the various data 24d or part of the various data 24d is loaded into the drive 26, the various data 24d or part of the various data 24d is transferred from the storage medium 26m into the storage unit 24. data may be read and stored. A part of the various data 24d may include, for example, model information or position/orientation information.

<2-2. Functional Configuration of Image Processing Apparatus>
FIG. 14 is a block diagram illustrating a functional configuration realized by the arithmetic processing section 25a. FIG. 14 illustrates various functions related to data processing realized by executing the program 24p in the arithmetic processing unit 25a.

As shown in FIG. 14, the arithmetic processing unit 25a includes, for example, a dividing unit 251, an estimating unit 252, an attribute adding unit 253, an output control unit 254, and a setting unit 255. and For example, the memory 25b is used as a workspace for the processing of these units. At least part of the functional configuration realized by the arithmetic processing unit 25a may be configured by hardware such as a dedicated electronic circuit, for example.

<2-2-1. Division part>
The model information Wi includes information indicating the contour of the three-dimensional model Wm (hereinafter also referred to as "contour information"). The division unit 251 divides the outline of the three-dimensional model Wm into a plurality of areas (hereinafter “divided areas”) based on the outline information, for example, and generates information (hereinafter “divided information”) S indicating the divided areas. have the ability to obtain The classification information S is used in a process of obtaining information (hereinafter "estimated information") G indicating an image obtained by estimating the captured image (hereinafter "estimated image") (hereinafter "captured image estimation process"). . The division information S is accompanied by information about the reflection coefficient included in the model information Wi.

For example, model information Wi stored as various data 24 d in the storage unit 24 is input to the dividing unit 251 .

The processing for obtaining the segmented area is performed, for example, in two stages. In the first step, the surface of the three-dimensional model Wm is segmented into a plurality of regions based on information regarding the orientations of the plurality of planes forming the three-dimensional model Wm. For example, the normal vector of the plane is used as the information related to the direction. In the second step, the surface of the three-dimensional model Wm divided into a plurality of regions obtained in the first step is further divided into a plurality of regions based on the connection state of the planes of the plurality of planes constituting the three-dimensional model Wm. , and a partitioned area is obtained.

<2-2-2. Estimation part>
Specifically, the estimation unit 252 performs captured image estimation processing of obtaining estimation information G from the division information S and the position and orientation information R. FIG. The estimated image is an image obtained by estimating the captured image when the imaging unit I1 captures the inspection object W0 represented by the three-dimensional model Wm.

For example, regarding the imaging unit I1, parameters (position and orientation parameters) related to the position and orientation of the three-dimensional model Wm in a coordinate system (camera coordinate system) using the x'-axis, y'-axis, and z-axis', and the model information Wi , and the illumination information L, the estimated image, more specifically, the estimated information G is generated.

The illumination information L and the position and orientation information R are stored as various data 24 d in the storage unit 24 , for example, and input to the estimation unit 252 . Since such captured image estimation processing is a known technique, a brief description will be given below.

For example, the position of the segmented area in the xyz coordinate system (three-dimensional model coordinate system) is transformed into the position in the x'y'z' coordinate system (camera coordinate system) according to the position and orientation parameters. As illustrated in FIG. 4, the lighting unit F1 and the imaging unit I1 are in a fixed positional relationship, so the position of the lighting unit F1 is also transformed according to the position and orientation information R for each position of the imaging unit I1. Accordingly, the illumination information L is also converted into the x'y'z' coordinate system for each position of the imaging unit I1 according to the position/orientation information R.

In this manner, the estimation unit 252 obtains the position of the imaging unit I1 (and thus the direction of the optical axis Pi1), the position and reflection coefficient of the segmented area, and the intensity distribution of the light from the illumination unit F1 in the camera coordinate system. An estimated image is obtained by a well-known three-dimensional simulation using At this time, for example, imaging parameters are also used.

FIG. 15 is a diagram schematically showing an estimated image G1 that is estimated to be captured when the imaging unit I1 at the position P1 captures an image of the workpiece W0. FIG. 16 is a diagram schematically showing an estimated image G2 that is estimated to be captured when the imaging unit I1 at the position P2 captures an image of the workpiece W0.

In the estimated images G1 and G2, it is assumed that only a part of the work W0 is imaged, not the entire work W0. In the estimated images G1 and G2, except for the background of the three-dimensional model Wm, the greater the amount of reflected light, the higher the brightness.

<2-2-3. Attribute addition section>
The attribute addition unit 253 adds reflection information to the three-dimensional model Wm to generate a three-dimensional model Wd (see FIGS. 17 and 18 to be described later) of the workpiece W0. The reflection information is information indicating the state of reflection, for example, the amount of reflected light, in a portion of the outer contour of the work W0 that is estimated to be imaged by the imaging unit I1 (hereinafter “estimated imaging area”). The three-dimensional model Wd includes information indicating the outline of the workpiece W0 and reflection information.

Reflection information is obtained from the estimated information G. A well-known three-dimensional simulation obtains the amount of reflected light for each segmented area in the estimated image. Reflection information is added to the three-dimensional model Wm for each segmented area. The reflection state is added to the three-dimensional model Wm as one of the attributes together with the attribute of the segmented area, such as the position and orientation relative to the position vector M of the plane used for the segmented area.

A segmented area other than the estimated imaging area corresponds to an area of the work W0 at a position that is not estimated to be imaged. In such an area of the workpiece W0, the amount of reflected light is substantially 0, and for example, the reflection information is set to 0 and given to the divided areas.

A segmented area of the estimated imaging area corresponds to an area of the workpiece W0 at a position estimated to be imaged. In such a region of the workpiece W0, the amount of reflected light takes various values. Reflection information in a segmented area corresponding to such an area also takes various values according to the amount of reflected light.

In order to perform the above operation, the attribute addition unit 253 is supplied with the model information Wi, the classification information S, and the estimation information G, and the attribute addition unit 253 uses these to generate the three-dimensional model Wd, and the model information Wj. Output. The model information Wj is information indicating the three-dimensional model Wd.

Since the reflection information depends on the position of the imaging unit I1, it is set for each position of the imaging unit I1, for example. According to the above example, a three-dimensional model Wd is generated for each of the estimated images G1 and G2.

<2-2-4. Output control unit>
The output control unit 254 causes, for example, the output unit 23 to output various types of information in a manner recognizable by the operator. The output unit 23 visually outputs the three-dimensional model Wd as follows.

Observation position information T is input to the input unit 22 by, for example, an operator. The observation position information T is information indicating the observation position where the three-dimensional model Wd is observed.

The observation position may be represented in the observation position information T using, for example, an xyz coordinate system (three-dimensional model coordinate system) or an x'y'z' coordinate system (camera coordinate system). The observation position does not have to be associated with the position of the imaging unit I1.

The model information Wj and the observation position information T are input to the output control unit 254 . The output control section 254 generates image information Gs based on the model information Wj and the observation position information T, and outputs the image information Gs to the output section 23 . The image information Gs is information about the visual output of the three-dimensional model Wd viewed from the viewing position specified by the viewing position information T. FIG.

Based on the image information Gs, the output unit 23 outputs the three-dimensional model Wd visually, specifically as an image (hereinafter, the image is an "image with reflection information"). For example, the output unit 23 has a liquid crystal display, an image with reflection information is displayed on the liquid crystal display, and the operator visually recognizes the three-dimensional model Wd from the image with reflection information.

A three-dimensional model Wd visually recognized in the image with reflection information indicates the amount of reflected light based on the reflection information added based on the estimation information G. FIG. However, the amount of reflected light displayed in the image with reflection information does not indicate the amount of reflected light when the imaging unit I1 captures the image at the observation position, but indicates the amount of reflected light obtained at the position where the imaging unit I1 captures the image. Therefore, the amount of reflected light indicated in the image with reflection information does not depend on the observation position specified by the observation position information T. FIG.

FIG. 17 is a diagram showing an image with reflection information in which the three-dimensional model Wd generated based on the estimated image G1 is visually recognized. Of the three-dimensional model Wd, the area corresponding to the three-dimensional model Wm that does not appear in the estimated image G1 does not show the amount of reflected light, and only the outline is drawn.

FIG. 18 is a diagram showing an image with reflection information in which the three-dimensional model Wd generated based on the estimated image G2 is visually recognized. Of the three-dimensional model Wd, the area corresponding to the three-dimensional model Wm that does not appear in the estimated image G2 does not show the amount of reflected light, and only the outline is drawn.

The operator can visually recognize the image with reflection information to observe the three-dimensional model Wd from a desired observation position, and can determine whether or not the amount of reflected light obtained at the position imaged by the imaging unit I1 is sufficient for inspection. . In the example of FIG. 17, it can be visually recognized that the workpiece W0 has an unimaged area on the negative side of the y-axis with respect to the origin O. As shown in FIG. By such visual recognition, the operator instructs the input unit 22 to arrange the image pickup unit I1 at a position where the z' coordinate is smaller than the position P1 and to pick up an image, for example, in order to obtain an image of such an area. can be entered.

The operator may judge that the amount of reflected light on the negative direction side of the y-axis is insufficient for inspection even in a region in which the amount of reflected light is shown in the three-dimensional model Wd and the image of the workpiece W0 is visually recognized. do not have. For example, in order to obtain a large amount of reflected light, the operator can input an instruction to the input unit 22 to arrange the imaging unit I1 at a position where the y′ coordinate is smaller than the position P1 and perform imaging.

The three-dimensional model Wd may be generated based on a plurality of estimated images, the estimated images G1 and G2 in the above example, for one three-dimensional model Wm. Such a three-dimensional model Wd is displayed in the output unit 23 by superimposing, for example, the amount of reflected light displayed in FIG. 17 and the amount of reflected light displayed in FIG.

However, the amount of reflected light superimposed and displayed on the three-dimensional model Wd is not the amount of reflected light obtained at the position where the three-dimensional model Wd is observed, but the amount of reflected light estimated to be obtained at the imaging unit I1 at the position P1, It is superimposed with the amount of reflected light estimated to be obtained in the imaging unit I1 at the position P2, and is not the amount of reflected light obtained at the position where the three-dimensional model Wd is observed. Therefore, it is not always appropriate to determine whether or not the amount of reflected light is sufficient for inspection based on the amounts of reflected light superimposed in this way.

When the amounts of reflected light are superimposed in this way, the amounts of reflected light obtained by the imaging unit I1 at different positions are displayed in different manners, for example, using different color tones for each of the plurality of estimated images used to generate the three-dimensional model Wd. may be Displaying the amount of reflected light estimated to be obtained in imaging in a different manner for each position of the imaging unit I1 contributes to selection of the position of the imaging unit I1 newly set by the operator.

<2-2-5. Setting part>
The setting unit 255 sets various conditions according to the information received by the input unit 22 based on the operator's operation while the three-dimensional model Wd is visibly output by the output unit 23, for example. be able to. For example, the operator can easily set inspection conditions for the position of the imaging unit I1. The position is stored as various data 24d in the storage unit 24, for example.

<2-3. Image processing flow>
FIG. 19 is a flow chart showing an example of the flow of image processing executed by the image processing apparatus 200 according to the image processing method according to this embodiment. This processing flow can be realized, for example, by executing the program 24p in the arithmetic processing unit 25a. The flow of this process is started in response to the input of a signal via the input unit 22 by the operator while the program 24p and various data 24d are stored in the storage unit 24, for example. Here, for example, the processing from step S1 to step S3 shown in FIG. 19 is performed in the order of this description.

In step S1, captured image estimation processing is executed. In the above example, step S1 is executed by the arithmetic processing unit 25a, more specifically by the estimating unit 252. FIG. The estimation unit 252 obtains estimation information G based on the model information Wi, the position and orientation information R, and the illumination information L. FIG.

The model information Wi includes information indicating the outline of the three-dimensional model Wm of the workpiece W0. A work W0 is a subject to be imaged by the imaging unit I1. The position/orientation information R can be said to be position information indicating a relative positional relationship between the imaging unit I1 and the work W0, which is the subject of the imaging unit I1.

The lighting information L indicates the characteristics of the irradiation light that irradiates the workpiece W0, which is the subject. The irradiation light is emitted from the illumination unit F1 in the above example. The characteristic of the illuminating light is, for example, the intensity distribution of the illuminating light. The estimated information G is information indicating an estimated image. The estimated image is obtained by estimating the image (captured image) of the workpiece W0 captured by the imaging unit I1. For example, imaging parameter information is used to obtain the estimated image.

For example, when the position/orientation information R indicates that the imaging unit I1 is at the position P1, an estimated image G1 is obtained (see FIGS. 13 and 15). When the position/orientation information R indicates that the imaging unit I1 is at the position P2, an estimated image G2 is obtained (see FIGS. 13 and 16).

The estimated image does not necessarily have to be displayed. When displaying the estimated image, for example, the estimated information G is given to the output unit 23 via the bus line 2b, and the output unit 23 displays the estimated image based on the estimated information G under the control of the output control unit 254. (see dashed arrow in FIG. 14).

A plurality of estimated information G may be obtained in step S1. In line with the above example, estimated information G that indicates estimated images G1 and G2, respectively, may be obtained.

At step S2, a three-dimensional model Wd of the workpiece W0 is generated. In the above example, step S2 is executed by the arithmetic processing unit 25a, more specifically by the attribute adding unit 253. FIG.

In step S2, the model information Wj is generated by adding the reflection information to the model information Wi representing the three-dimensional model Wm. The model information Wj includes information indicating the outline of the three-dimensional model Wd. Reflection information indicates the state of reflected light. The reflected light referred to here is the light that is incident on the imaging unit I1 after the workpiece W0 reflects the irradiation light.

In step S3, an image with reflection information is displayed. In the above example, step S3 is executed by the output section 23 and the arithmetic processing section 25a, more specifically by the output control section 254. FIG.

The arithmetic processing unit 25a outputs the three-dimensional model Wd viewed from the observation position specified by the observation position information T to the output unit 23 as the image information Gs. The output unit 23 visually outputs the three-dimensional model Wd based on the image information Gs. In this manner, the output unit 23 visually outputs the three-dimensional model Wd based on the observation position at which the workpiece W0 is observed.

Using the observation position information T, the operator can observe the three-dimensional model Wd from various observation positions. Such observation improves the efficiency of confirmation work for obtaining the camera/work positional relationship in consideration of the amount of reflected light.

The estimation unit 252 may obtain a plurality of pieces of estimated information G corresponding to a plurality of pieces of position and orientation information R, respectively. In this case, the attribute addition unit 253 adds reflection information corresponding to each of the plurality of position and orientation information R to one piece of model information Wi based on a plurality of pieces of estimation information G to create one piece of model information Wj. Generate. Then, the output control unit 254 generates the image information Gs of the three-dimensional model Wd in which the reflection information corresponding to each of the plurality of pieces of position and orientation information R in the model information Wj is made different from each other. The output unit 23 visually outputs the three-dimensional model Wd based on the image information Gs.

The output unit 23 visually outputs the three-dimensional model Wd, more specifically, an image with reflection information, by making the reflection information corresponding to each of the plurality of position and orientation information R different from each other. As an example of visually outputting the reflection information corresponding to each of the plurality of pieces of position/orientation information R in different manners, different color tones may be mentioned.

The fact that the amount of reflected light estimated to be obtained in imaging is displayed in a different manner, for example, in a color tone for each position of the imaging unit I1 contributes to the selection of the position of the imaging unit I1 newly set by the operator.

<Deformation of imaging unit 12s>
The imaging unit 12s may include two or more imaging units I1, and may include two or more illumination units F1.

FIG. 20 is a diagram schematically showing another example of the physical configuration of the imaging unit 12s. FIG. 20 illustrates a case where the imaging unit 12s includes imaging units I1a, I1b, and I1c and lighting units F1a, F1b, and F1c. In such a case, the workpiece W0 can be imaged under a plurality of lighting conditions with fewer operations without moving the imaging unit 12s. For example, it is easy to obtain an image that fully captures workpieces W0 having various shapes.

Specifically, the workpiece W0 placed on the placement part Sg2 can be imaged from above by the imaging part I1a while being illuminated from above by the illumination part F1a. The workpiece W0 placed on the mounting part Sg2 can be imaged obliquely from above by the imaging part I1b while being illuminated obliquely from above by the illumination part F1b. The workpiece W0 placed on the mounting part Sg2 can be imaged in a substantially horizontal direction by the imaging part I1c while being illuminated substantially in the horizontal direction by the lighting part F1c.

In the example of FIG. 20, the angle θ1a formed between the optical axis Pi1a of the lens portion Lz1a as the optical system in the imaging portion I1a and the virtual horizontal plane Pn0 passing through the workpiece W0 is 90 degrees. At this time, the imaging unit I1a functions as a camera capable of imaging the workpiece W0 from above. The angle θ1b between the optical axis Pi1b of the lens portion Lz1b as an optical system in the imaging portion I1b and the horizontal plane Pn0 is 45 degrees. At this time, the imaging unit I1b functions as a camera capable of imaging the work W0 from obliquely above. The angle θ1c between the optical axis Pi1c of the lens portion Lz1c as an optical system in the imaging portion I1c and the horizontal plane Pn0 is 5 degrees. Therefore, the imaging unit I1c functions as a camera capable of imaging the workpiece W0 substantially along the horizontal direction.

For example, a configuration is adopted in which the optical axis Pi1a, the optical axis Pi1b, and the optical axis Pi1c intersect at substantially the center of the work W0. FIG. 20 illustrates a case where the optical axis Pi1a, the optical axis Pi1b, and the optical axis Pi1c are positioned along a virtual plane along the YZ plane.

For example, the lens part Lz1a may be positioned in a state of being inserted through the hole H1a in the illumination part F1a. From another point of view, the optical axis Pi1a of the lens portion Lz1a is set to pass through the hole portion H1a. For example, the lens portion Lz1b may be positioned in a state of being inserted through the hole H1b in the illumination portion F1b. From another point of view, the optical axis Pi1b of the lens portion Lz1b is set to pass through the hole portion H1b. For example, the lens part Lz1c may be positioned in a state of being inserted through the hole H1c in the lighting part F1c. From another point of view, the optical axis Pi1c of the lens portion Lz1c is set to pass through the hole H1c.

In the example of FIG. 20, the direction in which the imaging unit I1a faces when the workpiece W0 is imaged by the imaging unit I1a is indicated by an arrow along the optical axis Pi1a. The direction in which the image pickup unit I1b faces when the image pickup unit I1b picks up an image of the workpiece W0 is indicated by an arrow along the optical axis Pi1b. The direction in which the image pickup unit I1c faces when the image pickup unit I1c picks up an image of the workpiece W0 is indicated by an arrow along the optical axis Pi1c.

In the example of FIG. 20, the direction in which the illumination unit F1a can irradiate the work W0 placed on the placement unit Sg2 with light is depicted by a dashed-dotted line arrow Df1a. The illumination unit F1a functions as an illumination unit that irradiates the workpiece W0 with light from above.

A direction in which the illumination unit F1b can irradiate the work W0 placed on the placement unit Sg2 with light is depicted by a dashed-dotted line arrow Df1b. The illumination unit F1b functions as an illumination unit that irradiates the workpiece W0 with light obliquely from above.

The direction in which the illumination unit F1c can irradiate the work W0 placed on the placement unit Sg2 with light is depicted by a dashed-dotted line arrow Df1c. The illumination unit F1c functions as an illumination unit that irradiates the workpiece W0 with light along the horizontal direction.

FIG. 21 is a diagram schematically showing an example of the illumination unit F1a. A configuration including a light emitting region F1a1 as one region capable of irradiating light toward the work W0 can be adopted for the illumination unit F1a.

FIG. 22 is a diagram schematically showing an example of the illumination unit F1b. The illumination unit F1b may employ a configuration including oblique light emitting areas F1b1, F1b2, F1b3, F1b4, and F1b5 as five areas capable of irradiating light toward the workpiece W0.

FIG. 23 is a diagram schematically showing an example of the illumination unit F1c. A configuration including light emitting regions F1c1, F1c2, and F1c3 can be adopted for the illumination unit F1c as three regions capable of irradiating light toward the work W0.

For example, the light emitting region F1a1 has a hole H1a, the light emitting region F1b3 has a hole H1b, and the light emitting region F1c2 has a hole H1c. For example, when the illumination unit F1b is seen from directly above in the -Z direction, the light emitting regions F1b1, F1b2, F1b3, F1b4, and F1b5 are in the order described above, and are rotated by 45 degrees around the optical axis Pi1a. have For example, if the light-emitting region F1b3 is used as a reference, the light-emitting region F1b2 is arranged at a position rotated 45 degrees clockwise from the light-emitting region F1b3 about the optical axis Pi1a, and the light-emitting region F1b2 rotates clockwise from the light-emitting region F1b3 about the optical axis Pi1a. A light emitting region F1b1 is arranged at a position rotated by 90 degrees, and a light emitting region F1b4 is arranged at a position rotated counterclockwise from the light emitting region F1b3 by 45 degrees around the optical axis Pi1a. A light-emitting region F1b5 is arranged at a position rotated counterclockwise by 90 degrees.

For example, when the illumination unit F1c is seen from directly above in the -Z direction, the light emitting regions F1c1, F1c2, and F1c3 have a positional relationship rotated by 45 degrees around the optical axis Pi1a in the order shown. . For example, if the light-emitting region F1c2 is used as a reference, the light-emitting region F1c1 is arranged at a position rotated 45 degrees clockwise from the light-emitting region F1c2 about the optical axis Pi1a. The light emitting region F1c3 is arranged at a position rotated by 45 degrees.

The imaging units I1a, I1b, and I1c and the illumination units F1a, F1b, and F1c having the above configuration are connected to each other by, for example, connecting members to have an integral configuration. In this case, for example, the imaging unit 12s can be moved by moving the connecting member by the first moving mechanism 12t1. Since such an integrated structure is known in Patent Document 2, for example, its details are omitted.

Thus, for example, the illumination unit F1a includes one light-emitting region F1a1, the illumination unit F1b includes five light-emitting regions F1b1, F1b2, F1b3, F1b4, and F1b5, and the illumination unit F1c includes three light-emitting regions F1c1, F1c2, Having F1c3 can be reduced by switching the presence or absence of light irradiation to the workpiece W0 for each of the light emitting regions F1a1, F1b1, F1b2, F1b3, F1b4, F1b5, F1c1, F1c2, and F1c3 without moving the imaging unit 12s. Operation contributes to imaging in multiple lighting conditions. Therefore, for example, it is easy to pick up an image that fully captures the work W0 having various shapes.

For example, a configuration in which one of the illumination units F1b and F1c does not exist and two of the imaging units I1a, I1b, and I1c do not exist is conceivable. The angles θ1a, θ1b, and θ1c may be appropriately set between 0 degrees and 90 degrees.

When the workpiece W0 is imaged by the imaging unit I1a, light emission from the light emitting region F1a1 is not essential. When the image of the workpiece W0 is imaged by the imaging unit I1b, it is not essential that at least one of the light emitting regions F1b1, F1b2, F1b3, F1b4, and F1b5 emit light. When the image of the workpiece W0 is captured by the imaging unit I1c, it is not essential that at least one of the light emitting regions F1c1, F1c2, and F1c3 emit light. When any one of the imaging units I1a, I1b, and I1c captures an image of the workpiece W0, any one or more of the light emitting regions F1a1, F1b1, F1b2, F1b3, F1b4, F1b5, F1c1, F1c2, and F1c3 emit light from the imaging unit. It contributes to any imaging of I1a, I1b, and I1c. The fact that the optical axis of the lens of the imaging unit and the main irradiation direction of the irradiation light are non-parallel facilitates the imaging of images capturing various shapes such as unevenness of the workpiece W0.

For example, in the image processing described above, the illumination information L indicates the characteristics of the irradiation light emitted from the light emitting region F1b1, and the reflection information indicates that the irradiation light emitted from the light emitting region F1b1 is reflected by the workpiece W0 and enters the imaging unit I1a. Indicates the lighting conditions. Alternatively, for example, the illumination information L indicates the characteristics of the irradiation light emitted from the light emitting regions F1b1 and F1b2, and the reflection information is the light incident on the imaging unit I1c after the workpiece W0 reflects the irradiation light emitted from the light emitting regions F1b1 and F1b2. indicates the state of

<Other variations>
In the above embodiments, for example, the operation of the inspection system 1 may be controlled by one or more controllers such as the integrated controller C0. Here, for example, the inspection system 1 may be regarded as one imaging device that images the workpiece W0.

Needless to say, all or part of each of the above-described embodiments and various modifications can be appropriately combined within a consistent range.

2 information processing device 22 input unit 23 output unit 24p program 25a arithmetic processing unit 200 image processing unit 251 division unit 252 estimation unit 253 attribute addition unit 254 output control unit G estimated information G1, G2 estimated image Gs image information I1 imaging unit L lighting Information R Position information S Classification information S1, S2, S3 Step T Observation position information W0 Work (subject, inspection object)
Wd (second) three-dimensional model Wi (first) model information Wj (second) model information Wm (first) three-dimensional model

Claims (7)

first model information including information indicating an outline of a first three-dimensional model of a subject; position information indicating a relative positional relationship between the subject and an imaging unit that captures the subject; and the subject. an estimating unit that obtains estimation information, which is information indicating an estimated image obtained by estimating the image of the subject captured by the imaging unit, based on illumination information that indicates the characteristics of the illumination light that irradiates the
Based on the estimation information and the first model information, reflection information, which is information indicating a state of reflected light, which is light incident on the imaging unit after the irradiation light is reflected by the subject, is set to the first model information. an attribute addition unit that generates second model information including information indicating the contour of the second three-dimensional model of the subject by adding to the model information of
and an output unit that visually outputs the second three-dimensional model based on an observation position from which the subject is observed. The image processing device according to claim 1,
The estimating unit obtains a plurality of pieces of estimated information respectively corresponding to a plurality of pieces of position information,
The attribute addition unit adds the reflection information corresponding to each of the plurality of position information to the one piece of first model information based on the plurality of pieces of estimated information, generate model information for
The image processing device, wherein the output unit visually outputs the second three-dimensional model by making the reflection information corresponding to each of the plurality of position information different from each other. The image processing device according to claim 1 or claim 2,
an input unit for inputting observation position information, which is information indicating the observation position;
generating image information, which is information about a visual output of the second three-dimensional model, based on the observation position information and the second model information, and outputting the image information to the output unit; further comprising a control unit,
The image processing device, wherein the output unit visually outputs the second three-dimensional model based on the image information. The image processing device according to any one of claims 1 to 3,
The image processing device, wherein the output unit displays the estimated image. The image processing device according to any one of claims 1 to 4,
further comprising a division unit that divides the outline of the first three-dimensional model into a plurality of regions and obtains division information that is information indicating the regions;
The image processing apparatus, wherein the attribute addition unit uses the division information to add the reflection information to the first model information for each region to generate the second model information. A program that causes the information processing device to function as the image processing device according to any one of claims 1 to 5 by being executed by an arithmetic processing unit included in the information processing device. first model information including information indicating an outline of a first three-dimensional model of a subject; position information indicating a relative positional relationship between the subject and an imaging unit that captures the subject; and the subject. a step of obtaining estimation information, which is information indicating an estimated image obtained by estimating the image of the subject captured by the imaging unit, based on illumination information indicating the characteristics of the illumination light that irradiates the subject;
Based on the estimation information and the first model information, reflection information, which is information indicating a state of reflected light, which is light incident on the imaging unit after the irradiation light is reflected by the subject, is set to the first model information. adding to the model information of to generate a second three-dimensional model of the subject;
and visually outputting the second three-dimensional model based on an observation position from which the subject is observed.

Images