WO2023189125A1

WO2023189125A1 - Camera device, image generation method, and system

Info

Publication number: WO2023189125A1
Application number: PCT/JP2023/007444
Authority: WO
Inventors: 利章篠原; 雄一畑瀬
Original assignee: ｉ－ＰＲＯ株式会社
Priority date: 2022-03-31
Filing date: 2023-02-28
Publication date: 2023-10-05

Abstract

This camera device comprises: an irradiation unit that irradiates an object with light using a first irradiation mode and a second irradiation mode different from the first irradiation mode; an irradiation control unit that controls the irradiation unit; an image capturing unit that generates a first captured image in which the object is captured while irradiated with light using the first irradiation mode, and a second captured image in which the object is captured while irradiated with light using the second irradiation mode; and a generation unit that generates a range image for identifying the range between a reference point and the object on the basis of the first captured image and the second captured image.

Description

Camera device, image generation method and system

The present disclosure relates to a camera device, an image generation method, and a system.

Patent Document 1 discloses a first illumination device that emits first light including a first wavelength, and a second illumination device that emits second light that includes a second wavelength different from the first wavelength. An image processing device is disclosed that includes: and an imaging device that captures an image of a target object. The imaging device images the object in a state in which the object is irradiated with first light from a first illumination device and second light is irradiated onto the object from a second illumination device. The image processing device generates a gray-scale image consisting of pixels imaged through a first optical filter that transmits a first wavelength, and pixels that are imaged through a second optical filter that transmits a second wavelength. Generate a distance image containing pixels.

Japanese Patent Application Publication No. 2020-193867

According to Patent Document 1, an environment in which a complex background (for example, a background in which not only the parts to be inspected but also various elements such as the pattern on the surface of the inspection jig, the structure of the machine, the tray, etc. are reflected) is reflected. However, image inspections such as determining the quality of parts can be performed stably and quickly. However, Patent Document 1 does not consider the case where the background is an area that has substantially no texture (texture such as a pattern). The area where there is substantially no texture as used herein is not limited to an area where there is no texture at all, but also conceptually includes an area where the presence of texture is so inconspicuous within the angle of view that it is difficult to distinguish. In other words, in Patent Document 1, when an object (for example, an object to be picked up by an end effector such as a robot hand) is placed in a background that is an area with substantially no texture, the distance to the object is accurately recognized. This sometimes made it difficult to pick up the object.

The present disclosure has been devised in view of the above-mentioned conventional circumstances, and aims to recognize the distance to an object with high precision.

The present disclosure includes: an illumination section that irradiates light to a target object using a first irradiation mode and a second irradiation mode that is different from the first irradiation mode; an illumination control section that controls the illumination section; an imaging unit that generates a first captured image of the object while being irradiated with light in an irradiation mode and a second captured image of the object while being irradiated with light in the second irradiation mode; , a generation unit that generates a distance image for specifying a distance between a reference point and the target object based on the first captured image and the second captured image. .

The present disclosure also provides a step of controlling an illumination unit that irradiates light to a target object using a first irradiation mode and a second irradiation mode different from the first irradiation mode; generating a first captured image of the target object in a irradiated state and a second captured image of the target object in a state of irradiation with light in the second irradiation mode; and a step of generating the first captured image. and the step of generating a distance image for specifying the distance between the reference point and the target object based on the second captured image.

The present disclosure also provides an illumination unit that irradiates light to a target object using a first irradiation mode and a second irradiation mode that is different from the first irradiation mode, an illumination control unit that controls the illumination unit, and Imaging that generates a first captured image that captures the object while being irradiated with light in a first irradiation mode, and a second captured image that captures the object while being irradiated with light in the second irradiation mode. and a generation unit that generates a distance image for specifying the distance between the reference point and the target object based on the first captured image and the second captured image. do.

Note that these general or specific aspects may be realized by any combination including one or more of a system, a device, a method, an integrated circuit, a computer program, or a recording medium.

According to the present disclosure, the distance to the target object can be recognized with high accuracy.

Further advantages and effects of one aspect of the present disclosure will become apparent from the specification and drawings. Such advantages and/or effects may be provided by each of the embodiments and the features described in the specification and drawings, but not all need to be provided in order to obtain one or more of the same features. There isn't.

FIG. 1 is a diagram showing an example of the system configuration of the system. FIG. 2A is a diagram showing an example of a time chart in which illumination of the illumination pattern P1 and illumination of the illumination pattern P2 are performed alternately. FIG. 2B is a diagram showing another example of a time chart in which the illumination of the illumination pattern P1 and the illumination of the illumination pattern P2 are alternately performed. FIG. 3 is a flowchart showing an example of the first operation procedure of the camera device. FIG. 4 is a flowchart illustrating an example of the second operation procedure of the camera device. FIG. 5 is a diagram showing an example of a UI setting screen when lighting is performed using a plurality of different lighting patterns. FIG. 6 is a diagram showing an example of a UI setting screen when the same illumination pattern is irradiated with shifted positions. FIG. 7 is a diagram showing an example of a UI setting screen when emitting light with the dot pitch reduced by a specified magnification. FIG. 8 is a diagram showing an example of an operation outline and an example of a UI setting screen when controlling a lighting area using the processing results of the AI processing unit. FIG. 9A is a flowchart illustrating an example of the third operation procedure of the camera device. FIG. 9B is a flowchart showing another example of the third operation procedure of the camera device. FIG. 10 is a diagram showing an example of an operation outline and an example of a UI setting screen when controlling illumination intensity using the processing result of the AI processing unit. FIG. 11A is a flowchart illustrating an example of the fourth operation procedure of the camera device. FIG. 11B is a flowchart showing another example of the fourth operation procedure of the camera device. FIG. 12 is a flowchart illustrating an example of the fifth operation procedure of the camera device.

Hereinafter, embodiments specifically disclosing a camera device, an image generation method, and a system according to the present disclosure will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of well-known matters or redundant explanations of substantially the same configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter recited in the claims.

In Embodiment 1 below, a use case in which an object is imaged by a camera device fixed to a robot hand (not shown) that is the tip of a robot arm (not shown) will be described. The target object is, for example, a background object that is an area with substantially no texture (for example, a white table or floor surface without a pattern, a black table or floor surface without a pattern), or a background object placed on a table as an example of a background object. This corresponds to an object photographed by a camera device, such as a workpiece as an example of a placed object. To simplify the following explanation, unless otherwise specified, the objects are a white table with virtually no texture and a workpiece placed on the white table (for example, a white vase, a red apple, etc.). I will list and explain. Note that the use case of Embodiment 1 does not need to be limited to the above-mentioned example. The object is, for example, an industrial product or an industrial part, and may be picked up by a robot hand. Note that the target object does not have to be limited to industrial products or industrial parts.

FIG. 1 is a diagram showing an example of the system configuration of the system 100. As shown in FIG. 1, the system 100 includes at least a camera device 10 and a PC (Personal Computer) 50. The camera device 10 and the PC 50 are connected so that data signals can be input and output to each other. Although not shown in FIG. 1, the camera device 10 may be connected to a robot controller (not shown) that can control the movement of a robot hand (not shown) to which the camera device 10 is fixed. .

The camera device 10 is, for example, fixed to a robot hand (not shown; the same applies hereinafter), which is the tip of a robot arm (not shown; the same applies hereinafter), and is movable integrally with the robot hand. The camera device 10 is fixed so as not to interfere with the robot hand (that is, so that the robot hand (not shown) is not included within the field of view of the camera device 10). The camera device 10 corresponds to the illumination mode by alternately using each of a plurality of types of illumination modes (in other words, structured illumination (see below)) in a time-sharing manner toward an object included within its angle of view. The object is imaged with compound eyes while irradiating structured illumination. That is, the camera device 10 corresponds to a so-called two-lens type stereo camera that includes a plurality of imaging modules (for example, a first imaging module CAP1 and a second imaging module CAP2, see below). The camera device 10 uses an image (for example, a distance image and/or parallax image). Further, the camera device 10 can specify the position information of the object based on the distance image and/or the parallax image. The camera device 10 outputs output data including the generated distance image and target object position information to the robot controller. Although only one camera device 10 is illustrated in FIG. 1, the system 100 may include a plurality of camera devices 10 having the same configuration. In this case, the plurality of camera devices 10 and the PC 50 are connected to each other so that data signals can be input and output.

The robot controller recognizes the distance from the camera device 10 (in other words, the robot hand) to the target object based on the output data from the camera device 10. Subsequently, the robot controller controls the movement of the robot hand (for example, picking up the object) according to the position of the object. Note that the movements performed by the robot hand are adaptively determined according to the use case of the system 100, and are not limited to the above-mentioned picking up of objects.

The PC 50 has executable browser software (not shown) installed therein for setting various parameters used by the camera device 10. The PC 50 starts up browser software in response to an operation by a user of the system 100 (not shown; the same applies hereinafter), and displays various UI (User Interface) setting screens for the camera device 10. Details of various parameters and the UI setting screen will be described later.

The camera device 10 includes a first imaging module CAP1, a second imaging module CAP2, an SoC (System on a Chip) 15, a lighting control section 16, and a lighting section 17. The first imaging module CAP1 includes an L lens 11 and an L image sensor 13, and images the object from a first viewpoint. The second imaging module CAP2 includes an R lens 12 and an R image sensor 14, and images the object from a second viewpoint different from the first viewpoint. In the direction parallel to the installation surface of the camera device 10, the first imaging module CAP1 and the second imaging module CAP2 are arranged with an interval (eg, module installation interval) between them. That is, the second viewpoint corresponds to a viewpoint for capturing an image of the object from a position separated by the module installation distance from the first viewpoint in a direction parallel to the installation surface of the camera device 10. Note that the module installation interval may be defined with respect to a direction other than the direction parallel to the installation surface of the camera device 10. The first imaging module CAP1 and the second imaging module CAP2 correspond to the imaging section of this embodiment.

The L lens 11 includes, for example, a focus lens and a zoom lens. The L lens 11 is illuminated with structured illumination (see below) that corresponds to the first illumination mode from the illumination unit 17 toward the object, and the L lens 11 is illuminated by the object (for example, the object) included within the angle of view. Incident light, which is reflected light, enters. If either a visible light cut filter or an IR cut filter is disposed between the L lens 11 and the L image sensor 13, the incident light that entered the L lens 11 passes through the filter to the L image sensor 13. An optical image of the subject is formed on the light-receiving surface (imaging surface). As the L lens 11, lenses with various focal lengths or shooting ranges can be used depending on the installation location of the camera device 10, the shooting purpose, etc.

Note that the camera device 10 may include an L lens drive section (not shown; the same applies hereinafter) that controls the drive of the L lens 11. Further, the CPU (Central Processing Unit) 152 or the ISP (Image Signal Processor) 153 adjusts (changes) internal parameters related to driving the L lens 11 (for example, the position of the focus lens, the position of the zoom lens corresponding to the zoom magnification). Alternatively, the L lens 11 may be driven via the L lens driving section. Further, the L lens 11 may be fixedly arranged.

The R lens 12 includes, for example, a focus lens and a zoom lens. The R lens 12 is illuminated with structured illumination (see below) that corresponds to the second illumination mode from the illumination unit 17 toward the object, and when the object (for example, the object) included in the angle of view Incident light, which is reflected light, enters. If either a visible light cut filter or an IR cut filter is disposed between the R lens 12 and the R image sensor 14, the incident light that enters the R lens 12 passes through the filter to the R image sensor 14. An optical image of the subject is formed on the light-receiving surface (imaging surface). As the R lens 12, lenses with various focal lengths or shooting ranges can be used depending on the installation location of the camera device 10, the shooting purpose, etc.

Note that the camera device 10 may include an R lens drive section (not shown; the same applies hereinafter) that controls the drive of the R lens 12. Further, the CPU 152 or the ISP 153 adjusts (changes) internal parameters related to driving the R lens 12 (for example, the position of the focus lens, the position of the zoom lens corresponding to the zoom magnification), and drives the R lens 12 via the R lens driving section. It may be driven. Further, the R lens 12 may be fixedly arranged.

The visible light cut filter has a property of blocking visible light (for example, light having a wavelength of 400 to 760 [nm]) out of the incident light that has passed through the L lens 11 or the R lens 12 (that is, the light reflected by the subject). have The visible light cut filter blocks visible light among the incident light that has passed through the L lens 11 or the R lens 12. The camera device 10 may include a filter drive unit (not shown; the same applies hereinafter) that controls the drive of the visible light cut filter. In the case where the filter drive unit is provided, the visible light cut filter operates between the L lens 11 and the L image sensor 13 and between the R lens 12 and the R image sensor during a predetermined period (for example, at night) based on a control signal from the CPU 152 or the ISP 153. 14 via the filter drive section.

The IR cut filter has the property of passing visible light (for example, light having a wavelength of 400 to 760 [nm]) and blocking near-infrared light (for example, light having a wavelength of 780 [nm] or more). The IR cut filter blocks near-infrared light of the incident light that has passed through the L lens 11 or the R lens 12 and allows visible light to pass through. The camera device 10 may include a filter drive unit that controls the drive of the IR cut filter. In the case where the filter drive section is provided, the IR cut filter operates between the L lens 11 and the L image sensor 13 and between the R lens 12 and the R image sensor 14 during a predetermined period (for example, during the day) based on a control signal from the CPU 152 or the ISP 153. The filter drive unit is located between the filter drive unit and the filter drive unit.

The L image sensor 13 includes a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor in which a plurality of pixels suitable for imaging visible light or near-infrared light are arranged, and an exposure control circuit (not shown). omitted) and a signal processing circuit (not shown). The L image sensor 13 performs photoelectric conversion at predetermined intervals to convert light received by a light-receiving surface (imaging surface) made up of a plurality of pixels into an electrical signal. The predetermined interval of photoelectric conversion is determined according to the so-called frame rate (fps: frame per second). For example, if the frame rate is 120 [fps], the predetermined interval is 1/120 [second]. As a result, the L image sensor 13 is configured to respond to light reflected by the object while the illumination unit 17 is emitting structured illumination (see below) corresponding to the first irradiation mode toward the object. , a red component signal (R signal), a green component signal (G signal), and a blue component signal (B signal) are acquired temporally continuously for each pixel as electrical signals. Note that the L image sensor 13 may acquire a monochrome electrical signal instead of an RGB electrical signal. A signal processing circuit (not shown) of the L image sensor 13 converts an electrical signal (analog signal) into digital imaging data. The L image sensor 13 transfers digital image data to the memory 151 at predetermined intervals depending on the frame rate. The memory 151 stores digital image data received from the L image sensor 13. Note that the L image sensor 13 may send digital image data to the CPU 152 at predetermined intervals depending on the frame rate. Further, the L image sensor 13 adjusts (changes) internal parameters indicating exposure conditions (for example, exposure time, gain, frame rate) based on an exposure control signal from the CPU 152 or the ISP 153 using an exposure control circuit. Thereby, the camera device 10 can change the exposure conditions according to the surrounding environment, and can obtain imaging data with good image quality.

The R image sensor 14 includes a CCD sensor or CMOS sensor in which a plurality of pixels suitable for imaging visible light or near-infrared light are arranged, an exposure control circuit (not shown), and a signal processing circuit (not shown). include. The R image sensor 14 performs photoelectric conversion at predetermined intervals to convert light received by a light receiving surface (imaging surface) made up of a plurality of pixels into an electrical signal. The predetermined interval of photoelectric conversion is determined according to the so-called frame rate (fps). For example, if the frame rate is 120 [fps], the predetermined interval is 1/120 [second]. As a result, the R image sensor 14 is configured to respond to the light reflected by the object while the illumination unit 17 is emitting structured illumination (see below) corresponding to the second irradiation mode toward the object. , a red component signal (R signal), a green component signal (G signal), and a blue component signal (B signal) are acquired temporally continuously for each pixel as electrical signals. Note that the R image sensor 14 may acquire a monochrome electrical signal instead of an RGB electrical signal. A signal processing circuit (not shown) of the R image sensor 14 converts an electrical signal (analog signal) into digital imaging data. The R image sensor 14 transfers digital image data to the memory 151 at predetermined intervals depending on the frame rate. The memory 151 stores digital format imaging data received from the R image sensor 14. Note that the R image sensor 14 may send digital image data to the CPU 152 at predetermined intervals depending on the frame rate. Further, the R image sensor 14 adjusts (changes) internal parameters related to exposure conditions (for example, exposure time, gain, frame rate) by an exposure control circuit based on an exposure control signal from the CPU 152 or the ISP 153. Thereby, the camera device 10 can change the exposure conditions according to the surrounding environment, and can obtain imaging data with good image quality.

The SoC 15 is an integrated circuit product in which various electronic components are integrated on a single chip made up of an integrated circuit. The SoC 15 includes a memory 151, a CPU 152, an ISP 153, an AI processing section 154, and an external interface 155. In FIG. 1, the interface is illustrated as "I/F" for convenience.

The memory 151 includes at least a RAM (Random Access Memory) and a ROM (Read Only Memory). The memory 151 temporarily stores programs and control data necessary for executing the operations of the camera device 10, as well as data or information generated during the operation of each part of the camera device 10. The RAM is, for example, a work memory used when each part of the camera device 10 operates. The ROM stores and retains programs and control data for controlling each part of the camera device 10 in advance, for example. The memory 151 also stores a structured illumination pattern (not shown) set by the PC 50 or attribute information for specifying the pattern. The memory 151 also stores information about the waiting time (minute time ΔTa) from when the illumination unit 17 emits the first structured illumination of the illumination pattern P1 until it emits the second structured illumination of the illumination pattern P2, and the illumination unit 17 stores information on the waiting time (minute time ΔTb) from when the second structured illumination of the illumination pattern P2 is irradiated to when the first structured illumination of the illumination pattern P1 is irradiated.

The CPU 152 is a processor that functions as a controller that controls the overall operation of the camera device 10. The CPU 152 performs control processing for unifying the operations of each part of the camera device 10, data input/output processing with respect to each part of the camera device 10, data calculation processing, and data storage processing. The CPU 152 operates according to programs and control data stored in the memory 151. The CPU 152 uses the memory 151 during operation, and transfers data or information generated or acquired by the CPU 152 to the memory 151 for temporary storage. Further, the CPU 152 acquires image data from each of the L image sensor 13 and the R image sensor 14 from the memory 151, and calculates the parallax between the image data from the L image sensor 13 and the image data from the R image sensor 14. . The CPU 152 generates a distance image of an object included in the target object within the viewing angle based on the parallax calculation result. Note that the CPU 152 has a timer (not shown) or an illuminance sensor (not shown), and generates a control signal for arranging either the visible light cut filter or the IR cut filter based on the output of the timer or the illuminance sensor. It may also be generated and sent to the filter driver. Details of various processes performed by the CPU 152 will be described later.

The ISP 153 is a processor that manages various image processes performed within the camera device 10. The ISP 153 reads imaging data from the memory 151 and performs various image processing using the read imaging data. The ISP 153 uses the memory 151 during operation, and transfers data or information generated or acquired by the ISP 153 to the memory 151 for temporary storage. The ISP 153 performs resizing processing to convert the size of the imaging data read from the memory 151 into a size suitable for the area determination processing performed by the AI processing unit 154. Details of the area determination process will be described later. The ISP 153 determines the irradiation area or irradiation intensity of the second structured illumination (see below) that corresponds to the second irradiation mode to be emitted by the illumination unit 17 based on the processing result of the AI processing unit 154, and controls the illumination control unit 16. send to The ISP 153 includes image data (first left image P1L) captured by the L image sensor 13 in a state where the first structured illumination corresponding to the first irradiation mode is irradiated, and image data captured by the R image sensor 14 in the same state. Based on the imaging data (first right image P1R), a distance image for specifying the distance from the reference point to the target object is generated. Here, the reference point corresponds to the imaging planes and focal points of the L image sensor 13 and the R image sensor 14, the lens interfaces of the L lens 11 and the R lens 12, and the like. In other words, the reference point corresponds to a position definable by an optical element included in the camera device 10. Note that the definition of the reference point is not limited to the example described above. Note that the ISP 153 stores image data (first left image P1L) captured by the L image sensor 13 in a state where the first structured illumination corresponding to the first irradiation mode is irradiated, and image data captured by the R image sensor 14 in the same state. Based on the captured imaging data (first right image P1R), an exposure control signal is generated for adjusting internal parameters that determine the exposure conditions for imaging (that is, exposure) by each of the L image sensor 13 and the R image sensor 14. and may be sent to each of the L image sensor 13 and the R image sensor 14. Details of various processes performed by the ISP 153 will be described later.

The AI processing unit 154 is configured using, for example, a GPU (Graphics Processing Unit) and memory. Note that a DSP (Digital Signal Processor) may be used instead of or together with the GPU. The AI processing unit 154 uses AI (for example, a trained model for segmentation) to generate image data (the first Based on the first left image P1L) and the image data captured by the R image sensor 14 in the same state (the first right image P1R), area determination processing is performed for each image data (image). The area determination process corresponds to, for example, a process of determining an image area corresponding to the table and an image area corresponding to the work in an image obtained by capturing a workpiece placed on a table as imaging data. The AI processing unit 154 reads out the image data (for example, the first left image P1L, first right image P1R) that has been resized by the ISP 153 from the memory 151 and performs the above-described area determination process. The learned model used by the AI processing unit 154 is generated in advance through machine learning, for example. Here, the learned model refers to parameters ( For example, it corresponds to a model equipped with various weighting coefficients, etc.). The AI processing unit 154 stores the result of the area determination process in the memory 151.

The external interface 155 receives data requests from the PC 50 and transmits data generated or acquired by the CPU 152 or the ISP 153 to the PC 50 in accordance with the data requests. Although not shown in FIG. 1, when the robot controller and the camera device 10 are connected, the external interface 155 can transmit data generated or acquired by the CPU 152 or the ISP 153 to the robot controller. good.

The illumination control unit 16 is constituted by a control circuit for performing various controls on the first structured illumination and the second structured illumination from the illumination unit 17. The lighting control unit 16 generates control signals for controlling each pattern, irradiation mode, and irradiation timing of the first structured illumination and the second structured illumination irradiated by the illumination unit 17 based on the output from the SoC 15. and sends it to the illumination section 17. Here, the irradiation mode is an operation mode that determines which light, the first structured illumination or the second structured illumination, is irradiated by the illumination unit 17. Specifically, the illumination mode is divided into a first illumination mode in which the illumination unit 17 emits the first structured illumination, and a second illumination mode in which the illumination unit 17 emits the second structured illumination. . The illumination control unit 16 switches between the first irradiation mode and the second irradiation mode, for example, according to the time chart shown in FIG. 2A or the time chart shown in FIG. 2B.

The illumination unit 17 is configured by a projector including a light emitting element capable of illuminating each of the first structured illumination LG1 and the second structured illumination LG2. The illumination unit 17 irradiates the object with each of the first structured illumination LG1 and the second structured illumination LG2 based on the control signal from the illumination control unit 16. The illumination unit 17 also irradiates the entire object with the first structured illumination LG1, and further irradiates the entire object with the second structured illumination LG1 toward an illumination area (for example, an area corresponding to an arrangement object) controlled by the illumination control unit 16. irradiate with the chemical illumination LG2. Further, the illumination unit 17 irradiates the entire object with the first structured illumination LG1, and further irradiates the entire object with the second structured illumination LG2 having an illumination intensity (for example, irradiation brightness) controlled by the illumination control unit 16. Irradiates the entire object. Here, the first structured illumination LG1 is, for example, illumination having a texture in which a plurality of dots are arranged in a predetermined pattern. The second structured illumination LG2 is, for example, illumination having a texture in which a plurality of dots are arranged in a predetermined pattern. The arrangement positions of the plurality of dots are different between the first structured illumination LG1 and the second structured illumination LG2.

Note that although FIG. 1 shows an example in which each of the lighting control section 16 and the lighting section 17 is included in the camera device 10, each of the lighting control section 16 and the lighting section 17 is included in the camera device 10. 10 may be provided separately. In this case, the lighting control section 16 and the external interface 155 of the SoC 15 are connected so that data signals can be input and output.

FIG. 2A is a diagram showing an example of a time chart in which illumination of the illumination pattern P1 and illumination of the illumination pattern P2 are alternately performed. FIG. 2B is a diagram showing another example of a time chart in which the illumination of the illumination pattern P1 and the illumination of the illumination pattern P2 are alternately performed. The illumination of the illumination pattern P1 corresponds to the first structured illumination LG1, and the illumination of the illumination pattern P2 corresponds to the second structured illumination LG2. The horizontal axis in FIG. 2 indicates time.

As shown in FIG. 2A, the illumination control unit 16 provides structured illumination in two irradiation modes (i.e., the first irradiation The illumination pattern P1 corresponding to the mode and the illumination pattern P2 corresponding to the second irradiation mode are alternately time-divisionally irradiated from the illumination unit 17. For example, when the illumination pattern P1 is applied at time t1, the illumination pattern P2 is applied at time t2 after a minute time Δta has elapsed from time t1. When the illumination of the illumination pattern P2 is applied at time t2, the illumination of the illumination pattern P1 is applied at time t3 after a minute time Δtb has elapsed from time t2. When the illumination of the illumination pattern P1 is applied at time t3, the illumination of the illumination pattern P2 is applied at time t4 after a minute time Δta has elapsed from time t3. When the illumination of the illumination pattern P2 is applied at time t4, the illumination of the illumination pattern P1 is applied at time t5 after a short time Δtb has elapsed from time t4. When the illumination pattern P1 is irradiated at time t5, the illumination pattern P2 is irradiated at time t6 after a short time Δta has elapsed from time t5. When the illumination of the illumination pattern P2 is irradiated at time t6, the illumination of the illumination pattern P1 is irradiated at a time after a minute time Δtb has elapsed from the time t6. The same applies below. Here, the minute time Δta and the minute time Δtb may be the same or different, and their magnitude relationship is not particularly limited.

At time t1, the first left image P1L is generated by the L image sensor 13 under illumination of the illumination pattern P1 (the first structured illumination LG1 corresponding to the first illumination mode), and the R image sensor 14, the first right image P1R is generated. The first left image P1L and the first right image P1R have parallax with each other. The first left image P1L is an example of a first captured image captured from the first viewpoint. The first right image P1R is an example of a second captured image captured from the second viewpoint.

At time t2, the second left image P2L is generated by the L image sensor 13 under illumination of the illumination pattern P2 (second structured illumination LG2 corresponding to the second illumination mode), and the R image sensor 14, the second right image P2R is generated. The second left image P2L and the second right image P2R have parallax with each other. The second left image P2L is an example of the first captured image captured from the first viewpoint. The second right image P2R is an example of a second captured image captured from the second viewpoint.

The CPU 152 or the ISP 153 combines the first left image P1L captured during the irradiation with the first structured illumination LG1 and the second left image P2L captured during the irradiation with the second structured illumination LG2. Through this combining process, the CPU 152 or the ISP 153 generates a combined left image P3L that has more image features than the image features on the first left image P1L or the image features on the second left image P2L. The composite left image P3L is an example of the first composite image.

The CPU 152 or the ISP 153 combines the first right image P1R captured during the irradiation with the first structured illumination LG1 and the second right image P2R captured during the irradiation with the second structured illumination LG2. Through this combining process, the CPU 152 or the ISP 153 generates a combined right image P3R that has more image features than the image features on the first right image P1R or the image features on the second right image P2R. The composite right image P3R is an example of the second composite image. Therefore, the composite left image P3L corresponds to an image having parallax with respect to the composite right image P3R.

The CPU 152 or the ISP 153 calculates the parallax between the composite left image P3L and the composite right image P3R based on the composite left image P3L and the composite right image P3R. Further, the CPU 152 or the ISP 153 generates a distance image for specifying the distance from the camera device 10 to the target object.

Here, the target object is a situation in which an object (for example, a white vase) is placed on a background object (for example, a white table) that is an area that has virtually no texture, and a situation in which an object (for example, a white vase) is placed on a background object (for example, a white table) that is an area that has virtually no texture. Assume that a situation in which an object (for example, a red apple) is placed on a (for example, a white table) is imaged using a two-lens camera such as a stereo camera. In this case, the captured image of the white vase and white table has fewer colors, that is, fewer image features (feature points), than the captured image of the red apple and white table. Therefore, when capturing an image of a white vase and a white table, it is not possible to accurately calculate the parallax for each of the white vase and white table, and it is difficult to accurately recognize the distance from the camera device to the object (white vase and white table). It was difficult to do so.

To solve this problem, the camera device 10 according to the first embodiment has a first structured illumination LG1 having an illumination pattern P1 provided with a plurality of dots (for example, see FIG. 5), and the illumination pattern P1 is different from the first structured illumination LG1. The second structured illumination LG2 having the illumination pattern P2 is alternately irradiated. Furthermore, the camera device 10 combines the left images (that is, the first left image P1L and the second left image P2L) into a composite left image P3L and the right images (that is, the first right image P1R and the second right image P2R). The combined right image P3R and the combined right image P3R are respectively generated and the parallax is calculated. Therefore, the parallax is calculated in a state where each image feature on the composite left image P3L and the composite right image P3R is apparently increased by the composition process. Thereby, the camera device 10 can generate a distance image that allows highly accurate recognition of the distance to an object that is substantially free of texture.

Furthermore, as shown in FIG. 2B, the ratio between the minute time Δta and the minute time Δtb is determined from the illumination pattern P1 to the illumination pattern P2, assuming the shortest imaging interval (for example, 1/480 [second] if the frame rate is 480 [fps]). The interval may be shorter than in FIG. 2A, and the interval from illumination pattern P2 to illumination pattern P1 may be longer than in FIG. 2A. As a result, the temporal blurring of the object (for example, a moving object) reflected in the distance image generated by the irradiation of the illumination pattern P1 and the illumination pattern P2 is shortened, and the accuracy of the distance image generated based on image synthesis is improved. improves.

Next, a first operational procedure example of the camera device 10 according to the first embodiment will be described with reference to FIG. 3. FIG. 3 is a flowchart illustrating an example of the first operation procedure of the camera device 10. In the description of FIG. 3, an example will be described in which illumination of the illumination pattern P1 and imaging under the illumination, and illumination of the illumination pattern P2 and imaging under the illumination, are each performed asynchronously.

In FIG. 3, the illumination control unit 16 of the camera device 10 switches to the first illumination mode at time t1 (see FIG. 2), for example, and directs the illumination of the illumination pattern P1 (first structured illumination LG1) toward the target object. Irradiation is performed from the illumination unit 17 (step St1). Each of the first imaging module CAP1 and the second imaging module CAP2 of the camera device 10 images the object while being irradiated with the first structured illumination LG1 from the illumination unit 17 in step St1 (step St2). In step St2, two pieces of imaging data (for example, the first left image P1L and the first right image P1R) having parallax with each other are obtained. For example, the first left image P1L is obtained from the first imaging module CAP1, and the first right image P1R is obtained from the second imaging module CAP2. The CPU 152 or the ISP 153 of the camera device 10 waits for the elapse of a minute time Δta (standby) from the execution timing of step St1 or the execution timing of step St2 (step St3).

For example, at time t2 (see FIG. 2), the illumination control unit 16 of the camera device 10 switches to the second illumination mode and directs illumination of the illumination pattern P2 (second structured illumination LG2) from the illumination unit 17 toward the object. irradiate (step St4). Each of the first imaging module CAP1 and the second imaging module CAP2 of the camera device 10 images the object while being irradiated with the second structured illumination LG2 from the illumination unit 17 in step St4 (step St5). In step St5, two pieces of imaging data (eg, second left image P2L and second right image P2R) having parallax with each other are obtained. For example, the second left image P2L is obtained from the first imaging module CAP1, and the second right image P2R is obtained from the second imaging module CAP2.

The CPU 152 or the ISP 153 of the camera device 10 selects one of the two image data captured in step St2 that has a parallax with each other (for example, the first left image P1L) and one of the two image data that has a parallax that is captured in step St5. (for example, second left image P2L) (step St6). Similarly, the CPU 152 or the ISP 153 of the camera device 10 selects the other of the two pieces of image data (for example, the first right image P1R) that have a parallax from each other that are imaged in step St2, and the two images that have a parallax from each other that are imaged in step St5. The other data (for example, the second right image P2R) is synthesized (step St6). Further, the CPU 152 or the ISP 153 of the camera device 10 calculates the parallax between the two composite images (for example, the composite left image P3L and the composite right image P3R) obtained by the composition. The CPU 152 or ISP 153 of the camera device 10 uses the parallax calculation results to generate a distance image for specifying the distance between the reference point of the camera device 10 (see above) and the target object, and outputs it to the PC 50. (Step St6).

The CPU 152 or the ISP 153 of the camera device 10 waits for a minute time Δtb to elapse (standby) from the execution timing of step St4, the execution timing of step St5, or the execution timing of step St6 (step St7). After step St7, the processing of the camera device 10 returns to step St1.

Note that in step St6, the CPU 152 or the ISP 153 of the camera device 10 displays the first left image P1L captured by the first imaging module CAP1 and the first right image captured by the second imaging module CAP2 during irradiation with the illumination pattern P1. The P1R parallax may be calculated. Similarly, the CPU 152 or the ISP 153 of the camera device 10 determines the parallax difference between the second left image P2L captured by the first imaging module CAP1 and the second right image P2R captured by the second imaging module CAP2 during irradiation with the illumination pattern P2. may be calculated. The CPU 152 or the ISP 153 of the camera device 10 combines the parallax based on the first left image P1L and the first right image P1R with the parallax based on the second left image P2L and the second right image P2R, thereby determining the standard of the camera device 10. A distance image for specifying the distance between a point (see above) and a target object may be generated and output to the PC 50.

Further, in step St6, the CPU 152 or the ISP 153 of the camera device 10 displays a first left image P1L captured by the first imaging module CAP1 and a first right image captured by the second imaging module CAP2 during the irradiation with the illumination pattern P1. The P1R parallax may be calculated. Similarly, the CPU 152 or the ISP 153 of the camera device 10 determines the parallax difference between the second left image P2L captured by the first imaging module CAP1 and the second right image P2R captured by the second imaging module CAP2 during irradiation with the illumination pattern P2. may be calculated. The CPU 152 or ISP 153 of the camera device 10 generates a distance image from the parallax based on the first left image P1L and the first right image P1R, and further generates a distance image from the parallax based on the second left image P2L and the second right image P2R. may be generated. The CPU 152 or the ISP 153 of the camera device 10 generates a distance image for specifying the distance between the reference point of the camera device 10 (see above) and the target object by combining these two distance images, and You can also output to

Next, a second operational procedure example of the camera device 10 according to the first embodiment will be described with reference to FIG. 4. FIG. 4 is a flowchart illustrating an example of the second operation procedure of the camera device. In the explanation of FIG. 4, an example will be described in which illumination of the illumination pattern P1 and imaging under the illumination, and illumination of the illumination pattern P2 and imaging under the illumination are executed in synchronization. In the description of FIG. 4, the same step numbers are assigned to the same processes as those in FIG. 3 to simplify or omit the description, and different contents will be described.

In FIG. 4, the illumination control unit 16 of the camera device 10 switches to the first illumination mode at time t1 (see FIG. 2), for example, and directs the illumination of the illumination pattern P1 (first structured illumination LG1) toward the target object. Irradiation is performed from the illumination unit 17 (step St1A). Each of the first imaging module CAP1 and the second imaging module CAP2 of the camera device 10 images the object while being irradiated with the first structured illumination LG1 from the illumination unit 17 in step St1A (step St1A). That is, in step St1A, two pieces of imaging data (for example, the first left image P1L and the first right image P1R) having parallax with each other are obtained in synchronization with the irradiation of the first structured illumination LG1 from the illumination unit 17. For example, the first left image P1L is obtained from the first imaging module CAP1, and the first right image P1R is obtained from the second imaging module CAP2. The CPU 152 or the ISP 153 of the camera device 10 waits for a minute time Δta to elapse (standby) from the execution timing of step St1A (step St3A).

For example, at time t2 (see FIG. 2), the illumination control unit 16 of the camera device 10 switches to the second illumination mode and directs illumination of the illumination pattern P2 (second structured illumination LG2) from the illumination unit 17 toward the object. irradiate (step St4A). Each of the first imaging module CAP1 and the second imaging module CAP2 of the camera device 10 images the object while being irradiated with the second structured illumination LG2 from the illumination unit 17 in step St4A (step St4A). That is, in step St4A, two pieces of imaging data (for example, the second left image P2L and the second right image P2R) having parallax with each other are obtained in synchronization with the irradiation of the second structured illumination LG2 from the illumination unit 17. For example, the second left image P2L is obtained from the first imaging module CAP1, and the second right image P2R is obtained from the second imaging module CAP2.

The CPU 152 or the ISP 153 of the camera device 10 selects one of the two image data captured in step St1A that has a parallax with each other (for example, the first left image P1L) and one of the two image data that has a parallax that is captured in step St4A. (for example, second left image P2L) (step St6). Similarly, the CPU 152 or the ISP 153 of the camera device 10 selects the other of the two image data (for example, the first right image P1R) that has a parallax from each other that was imaged in step St1A, and the two images that have a parallax from each other that was imaged in step St4A. The other data (for example, the second right image P2R) is synthesized (step St6). Further, the CPU 152 or the ISP 153 of the camera device 10 calculates the parallax between the two composite images (for example, the composite left image P3L and the composite right image P3R) obtained by the composition. The CPU 152 or ISP 153 of the camera device 10 uses the parallax calculation results to generate a distance image for specifying the distance between the reference point of the camera device 10 (see above) and the target object, and outputs it to the PC 50. (Step St6).

The CPU 152 or the ISP 153 of the camera device 10 waits for a minute time Δtb to elapse (standby) from the execution timing of step St4A (step St7A). After step St7A, the processing of the camera device 10 returns to step St1A.

In addition, in step St6, the CPU 152 or the ISP 153 of the camera device 10 displays a first left image P1L captured by the first imaging module CAP1 and a first right image captured by the second imaging module CAP2 during the irradiation with the illumination pattern P1. The P1R parallax may be calculated. Similarly, the CPU 152 or the ISP 153 of the camera device 10 determines the parallax difference between the second left image P2L captured by the first imaging module CAP1 and the second right image P2R captured by the second imaging module CAP2 during irradiation with the illumination pattern P2. may be calculated. The CPU 152 or the ISP 153 of the camera device 10 generates a distance image from the parallax based on the first left image P1L and the first right image P1R, and further generates a distance image from the parallax based on the second left image P2L and the second right image P2R. may be generated. The CPU 152 or the ISP 153 of the camera device 10 generates a distance image for specifying the distance between the reference point of the camera device 10 (see above) and the target object by combining these two distance images, and You can also output to

Next, a setting example of the first structured illumination LG1 and the second structured illumination LG2 emitted by the illumination unit 17 will be described with reference to FIGS. 5, 6, and 7. FIG. 5 is a diagram showing an example of a UI setting screen when lighting is performed using a plurality of different lighting patterns. FIG. 6 is a diagram showing an example of a UI setting screen when the same illumination pattern is irradiated with shifted positions. FIG. 7 is a diagram showing an example of a UI setting screen when emitting light with the dot pitch reduced by a specified magnification.

The UI setting screen WD1 shown in FIG. 5 is displayed on the display (not shown) of the PC 50 by the user's operation. The PC 50 displays a pattern image EX1 indicating the illumination pattern P1 corresponding to the first structured illumination LG1 and a pattern image EX2 indicating the illumination pattern P2 corresponding to the second structured illumination LG2 on the UI setting screen WD1. When the pattern image EX1 and the pattern image EX2 are specified and the OK button BT1 is pressed by the user's operation, the PC 50 sets the pattern image EX1 as the illumination pattern P1 corresponding to the first structured illumination LG1, and sets the pattern image EX2 as the illumination pattern P1 corresponding to the first structured illumination LG1. is set as the illumination pattern P2 corresponding to the second structured illumination LG2. The first irradiation mode is a mode in which a pattern (see pattern image EX1 in FIG. 5) in which a plurality of dots (an example of an irradiation figure) are arranged at a first interval is irradiated. The second irradiation mode is a mode in which a pattern (see pattern image EX2 in FIG. 5) in which a plurality of dots (an example of an irradiation figure) are arranged at second intervals is irradiated. Pattern image EX1 and pattern image EX2 both have a black background and a pattern in which a plurality of white dots are arranged irregularly. At least one is different. Two different types of structured illumination having a black background and a plurality of white dots are irradiated onto the object. As a result, even if the object has virtually no texture, such as a white vase placed on a white table, the captured image taken during structured illumination will have a different shape compared to when it is not illuminated with structured illumination. The structured illumination of the illumination patterns P1 and P2 set on the UI setting screen WD1 of 5 is reflected and the image features increase, so the accuracy of parallax calculation of the object (for example, a white table and a white vase) is improved, and the The accuracy of generating distance images of objects is also improved. Note that the pattern image EX1 of the illumination pattern P1 and the pattern image EX2 of the illumination pattern P2 may be provided in units of frames as shown in FIG. 5, or may be provided in units of lines constituting a frame. , or may be provided in units of blocks each consisting of a plurality of pixels.

Furthermore, in the following description, the dots in the pattern image are just an example of the irradiation figure, and the irradiation figure is not limited to dots (dots or minute circles). The irradiation figure may be, for example, not only a circle but also a polygon.

The set contents are input from the PC 50 to the CPU 152 or the ISP 153 via the external interface 155. The CPU 152 or the ISP 153 sets the illumination pattern P1 corresponding to the first structured illumination LG1 and the illumination pattern P2 corresponding to the second structured illumination LG2 in the illumination control unit 16. Thereby, the illumination unit 17 can emit the first structured illumination LG1 having the illumination pattern P1 (see pattern image EX1), and the second structured illumination LG2 having the illumination pattern P2 (see pattern image EX2). It is possible to irradiate In addition, when re-designating a pattern image, for example, the NG button BT2 is pressed by the user's operation.

The UI setting screen WD2 shown in FIG. 6 is displayed on the display (not shown) of the PC 50 by the user's operation. The PC 50 includes a pattern image EX1 indicating an illumination pattern P1 corresponding to the first structured illumination LG1, and an illumination corresponding to the second structured illumination LG2 obtained by shifting the pattern image EX1 by Δm on the two-dimensional coordinate axis (see FIG. 6). A pattern image EX1A indicating pattern P2 is displayed on the UI setting screen WD1. The designation of Δm is confirmed, for example, after the pattern image EX1 is designated by the user's operation, the value (numeric value) of Δm is input into the designation field FLD1 by the user's operation, and the OK button BT1 is pressed. After this determination, the PC 50 sets the pattern image EX1 as the illumination pattern P1 corresponding to the first structured illumination LG1, and sets the pattern image EX1A as the illumination pattern P2 corresponding to the second structured illumination LG2. The first irradiation mode is a mode in which a pattern (see pattern image EX1 in FIG. 6) in which a plurality of dots (an example of an irradiation figure) are arranged at third intervals is irradiated. The second irradiation mode is a mode in which a pattern (see pattern image EX1A in FIG. 6) in which a plurality of dots (an example of an irradiation figure) arranged at third intervals is shifted by a predetermined amount is irradiated. The relationship between pattern image EX1 and pattern image EX1A is similar to the relationship between pattern image EX1 and pattern image EX2, in that both have a black background and a pattern in which a plurality of white dots are irregularly arranged. However, the location, spacing, and number of dots are different. Therefore, in the setting example of FIG. 6 as well, the object is irradiated with two different types of structured illumination in which the background color is black and a plurality of white dots are arranged. As a result, even if the object has virtually no texture, such as a white vase placed on a white table, the captured image taken during structured illumination will have a different shape compared to when it is not illuminated with structured illumination. The structured illumination of the illumination patterns P1 and P2 set on the UI setting screen WD2 of 6 is reflected and the image features increase, so the accuracy of parallax calculation of the object (for example, a white table and a white vase) is improved, and the The accuracy of generating distance images of objects is also improved. Note that the pattern image EX1 of the illumination pattern P1 and the pattern image EX1A of the illumination pattern P2 may be provided in units of frames as shown in FIG. 6, or may be provided in units of lines constituting a frame. , or may be provided in units of blocks each consisting of a plurality of pixels.

The set contents are input from the PC 50 to the CPU 152 or the ISP 153 via the external interface 155. The CPU 152 or the ISP 153 sets the illumination pattern P1 corresponding to the first structured illumination LG1 and the illumination pattern P2 corresponding to the second structured illumination LG2 in the illumination control unit 16. Thereby, the illumination unit 17 can emit the first structured illumination LG1 having the illumination pattern P1 (see pattern image EX1), and the second structured illumination LG2 having the illumination pattern P2 (see pattern image EX1A). It is possible to irradiate In addition, when re-designating a pattern image, for example, the NG button BT2 is pressed by the user's operation.

The UI setting screen WD3 shown in FIG. 7 is displayed on the display (not shown) of the PC 50 by the user's operation. The PC 50 includes a pattern image EX1 showing an illumination pattern P1 corresponding to the first structured illumination LG1, and an illumination pattern P2 corresponding to the second structured illumination LG2, which is obtained by reducing the pitch of two arbitrary dots in the pattern image EX1 by Δr times. A pattern image EX1B showing . is displayed on the UI setting screen WD1. Δr is Δr=(d2/d1), where d1 is the pitch of any two dots in the pattern image EX1, and d2 is the corresponding reduced dot pitch in the pattern image EX1B. The designation of Δr is confirmed, for example, after the pattern image EX1 is designated by the user's operation, the value (numerical value) of Δr is input into the designation field FLD2 by the user's operation, and the OK button BT1 is pressed. After this determination, the PC 50 sets the pattern image EX1 as the illumination pattern P1 corresponding to the first structured illumination LG1, and sets the pattern image EX1B as the illumination pattern P2 corresponding to the second structured illumination LG2. The first irradiation mode is a mode in which a pattern (see pattern image EX1 in FIG. 7) in which a plurality of dots (an example of an irradiation figure) having a first size are arranged is irradiated. The second irradiation mode is a mode in which a pattern (see pattern image EX1B in FIG. 7) in which a plurality of dots (an example of an irradiation figure) having a second size smaller than the first size is arranged is irradiated. The relationship between pattern image EX1 and pattern image EX1B is similar to the relationship between pattern image EX1 and pattern image EX2 or pattern image EX1A, in which both have a black background and a pattern in which a plurality of white dots are irregularly arranged. Although they have one thing in common, the location and number of dots are different. Therefore, in the setting example of FIG. 7 as well, the object is irradiated with two different types of structured illumination in which the background color is black and a plurality of white dots are arranged. As a result, even if the object has virtually no texture, such as a white vase placed on a white table, the captured image taken during structured illumination will have a different shape compared to when it is not illuminated with structured illumination. The structured illumination of the illumination patterns P1 and P2 set on the UI setting screen WD3 of 7 is reflected and the image features increase, so the accuracy of parallax calculation of the object is improved, and the accuracy of generation of the distance image of the object is also improved. do. Note that the pattern image EX1 of the illumination pattern P1 and the pattern image EX1B of the illumination pattern P2 may be provided in units of frames as shown in FIG. 5, or may be provided in units of lines constituting a frame. , or may be provided in units of blocks each consisting of a plurality of pixels.

The set contents are input from the PC 50 to the CPU 152 or the ISP 153 via the external interface 155. The CPU 152 or the ISP 153 sets the illumination pattern P1 corresponding to the first structured illumination LG1 and the illumination pattern P2 corresponding to the second structured illumination LG2 in the illumination control unit 16. Thereby, the illumination unit 17 can emit the first structured illumination LG1 having the illumination pattern P1 (see pattern image EX1), and the second structured illumination LG2 having the illumination pattern P2 (see pattern image EX1B). It is possible to irradiate In addition, when re-designating a pattern image, for example, the NG button BT2 is pressed by the user's operation.

Further, the camera device 10 generates the illumination pattern P2 corresponding to the second structured illumination LG2 based on the processing result of the AI processing unit 154 on the captured image taken during the irradiation of the object with the first structured illumination LG1. It may also be controlled to change dynamically. This control example will be explained with reference to the set of FIGS. 8 and 9A/9B, and the set of FIGS. 10 and 11, respectively.

FIG. 8 is a diagram showing an example of an operation outline and an example of a UI setting screen when controlling a lighting area using the processing results of the AI processing unit 154. FIG. 9A is a flowchart illustrating an example of the third operation procedure of the camera device 10. FIG. 9B is a flowchart showing another example of the third operation procedure of the camera device 10. In the description of FIGS. 9A and 9B, illumination of the first structured illumination LG1 and imaging under the first structured illumination LG1, illumination of the second structured illumination LG2 and imaging under the second structured illumination LG2 are described. , are executed synchronously. In the explanation of FIG. 9B, the same step numbers are assigned to processes that overlap with the explanation of FIG. 9A to simplify or omit the explanation, and different contents will be explained.

Before the structured illumination is irradiated from the illumination unit 17 of the camera device 10, the captured image IMG0 captured by either the first imaging module CAP1 or the second imaging module CAP2 (for example, the first imaging module CAP1) includes: A background object BKG1 with substantially no texture and a workpiece OB1 are shown. Since there are few image features in the captured image IMG0 from the first imaging module CAP1 and the second imaging module CAP2, it is difficult to accurately grasp the distance from the camera device 10 to the workpiece OB1.

Therefore, as shown in FIG. 8, the first imaging module CAP1 and the second imaging module CAP2 of the camera device 10 are irradiated with the first structured illumination LG1 (for example, the illumination pattern P1 corresponding to the pattern image EX1) from the illumination unit 17. In this state, the workpiece OB1 and the background object are imaged. Through this imaging, a captured image IMG1 is obtained. The AI processing unit 154 of the camera device 10 uses AI (for example, a trained model for segmentation) to perform region determination processing for determining the region of the workpiece OB1 in the captured image IMG1. The CPU 152 or the ISP 153 of the camera device 10 sets the third irradiation mode to the illumination control unit 16. That is, the CPU 152 or the ISP 153 of the camera device 10 determines the irradiation area for irradiating the second structured illumination LG2 (an example of the first light) based on the result of the area determination process by the AI processing unit 154, and The illumination control unit 16 is controlled to irradiate the determined irradiation area with the second structured illumination LG2. Further, the CPU 152 or the ISP 153 of the camera device 10 applies a second light (a light different from the first light) to an area that does not correspond to the area of the workpiece OB1 in the captured image IMG1 based on the result of the area determination process by the AI processing unit 154. The illumination control unit 16 is controlled to irradiate the light (an example). The illumination control unit 16 causes the illumination unit 17 to irradiate the second structured illumination LG2 so as to narrow down (restrict) the irradiation range to the determined irradiation area, and to apply the structured illumination to areas other than the irradiation area. The illumination section 17 irradiates the second light (for example, white light).

The first imaging module CAP1 and the second imaging module CAP2 of the camera device 10 are provided with a second structured illumination LG2 (for example, a part of the illumination pattern P1 corresponding to the pattern image EX1 corresponding to the area of the irradiation area) from the illumination unit 17. The workpiece OB1 and the background object are imaged while being irradiated. Through this imaging, a captured image IMG2 is obtained. The captured image IMG2 includes an image region CT1 of a portion illuminated with the second structured illumination LG2 and an image region CT0 of a portion not illuminated with the second structured illumination LG2 within the entire object within the viewing angle. exist. The CPU 152 or the ISP 153 of the camera device 10 synthesizes one captured image IMG1 from the first imaging module CAP1 and one captured image IMG2 from the second imaging module CAP2 so that the workpiece OB1 overlaps, and creates one composite image. Then, the other captured image IMG1 from the first imaging module CAP1 and the other captured image IMG2 from the second imaging module CAP2 are combined so that the workpiece OB1 overlaps to generate the other composite image. The CPU 152 or the ISP 153 of the camera device 10 generates a distance image for specifying the distance from the camera device 10 to the workpiece OB1 based on the two composite images. Further, the CPU 152 or the ISP 153 of the camera device 10 recognizes the workpiece OB1 (for example, a screw) from the two composite images, and generates a recognition result screen WD4 that specifies the size and distance of the workpiece OB1 (for example, a screw) from the recognition result. , may be sent to the PC50. The PC 50 displays the recognition result screen WD4 sent from the camera device 10.

The recognition result screen WD4 includes at least an image of the image area CT1 of the portion irradiated with the second structured illumination LG2, for example, and a display area DTL1 of detailed information on the workpiece OB1 (for example, a screw). Furthermore, a frame WK1 indicating the shape of the workpiece OB1 (for example, a screw) is shown in the image of the image area CT1. This frame WK1 is added by the CPU 152 or the ISP 153 of the camera device 10. The display area DTL1 shows the size (for example, 100 mm x 50 mm) and distance (30 cm) of the recognized workpiece OB1 (for example, a screw) and the distance from the camera device 10 as detailed information about the workpiece OB1 (for example, a screw).

In FIG. 9A, the illumination control unit 16 of the camera device 10 switches to the third illumination mode at time t1 (see FIG. 2), for example, and directs the illumination of the illumination pattern P1 (first structured illumination LG1) toward the target object. Irradiation is performed from the illumination unit 17 (step St1A). Each of the first imaging module CAP1 and the second imaging module CAP2 of the camera device 10 images the object while being irradiated with the first structured illumination LG1 from the illumination unit 17 in step St1A (step St1A). That is, in step St1A, two pieces of imaging data (for example, the first left image P1L and the first right image P1R) having parallax with each other are obtained in synchronization with the irradiation of the first structured illumination LG1 from the illumination unit 17. For example, the first left image P1L is obtained from the first imaging module CAP1, and the first right image P1R is obtained from the second imaging module CAP2. The CPU 152 or the ISP 153 of the camera device 10 waits for a minute time Δta to elapse (standby) from the execution timing of step St1A (step St3A).

Furthermore, the AI processing unit 154 of the camera device 10 uses AI (for example, a trained model for segmentation) to perform region determination processing to determine the region of the workpiece OB1 in the captured image IMG1 (step St11). The CPU 152 or the ISP 153 of the camera device 10 determines the irradiation area for irradiating the second structured illumination LG2 based on the result of the area determination process by the AI processing unit 154 (Step St12).

The CPU 152 or the ISP 153 of the camera device 10 irradiates the irradiation area determined in step St12 with the second structured illumination LG2, and irradiates areas other than the irradiation area with second light (for example, white light). Controls the lighting control section 16. The lighting control unit 16 switches to the second irradiation mode at time t2 (see FIG. 2), for example, and lights the second structured lighting LG2 so as to narrow down (limit) the irradiation range to only the irradiation area determined in step St12. The second light (for example, white light) is irradiated from the illumination section 17 to a region other than the irradiation area (step St4A). Each of the first imaging module CAP1 and the second imaging module CAP2 of the camera device 10 images the object while being irradiated with the second structured illumination LG2 from the illumination unit 17 in step St4A (step St4A). That is, in step St4A, two pieces of imaging data (for example, the second left image P2L and the second right image P2R) having parallax with each other are obtained in synchronization with the irradiation of the second structured illumination LG2 from the illumination unit 17. For example, the second left image P2L is obtained from the first imaging module CAP1, and the second right image P2R is obtained from the second imaging module CAP2.

Note that, as shown in FIG. 9B, the AI processing unit 154 of the camera device 10 uses AI (for example, a trained model for segmentation) that inputs the distance image generated in step St6 to Area determination processing may be performed to determine the area of work OB1 (step St11A). The CPU 152 or the ISP 153 of the camera device 10 may determine the irradiation area for irradiating the second structured illumination LG2 based on the result of the area determination process by the AI processing unit 154 (Step St12A). After step St12A, the processing of the camera device 10 returns to step St1A.

FIG. 10 is a diagram showing an example of an operation outline and an example of a UI setting screen when controlling the illumination intensity using the processing results of the AI processing unit 154. FIG. 11A is a flowchart illustrating an example of the fourth operation procedure of the camera device 10. FIG. 11B is a flowchart showing another example of the fourth operation procedure of the camera device 10. In the description of FIGS. 11A and 11B, illumination of the first structured illumination LG1 and imaging under the first structured illumination LG1, illumination of the second structured illumination LG2 and imaging under the second structured illumination LG2 are described. , are executed synchronously. In the explanation of FIG. 11B, the same step numbers are assigned to processes that overlap with the explanation of FIG. 11A to simplify or omit the explanation, and different contents will be explained.

Before the structured illumination is irradiated from the illumination unit 17 of the camera device 10, the captured image IMG0 captured by one of the first imaging module CAP1 and the second imaging module CAP2 (for example, the first imaging module CAP1) includes: A background object BKG1 with substantially no texture and a workpiece OB1 are shown. Since there are few image features in the captured image IMG0 from the first imaging module CAP1 and the second imaging module CAP2, it is difficult to accurately grasp the distance from the camera device 10 to the workpiece OB1.

Therefore, as shown in FIG. 10, the first imaging module CAP1 and the second imaging module CAP2 of the camera device 10 are irradiated with the first structured illumination LG1 (for example, the illumination pattern P1 corresponding to the pattern image EX1) from the illumination unit 17. In this state, the workpiece OB1 and the background object are imaged. Through this imaging, a captured image IMG1 is obtained. The AI processing unit 154 of the camera device 10 uses AI (for example, a trained model for segmentation) to perform region determination processing for determining the region of the workpiece OB1 in the captured image IMG1. The CPU 152 or the ISP 153 of the camera device 10 sets the fourth irradiation mode to the illumination control unit 16. That is, the CPU 152 or the ISP 153 of the camera device 10 determines the first irradiation intensity (for example, brightness) of the second structured illumination LG2 based on the result of the area determination process by the AI processing unit 154 (for example, the brightness in the area of the workpiece OB1). and an irradiation area for irradiating with the second structured illumination LG2. The CPU 152 or ISP 153 of the camera device 10 controls the illumination control unit 16 to irradiate the determined irradiation area with the second structured illumination LG2 having the determined first irradiation intensity (for example, first brightness). . Further, the CPU 152 or the ISP 153 of the camera device 10 applies a second irradiation intensity (the first irradiation intensity is The illumination control unit 16 is controlled to irradiate the second structured illumination LG2 with different irradiation intensities (an example). The illumination control unit 16 causes the illumination unit 17 to irradiate the irradiation area of the determined workpiece OB1 with the second structured illumination LG2 at the first irradiation intensity, and to irradiate the area other than the irradiation area with the second irradiation intensity (for example, the second structured illumination LG2). The second structured illumination LG2 having a second luminance different from the first luminance) is emitted from the illumination unit 17. For example, when the background object has a color close to black, the reflectance decreases, so the recognition accuracy of the workpiece may deteriorate with only the captured image IMG1 captured during irradiation with the first structured illumination LG1. Therefore, the camera device 10 aims to improve the recognition accuracy of the workpiece by being irradiated with the second structured illumination LG2, which has a higher irradiation intensity (for example, brightness) than the first structured illumination LG1.

The first imaging module CAP1 and the second imaging module CAP2 of the camera device 10 image the workpiece OB1 and the background object while being irradiated with the second structured illumination LG2 from the illumination unit 17. Through this imaging, a captured image IMG3 is obtained. The captured image IMG3 is an image in which an object within the same angle of view as the captured image IMG1 is imaged, and is illuminated with a second structured illumination LG2 having a different irradiation intensity (for example, brightness) than the first structured illumination LG1. It is. The CPU 152 or the ISP 153 of the camera device 10 combines one of the captured images IMG1 from the first imaging module CAP1 and one of the captured images IMG3 from the second imaging module CAP2 so that the workpiece OB1 overlaps, and creates one composite image. Then, the other captured image IMG1 from the first imaging module CAP1 and the other captured image IMG3 from the second imaging module CAP2 are combined so that the workpiece OB1 overlaps to generate the other composite image. The CPU 152 or the ISP 153 of the camera device 10 generates a distance image for specifying the distance from the camera device 10 to the workpiece OB1 based on the two composite images. Further, the CPU 152 or the ISP 153 of the camera device 10 may generate a UI setting screen WD5 for setting the irradiation intensity (for example, brightness) of the second structured illumination LG2 according to a user's operation, and send it to the PC 50. The PC 50 displays the UI setting screen WD5 sent from the camera device 10.

The UI setting screen WD5 shown in FIG. 10 is displayed on the display (not shown) of the PC 50 when sent from the camera device 10. The PC 50 displays a UI setting screen WD5 including a specification field FLD3 for prompting the user to input the relative ratio of the irradiation intensity (for example, brightness) of the second structured lighting LG2, an OK button BT1, and an NG button BT2. . The designation of the relative ratio of illumination intensity is confirmed by inputting it into the designation field FLD3 through a user operation and pressing the OK button BT1. After this determination, the PC 50 sets parameter values that respectively indicate the irradiation intensity (for example, brightness) of the first structured illumination LG1 and the irradiation intensity (for example, brightness) of the second structured illumination LG2.

This set parameter value is input from the PC 50 to the CPU 152 or the ISP 153 via the external interface 155. The CPU 152 or the ISP 153 sets the illumination intensity (for example, brightness) of the first structured illumination LG1 and a pattern image corresponding to the illumination pattern, and the illumination intensity (for example, brightness) of the second structured illumination in the illumination control unit 16. Thereby, the illumination unit 17 can emit the first structured illumination LG1 having the illumination pattern P1 (see pattern image EX1), and has the same illumination pattern P1 but different illumination intensities (for example, brightness). A second structured illumination LG2 can be emitted. Note that when respecifying the relative ratio of illumination intensity, for example, the NG button BT2 is pressed by the user's operation.

In FIG. 11A, the illumination control unit 16 of the camera device 10 switches to the fourth illumination mode at time t1 (see FIG. 2), for example, and directs the illumination of the illumination pattern P1 (first structured illumination LG1) toward the target object. Irradiation is performed from the illumination unit 17 (step St1A). Each of the first imaging module CAP1 and the second imaging module CAP2 of the camera device 10 images the object while being irradiated with the first structured illumination LG1 from the illumination unit 17 in step St1A (step St1A). That is, in step St1A, two pieces of imaging data (for example, the first left image P1L and the first right image P1R) having parallax with each other are obtained in synchronization with the irradiation of the first structured illumination LG1 from the illumination unit 17. For example, the first left image P1L is obtained from the first imaging module CAP1, and the first right image P1R is obtained from the second imaging module CAP2. The CPU 152 or the ISP 153 of the camera device 10 waits for a minute time Δta to elapse (standby) from the execution timing of step St1A (step St3A).

Furthermore, the AI processing unit 154 of the camera device 10 uses AI (for example, a trained model for segmentation) to perform region determination processing to determine the region of the workpiece OB1 in the captured image IMG1 (step St11). The CPU 152 or the ISP 153 of the camera device 10 sets the first irradiation intensity (for example, brightness) of the second structured illumination LG2 and the second structured illumination LG2 based on the result of the area determination processing by the AI processing unit 154. The irradiation area is determined (Step St21). Furthermore, the CPU 152 or the ISP 153 of the camera device 10 determines a synthesis coefficient indicating the synthesis ratio of the captured image IMG1 during irradiation with the first structured illumination LG1 and the captured image during irradiation with the second structured illumination LG2 (step St21). This synthesis coefficient may be stored in the memory 151 in advance, or may be dynamically determined based on the brightness within the region of the workpiece OB1 in the captured image IMG1.

The CPU 152 or the ISP 153 of the camera device 10 irradiates the determined irradiation area with the second structured illumination LG2 having the first irradiation intensity (for example, brightness) determined in step St21, and irradiates the area other than the irradiation area. The illumination control unit 16 is controlled to irradiate the second structured illumination LG2 with the second irradiation intensity (for example, brightness). The illumination control unit 16 causes the illumination unit 17 to irradiate the determined irradiation area with the second structured illumination LG2 having the determined first irradiation intensity (for example, brightness), and causes the illumination unit 17 to irradiate the determined irradiation area with the second structured illumination LG2, and to apply the second irradiation to an area other than the irradiation area. The second structured illumination LG2 of high intensity (for example, brightness) is emitted from the illumination unit 17 (step St4A). Each of the first imaging module CAP1 and the second imaging module CAP2 of the camera device 10 images the object while being irradiated with the second structured illumination LG2 from the illumination unit 17 in step St4A (step St4A). That is, in step St4A, two pieces of imaging data (for example, the second left image P2L and the second right image P2R) having parallax with each other are obtained in synchronization with the irradiation of the second structured illumination LG2 from the illumination unit 17. For example, the second left image P2L is obtained from the first imaging module CAP1, and the second right image P2R is obtained from the second imaging module CAP2.

Using the synthesis coefficient determined in step St21, the CPU 152 or ISP 153 of the camera device 10 selects one of the two imaged data (for example, the first left image P1L) that has a parallax from each other that was imaged in step St1A, and the image captured in step St4A. One of the two captured image data having a parallax (for example, the second left image P2L) is synthesized (step St6). Similarly, the CPU 152 or the ISP 153 of the camera device 10 uses the synthesis coefficient determined in step St21 to combine the other of the two pieces of imaged data (for example, the first right image P1R) that have a parallax with each other that was imaged in step St1A. The other image data (for example, second right image P2R) of the two imaged data having a parallax that are imaged in St4A are combined (step St6). Further, the CPU 152 or the ISP 153 of the camera device 10 calculates the parallax between the two composite images (for example, the composite left image P3L and the composite right image P3R) obtained by the composition. The CPU 152 or ISP 153 of the camera device 10 uses the parallax calculation results to generate a distance image for specifying the distance between the reference point of the camera device 10 (see above) and the target object, and outputs it to the PC 50. (Step St6).

Note that, as shown in FIG. 11B, the AI processing unit 154 of the camera device 10 uses AI (for example, a trained model for segmentation) that inputs the distance image generated in step St6 to Area determination processing may be performed to determine the area of work OB1 (step St11A). The CPU 152 or the ISP 153 of the camera device 10 sets the first irradiation intensity (for example, brightness) of the second structured illumination LG2 and the second structured illumination LG2 based on the result of the area determination processing by the AI processing unit 154. The irradiation area may be determined (Step St21A). After step St21A, the processing of the camera device 10 returns to step St1A.

Further, the CPU 152 or the ISP 153 of the camera device 10 may set the fifth irradiation mode to the illumination control unit 16. That is, the CPU 152 or the ISP 153 of the camera device 10 generates the illumination pattern P2 corresponding to the second structured illumination LG2 as a distance image generation result for a captured image captured while the object is irradiated with the first structured illumination LG1. It may also be controlled to change dynamically based on. An example of this control will be explained with reference to FIG. 12. FIG. 12 is a flowchart illustrating an example of the fifth operation procedure of the camera device 10. In the explanation of FIG. 12, the illumination of the first structured illumination LG1 and the imaging under the first structured illumination LG1, and the illumination of the second structured illumination LG2 and the imaging under the second structured illumination LG2, respectively. executed synchronously.

In FIG. 12, the illumination control unit 16 of the camera device 10 switches to the fifth illumination mode at time t1 (see FIG. 2), for example, and directs the illumination of the illumination pattern P1 (first structured illumination LG1) toward the object. Irradiation is performed from the illumination unit 17 (step St1A). Each of the first imaging module CAP1 and the second imaging module CAP2 of the camera device 10 images the object while being irradiated with the first structured illumination LG1 from the illumination unit 17 in step St1A (step St1A). That is, in step St1A, two pieces of imaging data (for example, the first left image P1L and the first right image P1R) having parallax with each other are obtained in synchronization with the irradiation of the first structured illumination LG1 from the illumination unit 17. For example, the first left image P1L is obtained from the first imaging module CAP1, and the first right image P1R is obtained from the second imaging module CAP2. The CPU 152 or the ISP 153 of the camera device 10 waits for a minute time Δta to elapse (standby) from the execution timing of step St1A (step St3A).

Further, the CPU 152 or the ISP 153 of the camera device 10 stores two image data (for example, the first left image P1L and the first right image P1R) that have parallax from each other and that are imaged during the irradiation with the first structured illumination LG1 in step St1A. Parallax is calculated based on , and a distance image for specifying the distance to the target object is generated (Step St31). The CPU 152 or the ISP 153 of the camera device 10 measures the distance from the camera device 10 to the object based on the distance image generated in step St31 (step St32). However, the accuracy of the distance measured in this step St32 does not necessarily have to be high.

The CPU 152 or ISP 153 of the camera device 10 determines the irradiation area for irradiating the second structured illumination LG2 based on the distance to the object measured in step St32. The CPU 152 or the ISP 153 of the camera device 10 controls the illumination control unit 16 to irradiate the irradiation area determined in step St12 with the second structured illumination LG2. The illumination control unit 16 causes the illumination unit 17 to irradiate the second structured illumination LG2 so as to narrow down (restrict) the irradiation range to only the determined irradiation area (step St4A). Each of the first imaging module CAP1 and the second imaging module CAP2 of the camera device 10 images the object while being irradiated with the second structured illumination LG2 from the illumination unit 17 in step St4A (step St4A). That is, in step St4A, two pieces of imaging data (for example, the second left image P2L and the second right image P2R) having parallax with each other are obtained in synchronization with the irradiation of the second structured illumination LG2 from the illumination unit 17. For example, the second left image P2L is obtained from the first imaging module CAP1, and the second right image P2R is obtained from the second imaging module CAP2.

As described above, the camera device 10 according to the first embodiment has the illumination section 17 that irradiates light onto the object and the illumination section 17 in the first irradiation mode and the second irradiation mode that is different from the first irradiation mode. The illumination control unit 16 to control, a first image taken of the object while being irradiated with light in the first irradiation mode, and a second image taken of the object while being irradiated with light in the second irradiation mode. In order to specify the distance between the reference point and the target object based on the imaging unit (for example, the first imaging module CAP1 and the second imaging module CAP2) that generates the image and the first captured image and the second captured image. A generation unit (for example, CPU 152 or ISP 153) that generates a distance image. Thereby, since the camera device 10 uses two types of structured illumination with different illumination patterns, it is possible to recognize the distance to the target object with high accuracy.

The imaging unit also includes a first imaging module CAP1 that images the object from a first viewpoint, and a second imaging module CAP2 that images the object from a second viewpoint different from the first viewpoint. The generation unit generates a first composite image (for example, a composite left image P3L) obtained by combining a first captured image captured from the first viewpoint and a second captured image captured from the first viewpoint, and a first composite image captured from the second viewpoint. A distance image is generated based on a second composite image (eg, composite right image P3R) obtained by combining the captured image and a second captured image captured from the second viewpoint. Thereby, the camera device 10 can calculate the parallax by apparently increasing each image feature on the composite left image P3L and the composite right image P3R through the composition process, and can recognize the distance to the object with high accuracy. Can generate distance images.

Further, the first irradiation mode is a mode in which a pattern (for example, pattern image EX1) in which a plurality of irradiation figures are arranged at a first interval is irradiated. The second irradiation mode is a mode in which a pattern (for example, pattern image EX2) in which a plurality of irradiation figures are arranged at a second interval different from the first interval is irradiated. Thereby, the camera device 10 uses the first structured illumination LG1 and the second structured illumination LG2, which have two types of illumination patterns in which the arrangement of a plurality of dots is different, to improve the accuracy of parallax calculation of the object. It is also possible to improve the accuracy of generating distance images of objects.

Further, the first irradiation mode is a mode in which a pattern (for example, pattern image EX1) in which a plurality of irradiation figures are arranged at a third interval is irradiated. The second irradiation mode is a mode in which a pattern in which a plurality of irradiation figures arranged at third intervals are shifted by a predetermined amount is irradiated. As a result, the camera device 10 can easily shift one illumination pattern by a predetermined amount in a predetermined direction, and the first structured illumination light LG1 and The two-structured illumination LG2 can be used, the accuracy of calculating the parallax of the object can be improved, and the accuracy of generating a distance image of the object can also be improved.

Further, the first irradiation mode is a mode in which a pattern in which irradiation figures having a first size are arranged is irradiated. The second irradiation mode is a mode in which a pattern in which irradiation figures having a second size smaller than the first size are arranged is irradiated. As a result, the camera device 10 uses a simple method of reducing the dot pitch of any two dots in one illumination pattern by a predetermined factor to create a first structure having two types of illumination patterns in which the arrangement of a plurality of dots is different. The structured illumination LG1 and the second structured illumination LG2 can be used, the accuracy of parallax calculation of the object can be improved, and the accuracy of generation of a distance image of the object can also be improved.

In addition, the camera device 10 further includes an AI processing unit 154 that determines the area of the object in the image captured under the first structured illumination LG1. The illumination control unit 16 causes the illumination unit 17 to execute a third irradiation mode in which a region of the object is irradiated with the first light and a region that does not correspond to the object is irradiated with the second light different from the first light. Thereby, the camera device 10 can dynamically determine the area of the object in the image captured under the first structured illumination LG1, even if the object is a movable object, and move the object to the appropriate area. A second structured illumination LG2 can be emitted.

In addition, the camera device 10 further includes an AI processing unit 154 that determines the area of the object in the image captured under the first structured illumination LG1. The illumination control unit 16 irradiates an area of the object with a first irradiation intensity determined based on the brightness of the area of the object in the image, and irradiates an area that does not correspond to the object with a second irradiation intensity different from the first irradiation intensity. The illumination unit 17 is caused to execute a fourth irradiation mode in which irradiation is performed by. As a result, even if the background object has a color close to black, for example, the camera device 10 can emit the second structured illumination LG2 with increased irradiation intensity (for example, brightness) compared to the first structured illumination LG1. , it is possible to improve the recognition accuracy of the workpiece.

Further, the illumination control unit 16 determines the distance between the reference point and the target object from a first distance image (e.g., corresponding to the distance image generated in step St31 in FIG. 12) based on the first image taken from the first viewpoint. The illumination unit 17 is caused to execute a fifth illumination mode in which the illumination area is determined based on the distance between the reference point and the object and is illuminated. Thereby, the camera device 10 measures the distance to the object from the above-described first distance image generated only by irradiation with the first structured illumination LG1, and uses this distance measurement result. The irradiation area of the second structured lighting LG2 can be changed by Therefore, even if the target object is a movable object, for example, the camera device 10 can roughly grasp the distance and position to the target object from the imaging result under the first structured illumination LG1. The second structured illumination LG2 can be appropriately irradiated to an area where there is a high possibility of being exposed.

Furthermore, the irradiation figure is either circular or polygonal. Thereby, the camera device 10 can increase the image features of the captured image under the first structured illumination LG1 and the captured image under the second structured illumination LG2, and can increase the image characteristics of the captured image under the first structured illumination LG1 and the captured image under the second structured illumination LG2. Highly accurate distance images can be generated. Further, the elements included in the camera device 10 of this embodiment may be operated as a system. For example, the system 100 may be operated by mounting the first imaging module CAP1, the second imaging module CAP2, and elements other than the first imaging module CAP1 and the second imaging module CAP2 in different devices.

Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that those skilled in the art can come up with various changes, modifications, substitutions, additions, deletions, and equivalent examples within the scope of the claims, and It is understood that it naturally falls within the technical scope of the present disclosure. Further, each component in the various embodiments described above may be combined as desired without departing from the spirit of the invention.

Note that this application is based on a Japanese patent application (Japanese Patent Application No. 2022-059664) filed on March 31, 2022, the contents of which are incorporated as a reference in this application.

The present disclosure is useful as a camera device, image generation method, and system that recognize the distance to an object with high accuracy.

10 Camera device 11 L lens 12 R lens 13 L image sensor 14 R image sensor 15 SoC
16 Lighting control section 17 Lighting section 50 PC
100 System 151 Memory 152 CPU
153 ISP
154 AI processing unit 155 External interface CAP1 First imaging module CAP2 Second imaging module

Claims

an illumination unit that irradiates a target with light in a first irradiation mode and a second irradiation mode different from the first irradiation mode;
a lighting control section that controls the lighting section;
Generating a first captured image of the object while being irradiated with light in the first irradiation mode and a second captured image of the object while being irradiated with light in the second irradiation mode. an imaging unit;
a generation unit that generates a distance image for specifying the distance between the reference point and the target object based on the first captured image and the second captured image;
camera equipment.
The imaging unit includes a first imaging module that images the object from a first viewpoint, and a second imaging module that images the object from a second viewpoint different from the first viewpoint,
The generation unit generates a first composite image obtained by combining the first captured image captured from the first viewpoint and the second captured image captured from the first viewpoint, and a first composite image obtained by synthesizing the first captured image captured from the first viewpoint, and the first composite image captured from the second viewpoint. generating the distance image based on a second composite image obtained by combining the first captured image and the second captured image captured from the second viewpoint;
The camera device according to claim 1.
The first irradiation mode is a mode in which a pattern in which a plurality of irradiation figures are arranged at a first interval is irradiated,
The second irradiation mode is a mode in which a pattern in which a plurality of irradiation figures are arranged at a second interval different from the first interval is irradiated.
The camera device according to claim 1.
The first irradiation mode is a mode in which a pattern in which a plurality of irradiation figures are arranged at third intervals is irradiated,
The second irradiation mode is a mode in which a pattern in which a plurality of irradiation figures arranged at the third interval are shifted by a predetermined amount is irradiated.
The camera device according to claim 1.
The first irradiation mode is a mode in which a pattern in which irradiation figures having a first size are arranged is irradiated,
The second irradiation mode is a mode in which a pattern in which irradiation figures having a second size smaller than the first size are arranged is irradiated.
The camera device according to claim 1.
an AI processing unit that determines a region of the object in an image captured under the first irradiation mode;
The lighting control section includes:
causing the illumination unit to execute a third irradiation mode in which a region of the target object is irradiated with first light and a region not corresponding to the target object is irradiated with second light different from the first light;
The camera device according to claim 1.
an AI processing unit that determines a region of the object in an image captured under the first irradiation mode;
The lighting control section includes:
The area of the object is irradiated with a first irradiation intensity determined based on the brightness of the area of the object in the image, and the area that does not correspond to the object is irradiated with a second irradiation intensity different from the first irradiation intensity. causing the illumination unit to execute a fourth irradiation mode for irradiation;
The camera device according to claim 1.
The lighting control section includes:
Determine the distance between the reference point and the object from a first distance image based on the first image taken from the first viewpoint, and calculate the distance between the reference point and the object based on the distance between the reference point and the object. causing the illumination unit to execute a fifth illumination mode in which illumination is applied to an illumination area determined by
The camera device according to claim 2.
The irradiation figure is
Either circular or polygonal,
The camera device according to any one of claims 3 to 5.
controlling an illumination unit that irradiates the object with light in a first irradiation mode and a second irradiation mode different from the first irradiation mode;
Generating a first captured image of the object while being irradiated with light in the first irradiation mode and a second captured image of the object while being irradiated with light in the second irradiation mode. step and
generating a distance image for specifying the distance between the reference point and the target object based on the first captured image and the second captured image;
Image generation method.
an illumination unit that irradiates a target with light in a first irradiation mode and a second irradiation mode different from the first irradiation mode;
a lighting control section that controls the lighting section;
Generating a first captured image of the object while being irradiated with light in the first irradiation mode and a second captured image of the object while being irradiated with light in the second irradiation mode. an imaging unit;
a generation unit that generates a distance image for specifying the distance between the reference point and the target object based on the first captured image and the second captured image;
system.