JP7399160B2 - Light pattern irradiation control device and light pattern irradiation control method - Google Patents

Description

The present invention relates to a light pattern irradiation control device and a light pattern irradiation control method.

2. Description of the Related Art A technique for generating depth data from an image photographed by a photographing unit is known. The depth data indicates, for example, the position of each position in real space appearing in a photographed image in a direction intersecting the projection plane of the image.

For planar objects such as floors, walls, tables, etc., it is generally difficult to generate depth data because there are no or only a few feature points. In order to generate highly accurate depth data even in such locations, a high-power projector is used to generate a given light pattern, such as a random pattern or a pattern with a predetermined shape, in synchronization with the camera's exposure timing. There is a technology to irradiate.

However, with the above technology, the light pattern is irradiated over a wide range, resulting in wasteful power consumption.

The present invention has been made in view of the above circumstances, and one of its objects is to provide a light pattern irradiation control device and a light pattern irradiation control device capable of reducing power consumption related to irradiation of a given light pattern for generating depth data. An object of the present invention is to provide a pattern irradiation control method.

In order to solve the above problems, a light pattern irradiation control device according to the present invention provides an irradiation device that specifies, based on a photographed image, an irradiation area that occupies a part of the image and is to be irradiated with a given light pattern. an area identification unit; an irradiation control unit that controls the irradiation unit to irradiate the light pattern to a location in real space appearing in the irradiation area; and a depth data generation unit that generates depth data for each position in the real space appearing in the image.

In one aspect of the present invention, the irradiation area specifying unit specifies the irradiation area based on reliability of an execution result of stereo matching processing.

Further, in one aspect of the invention, the irradiation area specifying unit specifies the irradiation area based on the number of feature points extracted for each of a plurality of areas in a captured image.

Further, in one aspect of the present invention, the irradiation control section controls at least one of the position and orientation of the irradiation section.

Alternatively, the irradiation control section controls at least one of the position and orientation of a reflecting member included in the irradiation section that reflects the light pattern.

Alternatively, the irradiation unit includes a plurality of light-emitting modules that emit the light pattern, and the irradiation control unit selects a light emitting module from among the plurality of light-emitting modules to illuminate the area in the real space appearing in the irradiation area. The light emitting module associated with the location is selected, and the selected light emitting module is controlled to emit the light pattern.

Alternatively, the irradiation unit includes a scanning projector that irradiates a location including a part of the location in real space that appears in the irradiation area, and the irradiation control unit directs the light pattern to the irradiation area. The scanning projector is controlled to illuminate a location in the displayed real space.

Alternatively, the irradiation unit is mounted on a robot, and the irradiation control unit controls at least one of the position and orientation of the robot.

Further, the light pattern irradiation control method according to the present invention includes a step of specifying, based on a photographed image, an irradiation area that occupies a part of the image and is to be irradiated with a given light pattern; , controlling an irradiation unit to irradiate a location in the real space that appears in the irradiation area; and based on an image of the location where the light pattern is irradiated, the light pattern in the real space that appears in the image. generating depth data for each position.

1 is a diagram showing an example of the configuration of a light pattern irradiation control device according to an embodiment of the present invention. FIG. 3 is a flow diagram showing an example of processing executed by the light pattern irradiation control device according to an embodiment of the present invention. FIG. 3 is a diagram schematically showing an example of irradiation control. FIG. 3 is a diagram schematically showing an example of irradiation control. FIG. 3 is a diagram schematically showing an example of irradiation control. FIG. 3 is a diagram showing an example of the relationship between exposure timing and light emission timing. FIG. 6 is a diagram schematically showing an example of an image taken when no light pattern is irradiated. It is a figure which shows typically an example of the image image|photographed when irradiation of a light pattern was performed.

Hereinafter, one embodiment of the present invention will be described with reference to the drawings.

FIG. 1 is a diagram showing an example of the configuration of a light pattern irradiation control device 1 according to an embodiment of the present invention.

As shown in FIG. 1, the light pattern irradiation control device 1 according to the present embodiment includes an imaging section 10, a depth engine 12, a feature extraction engine 14, a CPU (Central Processing Unit) 16, a driver 18, and an irradiation section 20. included. The imaging unit 10 is, for example, a stereo camera, and includes a camera 22a, a camera 22b, an ISP (Image Signal Processor) 24a, and an ISP 24b.

The photographing unit 10 according to this embodiment photographs images at a predetermined frame rate (for example, 120 fps or 60 fps). In this embodiment, the timing of image capture by the camera 22a and the image capture timing of the camera 22b are synchronized, and a pair of images are generated at a predetermined frame rate.

The camera 22a and the camera 22b according to this embodiment are cameras capable of photographing both visible light and infrared light. Here, the camera 22a and the camera 22b may be, for example, a monochrome camera from which a filter for cutting infrared light is removed. Alternatively, the cameras 22a and 22b may be color cameras from which a filter for cutting infrared light is removed, for example. Furthermore, the cameras 22a and 22b are equipped with pixels not only for visible light but also for infrared light, and generate visible light data and infrared light data separately. Good too.

An image photographed by the photographing unit 10 is input to a depth engine 12 and a feature extraction engine 14. In this embodiment, an image photographed by the camera 22a is input to the depth engine 12 and the feature extraction engine 14 after being subjected to image processing by the ISP 24a. Further, the image photographed by the camera 22b is input to the depth engine 12 and the feature extraction engine 14 after being subjected to image processing by the ISP 24b.

In this embodiment, the depth engine 12 constitutes a device that, as a depth data generation section, generates depth data and reliability data based on an image photographed by the photographing section 10, for example. For example, the depth engine 12 performs stereo matching processing on the image photographed by the photographing unit 10. The stereo matching process includes a process of matching mutually similar regions in a pair of images photographed by the photographing unit 10 by template matching. Here, for example, a process is executed to associate an area within an image photographed by the camera 22a and subjected to image processing by the ISP 24a with an area within an image photographed by the camera 22b and subjected to image processing by the ISP 24b. For example, the above-described region mapping may be performed in units of regions of a predetermined size (for example, 20 pixels square). Hereinafter, this area will be referred to as a unit area. The depth engine 12 then generates depth data and reliability data based on the execution results of the stereo matching process.

Here, the depth data is data indicating the position of each position in real space appearing in a photographed image in a direction intersecting the projection plane of the image. Moreover, reliability data is data indicating the reliability of the execution result of the stereo matching process (for example, the reliability of depth data associated with the reliability data). In the following description, it is assumed that the higher the reliability indicated by the reliability data, the larger the value is set to the reliability data.

The depth data may be, for example, a depth image including a plurality of pixels. In this case, each pixel in the depth image is associated with a unit area. Furthermore, each pixel in the depth image is also associated with a position in real space that appears in the corresponding unit area. For each pixel in the depth image, for example, a pixel value corresponding to a position in the corresponding real space in a direction intersecting the projection plane of the image photographed by the photographing unit 10 is set.

In this case, reliability data associated with each pixel in the depth image may be generated. Here, the areas with low reliability indicated by the reliability data include, for example, areas where there are no feature points or where the number of feature points is small and therefore cannot be correlated, or areas where multiple similar areas exist. The reliability indicated by the reliability data also decreases due to the occurrence of noise and occlusion.

When a color camera is used, in the stereo matching process, for example, unit areas may be associated with each other based on the luminance value of each pixel. Furthermore, if a camera is used that is equipped with pixels for infrared light as well as for visible light, monochrome pixel data may be used to generate the depth data.

In this embodiment, the feature extraction engine 14 is, for example, a device that executes a process of extracting feature points from an image photographed by the photographing unit 10. Note that the feature extraction engine 14 may not be included in the light pattern irradiation control device 1 according to this embodiment.

In this embodiment, the CPU 16 is, for example, a processor that operates according to a program. The CPU 16 according to the present embodiment, as an irradiation area specifying unit, executes a process of specifying an irradiation area that occupies a part of the image and is to be irradiated with a light pattern, based on the captured image. Here, for example, one or more of the above-mentioned unit areas may be specified as the irradiation area.

In this embodiment, the driver 18 constitutes a device that controls the irradiation unit 20 based on a control signal received from the CPU 16, for example, as an irradiation control unit.

In this embodiment, the irradiation unit 20 is, for example, a device that irradiates an object with light such as infrared light. The irradiation unit 20 may be, for example, a VCSEL (Vertical Cavity Surface Emitting LASER) module or a scanning projector such as a MEMS (Micro Electro Mechanical Systems) projector.

In this embodiment, the driver 18 controls the irradiation unit 20 to irradiate a given light pattern onto a location in real space that appears in the irradiation area specified by the CPU 16. Here, the light pattern may be, for example, a random pattern or a pattern of a predetermined shape.

Note that the above functions may be implemented by the light pattern irradiation control device 1 executing a program including commands corresponding to the above functions. The program is supplied to the light pattern irradiation control device 1 via a computer-readable information storage medium such as an optical disk, a magnetic disk, a magnetic tape, a magneto-optical disk, or a flash memory, or via the Internet. It's okay.

Here, an example of the flow of processing performed in the light pattern irradiation control device 1 according to the present embodiment will be described with reference to the flowchart illustrated in FIG. 2. In this processing example, the processes shown in S101 to S109 shown in FIG. 2 are repeatedly executed so that the processes shown in S101 or S106 shown in FIG. 2 are executed at a predetermined frame rate.

First, the photographing unit 10 photographs a pair of images (S101).

Then, the depth engine 12 performs stereo matching processing on the pair of images photographed in the processing shown in S101 (S102). By executing the stereo matching process, for example, depth data associated with each of the plurality of unit areas and reliability data indicating the reliability of the execution result of the stereo matching process for the unit area are generated. Ru.

Then, the CPU 16 checks whether the reliability value indicated by the reliability data generated in the process shown in S102 is greater than or equal to a predetermined value (S103). Here, for example, it may be checked whether the average value of the reliability values indicated by the reliability data associated with each of the plurality of unit areas is equal to or greater than a predetermined value.

If it is confirmed in the process shown in S103 that the reliability value is greater than or equal to the predetermined value (S103: Y), the process returns to S101.

If it is confirmed in the process shown in S103 that the reliability value is not greater than or equal to the predetermined value (S103: N), the CPU 16 specifies the irradiation area (S104). In the process shown in S104, for example, the CPU 16 specifies the irradiation area based on the reliability of the execution result of the stereo matching process. Here, a unit area corresponding to reliability data indicating reliability lower than a predetermined reliability may be specified as the irradiation area. In this case, multiple irradiation areas may be specified. Alternatively, a unit area corresponding to reliability data indicating the lowest reliability may be specified as the irradiation area.

Then, the driver 18 executes settings for irradiation by the irradiation unit 20 (S105). In the process shown in S105, the driver 18 may perform irradiation settings according to a control signal sent from the CPU 16 to the driver 18. Further, in the process shown in S105, the driver 18 may turn on the power of the irradiation unit 20.

Then, the driver 18 controls the irradiation unit 20 to irradiate a given light pattern to a location in the real space appearing in the irradiation area specified in the process shown in S104, and the imaging unit 10 An image is photographed (S106). In the process shown in S106, the driver 18 may perform irradiation according to a control signal sent from the CPU 16 to the driver 18. In the process shown in S106, the irradiation unit 20 irradiates with a given light pattern during a part of the period during which the cameras 22a and 22b are exposed.

As shown in FIG. 3, the irradiation unit 20 may include a light emitting module 26 that emits a given light pattern and a reflection member 28 that reflects the light pattern. In this case, the driver 18 may control at least one of the position and orientation of the reflecting member 28 in the process shown in S105. Then, in the process shown in S106, a given light pattern may be irradiated onto a location in the real space that appears in the irradiation area.

Further, as shown in FIG. 4, the driver 18 may control at least one of the position and orientation of the irradiation unit 20 itself in the process shown in S105. Then, in the process shown in S106, a given light pattern may be irradiated onto a location in the real space that appears in the irradiation area.

As in the above case, the driver 18 may be an actuator that controls the position and orientation of the irradiation unit 20.

Further, as shown in FIG. 5, the irradiation unit 20 may include a plurality of light emitting modules 26 that emit a given light pattern. Each of the plurality of light emitting modules 26 is associated with a different location in real space. In this case, in the process shown in S105, the driver 18 may select, from among the plurality of light emitting modules 26, the light emitting module 26 associated with the location in the real space appearing in the irradiation area. Then, in the process shown in S106, the driver 18 may control the selected light emitting module 26 to emit a given light pattern.

As in this case, the driver 18 may be a switch that switches the light emitting module 26 to emit a given light pattern.

Further, the irradiation unit 20 may include a scanning projector that irradiates a location including a part of the location in real space that appears in the irradiation area. Then, in the processes shown in S105 and S106, the driver 18 may control the scanning projector to irradiate a given light pattern to a location in the real space appearing in the irradiation area.

Note that in this embodiment, for example, the position and orientation of the imaging unit 10 and the position and orientation of the irradiation unit 20 may be measured. Then, the above-mentioned irradiation control (irradiation setting and irradiation) may be performed based on the result of the measurement.

Furthermore, in the case where the relative position and orientation of the irradiation unit 20 with respect to the position and orientation of the imaging unit 10 are fixed, the correspondence between the position and orientation of the imaging unit 10 and the position and orientation of the irradiation unit 20 is given. It is. Therefore, in this case, it is possible to perform the above-mentioned irradiation control by using a mathematical formula or the like expressing the given correspondence.

Further, for example, the irradiation section 20 may be mounted on a robot, and the relative position and orientation of the irradiation section 20 with respect to the robot may be fixed. Then, by moving the robot, the given light pattern may be irradiated onto a location in real space that appears in the irradiation area. In this case, the driver 18 may control at least one of the position and orientation of the robot on which the irradiation unit 20 is mounted.

For example, the irradiation unit 20 may be mounted on the head of the robot, and the position and orientation of the irradiation unit 20 relative to the head of the robot may be fixed. Then, by moving the robot's head, the given light pattern may be irradiated onto a location in real space that appears in the irradiation area. In this case, the driver 18 may control at least one of the position and orientation of the head of the robot on which the irradiation unit 20 is mounted.

When the process shown in S106 is completed, the depth engine 12 performs stereo matching processing on the pair of images photographed in the process shown in S106 (S107).

Then, the CPU 16 checks whether a predetermined light-off condition is satisfied (S108). Here, if it is confirmed that the light-off condition is satisfied (S108: Y), the driver 18 turns off the power to the irradiation unit 20, turns off the irradiation unit 20 (S109), and performs the process shown in S101. Return to

On the other hand, if it is confirmed that the light-off condition is not satisfied (S108: Y), the process returns to S104.

Here, the process shown in S108 may not be executed, and after the process shown in S107 is executed, the process shown in S109 may always be executed.

In this case, as shown in FIG. 6, frames in which the light emitting module 26 emits light and frames in which the light emitting module 26 does not emit light occur alternately.

As shown in FIG. 6, in this embodiment, for example, in a frame in which the light emitting module 26 emits light, the light emitting module 26 emits light during a part of the period during which the camera 22a and the camera 22b are exposed.

On the other hand, for frames in which the light emitting module 26 does not emit light, the depth engine 12 performs stereo matching processing based on images captured in a situation where the given light pattern is not irradiated. Then, based on the result of the stereo matching process performed on the frame in which the light emitting module 26 does not emit light, the irradiation area for the next frame is set.

For the next frame, the photographing unit 10 photographs the location illuminated with the given light pattern. Then, the depth engine 12 generates depth data for each position in the real space appearing in the image based on an image taken of a place illuminated with a given light pattern.

In the example of FIG. 6, an irradiation area to be irradiated with a given light pattern at time T2 is set based on the result of stereo matching processing based on an image photographed at time T1. Then, at time T3, which partially includes time T2, the irradiated location is photographed. Similarly, an irradiation area to be irradiated with a given light pattern at time T5 is set based on the result of stereo matching processing based on the image photographed at time T4. Then, at time T6, which partially includes time T5, the irradiated location is photographed.

FIG. 7 is a diagram schematically showing an example of an image photographed when irradiation with a given light pattern is not performed. FIG. 8 is a diagram schematically showing an example of an image photographed when the light pattern is irradiated. For example, when the image shown in FIG. 7 is captured at time T1, region R is specified as the irradiation region. Here, an area R on the table where there are no feature points or few feature points is set as the irradiation area. In this case, in the next frame, the region R is irradiated with a given light pattern, and the image shown in FIG. 8 is photographed at time T3.

Furthermore, in the process shown in S108, the CPU 16 may monitor the time elapsed since execution of the processes shown in S104 to S107 was started. Then, when the monitored time exceeds a predetermined time, the CPU 16 may determine that the lights-out time is satisfied, and the driver 18 may execute the process shown in S109. Alternatively, in the process shown in S108, the CPU 16 may monitor the number of times the processes shown in S104 to S107 are consecutively executed without executing the processes shown in S101 to S103. Then, when the number of monitored times exceeds a predetermined number of times, the CPU 16 may determine that the lights-out time is satisfied, and the driver 18 may execute the process shown in S109.

Further, in the process shown in S108, the CPU 16 may identify the difference between the image captured in the immediately previous frame and the image captured in the current frame. If the difference is larger than a predetermined difference, the CPU 16 may determine that the light-off time is satisfied, and the driver 18 may execute the process shown in S109. Alternatively, if the specified difference is smaller than the predetermined difference for a predetermined time or more, the CPU 16 may determine that the lights-out time is satisfied, and the driver 18 may execute the process shown in S109.

Further, the light pattern irradiation control device 1 may include a motion sensor. Then, if the value representing the magnitude of movement, which is the measurement result of the motion sensor in the frame, is larger than a predetermined value, the CPU 16 determines that the lights-out time is satisfied, and the driver 18 executes the process shown in S109. You may. Alternatively, if the value representing the magnitude of the movement described above is smaller than the predetermined value for a predetermined time or longer, the CPU 16 may determine that the lights-out time is satisfied, and the driver 18 may execute the process shown in S109.

In the present embodiment, frames in which the light emitting module 26 emits light and frames in which the position or orientation of the light emitting module 26 is changed may alternately occur. Furthermore, the position and orientation of the light emitting module 26 may be changed after the light emitting module 26 emits light and before the light emitting timing in the next frame.

Further, in the present embodiment, the CPU 16 may calculate the number of feature points extracted by the feature extraction engine 14 from the image for each of a plurality of regions within the image captured by the imaging unit 10. The area may be the same area as the above-mentioned unit area or may be a different area. Then, the CPU 16 may specify the irradiation area based on the calculated number of feature points. For example, an area in a captured image in which the number of calculated feature points is less than a predetermined number may be specified as the irradiation area. Alternatively, for example, an area in a captured image having the least calculated feature points may be specified as the irradiation area.

The reason for the low reliability indicated by the reliability data generated by the depth engine 12 may not be due to the absence of feature points or the lack of feature points. In such a case, it may be effective to specify the irradiation area based on the feature points extracted by the feature extraction engine 14.

Further, for example, the irradiation area may be specified based on the reliability data and the number of calculated feature points. For example, in the process shown in S104, when a plurality of regions corresponding to reliability data indicating a reliability lower than a predetermined reliability are identified, among the plurality of regions, the calculated feature points are the least. The area may be identified as the irradiation area.

Further, the CPU 16 or the feature extraction engine 14 may perform object recognition processing on the image photographed by the photographing unit 10. For example, an area where a predetermined object such as a floor, wall, table, etc. is recognized may be specified as the irradiation area.

Here, the light pattern irradiation control device 1 may include an acceleration sensor. The measurement results of the acceleration sensor (for example, information on the direction of gravity) may be used in the object recognition process described above.

Furthermore, when a plurality of irradiation areas are specified, one of the irradiation areas may be specified based on another criterion (for example, a given degree of importance). Then, the given light pattern may be irradiated onto a location in real space that appears in the specified irradiation area.

Further, when a plurality of irradiation areas are specified, a given light pattern may be sequentially irradiated to a location in the real space appearing in each irradiation area.

Furthermore, in this embodiment, even if the irradiation section 20 irradiates the light pattern, if the reliability indicated by the reliability data does not become higher than a predetermined reliability, the irradiation section 20 does not irradiate the light pattern. You can do it like this. For example, if it is continuously confirmed in the process shown in S103 that the average reliability value is not equal to or greater than the predetermined value over a predetermined number of frames, the light pattern irradiation control device 1 does not execute the processes shown in S104 and S105. You may then return to the process shown in S101. For example, if the cause of the low reliability is occlusion or noise, even if a light pattern is irradiated, the reliability indicated by the reliability data may not become higher than a predetermined reliability.

Further, in this embodiment, the light pattern irradiation control device 1 may include a motion sensor. If the amount of movement from the immediately previous frame is smaller than a predetermined amount of movement, the irradiation unit 20 may not emit the light pattern.

In this embodiment, as described above, based on the image photographed by the photographing unit 10, an irradiation area that occupies a part of the image and is to be irradiated with a light pattern is specified. The irradiation unit 20 is then controlled to irradiate a given light pattern onto a location in real space that appears in the irradiation area. Therefore, according to this embodiment, power consumption can be reduced more than when a given light pattern is irradiated over a wide range. For example, when a light pattern is irradiated to a place at a predetermined distance from the imaging unit 10, if the FOV (Field of View) is large, the irradiation output is proportional to the square of the irradiation radius. Therefore, if the irradiation radius is halved, the irradiation output will be 1/4.

Note that the present invention is not limited to the above-described embodiments.

For example, the imaging section 10 and the irradiation section 20 may be provided in different devices.

Furthermore, for example, an imaging unit (hereinafter referred to as a first imaging unit) that photographs an image used to generate depth data, and an imaging unit (hereinafter referred to as a second imaging unit) that photographs an image used to specify an irradiation area. ) may be a separate entity. In this case, irradiation control may be performed based on the image photographed by the second photographing section. Then, the first photographing unit may photograph an object illuminated with a given light pattern by the illumination control. Then, depth data may be generated based on the image photographed by the first photographing section. Here, for example, the second imaging section and the irradiation section 20 may be provided in an integrated device, and the first imaging section may be provided in a different device from the device.

Furthermore, in this embodiment, the irradiation area does not need to be specified based on the pair of images. For example, the feature extraction engine 14 may extract feature points from one image. Then, based on the result of the extraction, the irradiation area within the image may be specified.

Furthermore, the present invention is also applicable to the generation of depth data using methods other than the stereo matching method.

Further, the specific character strings and numerical values mentioned above and the specific character strings and numerical values in the drawings are merely examples, and the present invention is not limited to these character strings and numerical values.

Claims (4)

an irradiation area specifying unit that specifies an irradiation area that occupies a part of the image and is to be irradiated with a given light pattern based on the photographed image;
an irradiation control unit that controls an irradiation unit to irradiate the light pattern to a location in real space appearing in the irradiation area;
a depth data generation unit that generates depth data for each position in real space appearing in the image based on an image taken of a place irradiated with the light pattern ;
The irradiation unit includes a plurality of light emitting modules that emit the light pattern,
The irradiation control unit selects the light emitting module associated with a location in real space appearing in the irradiation area from among the plurality of light emitting modules, and causes the selected light emitting module to emit the light pattern. control to
A light pattern irradiation control device characterized by : The irradiation area specifying unit specifies the irradiation area based on the reliability of the execution result of the stereo matching process.
The light pattern irradiation control device according to claim 1, characterized in that: The irradiation area specifying unit specifies the irradiation area based on the number of feature points extracted for each of a plurality of areas in the captured image.
The light pattern irradiation control device according to claim 1 or 2, characterized in that: an irradiation area identification step of identifying an irradiation area that occupies a part of the image and is to be irradiated with a given light pattern, based on the photographed image;
an irradiation control step of controlling an irradiation unit to irradiate the light pattern to a location in real space appearing in the irradiation area;
a depth data generation step of generating depth data for each position in real space appearing in the image based on an image taken of a place irradiated with the light pattern ;
The irradiation unit includes a plurality of light emitting modules that emit the light pattern,
In the irradiation control step, a light emitting module that is associated with a location in real space appearing in the irradiation area is selected from among the plurality of light emitting modules, and the selected light emitting module emits the light pattern. controlled to
A light pattern irradiation control method characterized by :

Images