KR102284187B1 - Apparatus for stereo matching - Google Patents

Abstract

A stereo matching device is disclosed. According to the stereo matching apparatus according to an embodiment of the present invention, by acquiring an RGB image and/or an IR image and using them for depth map extraction, stereo matching can be accurately performed even in low light.

Description

Stereo matching device {APPARATUS FOR STEREO MATCHING}

The present invention relates to a stereo matching device.

As is well known, a person's vision is one of the senses for obtaining information about the surrounding environment, and through both eyes, the position of an object and its proximity can be recognized. In other words, visual information coming through both eyes is synthesized into one distance information, and activities are freely performed.

A stereo camera system is used to implement this visual structure into a machine.

In a stereo camera system, stereo matching is performed on images obtained by using two cameras. In this stereo matching process, the stereo camera obtains a depth map by using the binocular difference between the two cameras. This depth map data is used for recognition.

The technical problem to be solved by the present invention is to provide a stereo matching device that is used for stereo matching by acquiring an IR image in low light.

It may include a stereo matching unit that controls to acquire any one and performs stereo matching on the left/right images received from the camera unit.

In one embodiment of the present invention, the camera unit, a left image acquisition unit for acquiring any one of the left / right RGB image, RGB / IR image and IR image for the target; a right image acquisition unit for acquiring any one of left/right RGB images, RGB/IR images, and IR images of the target; a light receiving unit for detecting illuminance; And under the control of the stereo matching unit, it may include at least one or more radiating unit for emitting infrared light to the target.

In an embodiment of the present invention, the radiation unit may include a single or a plurality of infrared elements.

In an embodiment of the present invention, the infrared device may be any one of LD and LED.

In an embodiment of the present invention, the camera unit may further include a splitter disposed at the front end of the radiating unit and spreading infrared rays emitted from the radiating unit in a predetermined pattern.

In one embodiment of the present invention, the stereo matching unit, when the illuminance detected by the light receiving unit is equal to or less than a predetermined illuminance, a control unit for controlling the emission unit to change to an on state; a synchronization unit for adjusting the left/right images received from the left image acquisition unit and the right image acquisition unit to be received at the same time; an alignment unit for correcting the synchronized left/right images to be positioned on the same epipolar line; and a matching unit configured to extract a depth map by performing stereo matching on the aligned left and right images.

In an embodiment of the present invention, the stereo matching unit may further include a switching unit for switching on/off of the radiation unit under the control of the control unit.

In an embodiment of the present invention, it may further include an encoder unit for receiving calibration-related data from the camera unit and converting it into an alignment parameter.

In an embodiment of the present invention, the alignment unit may correct the left/right image by using the alignment parameter received from the encoder unit.

In an embodiment of the present invention, the control unit controls to change the emitting unit to an on state when the illuminance detected by the light receiving unit is less than or equal to a predetermined illuminance, and the left image acquisition unit and the right image acquisition unit are RGB/IR images or IR images You can control the acquisition of images.

In addition, in order to solve the above technical problem, a stereo matching device according to an embodiment of the present invention is a camera for acquiring left/right RGB images, or RGB images and IR images (RGB/IR images) for a target. wealth; and a stereo matching unit that controls the camera unit to acquire an RGB image or an RGB/IR image, and performs stereo matching on left/right images received from the camera unit.

In one embodiment of the present invention, the camera unit, a left image acquisition unit for acquiring the left / right RGB image or RGB / IR image of the target; a right image acquisition unit for acquiring left/right RGB images or RGB/IR images of the target; And under the control of the stereo matching unit, it may include at least one or more radiating unit for emitting infrared light to the target.

In an embodiment of the present invention, the radiation unit may include a single or a plurality of infrared elements.

In an embodiment of the present invention, the infrared device may be any one of LD and LED.

In an embodiment of the present invention, the camera unit may further include a splitter disposed at the front end of the radiating unit and spreading infrared rays emitted from the radiating unit in a predetermined pattern.

In an embodiment of the present invention, the stereo matching unit includes: a synchronization unit for adjusting the left/right images received from the left image acquisition unit and the right image acquisition unit to be received at the same time; an alignment unit for correcting the synchronized left/right images to be positioned on the same epipolar line; a matching unit for extracting a depth map by performing stereo matching on the aligned left/right images; and a light intensity detector controlling the emission unit to be turned on when the average of pixels of the left/right image is less than or equal to a threshold value.

In an embodiment of the present invention, the stereo matching unit may further include a switching unit for switching on/off of the radiation unit under the control of the light intensity detection unit.

In an embodiment of the present invention, it may further include an encoder unit for receiving calibration-related data from the camera unit and converting it into an alignment parameter.

In an embodiment of the present invention, the alignment unit may correct the left/right image by using the alignment parameter received from the encoder unit.

In one embodiment of the present invention, the control unit, when the average of the pixels of the left/right image is equal to or less than a threshold value, controls the emission unit to change to an on state, and the left image acquisition unit and the right image acquisition unit are RGB/IR You can control the acquisition of images.

By emitting infrared light to the target, the RGB image and the IR image are simultaneously acquired, and the depth map is accurately extracted even in low light, thereby accurately performing 3D stereo matching.

In addition, while performing a normal operation under normal illumination, it detects it in low light and acquires an IR image in addition to the RGB image, or acquires only the IR image, and accurately extracts a depth map using this, thereby 3D stereo matching It has the effect of performing accurately.

1 is a block diagram of a stereo matching apparatus according to a first embodiment of the present invention.
2 is a block diagram of a stereo matching apparatus according to a second embodiment of the present invention.
3 is a block diagram of a stereo matching apparatus according to a third embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention.

Hereinafter, an embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a stereo matching apparatus according to a first embodiment of the present invention.

As shown in the drawing, the stereo matching apparatus according to an embodiment of the present invention includes a camera unit 1 and a stereo matching unit 2 , and a depth map of a stereo image extracted from the stereo matching unit 2 . This is provided to the encoder unit 3 . The encoder unit 3 codes, for example, a stereo image received in H.264, and for a technology related to image coding, it is already apparent to those of ordinary skill in the art to which the present invention pertains. Therefore, a detailed description thereof will be omitted. Hereinafter, in the description of an embodiment of the present invention, it is the same.

In the description of one embodiment of the present invention, the camera unit 1, the stereo matching unit 2, and the encoder unit 3 are illustrated as being composed of separate hardware, but the present invention is not limited thereto, and the same hardware (eg, For example, a camera, etc.) may be implemented as a separate chip, or may be implemented as a single chip.

The camera unit 1 according to an embodiment of the present invention includes a left image acquisition unit 11, an infrared radiation unit 12 and a right image acquisition unit 13, and includes a red, green, blue (target) A Red-Green-Blue; RGB) image and an Infrared (IR) image are acquired. In one embodiment of the present invention, the components of the camera unit 1 are not limited to being arranged in the manner shown in the drawings, it will be apparent to those of ordinary skill in the art to which the present invention pertains. will be.

The left image acquisition unit 11 and the right image acquisition unit 13 may include, for example, a camera sensor such as a Charge Coupled Device (CCD) or a Complementary Metal-Oxide Semiconductor (CMOS) sensor. As , the left and right RGB images and IR images are acquired, respectively. However, it does not mean that the left image acquisition unit 11 and the right image acquisition unit 13 of the present invention are limited to CCD or CMOS sensors, and the application of other devices performing similar functions is not excluded. is obvious to those of ordinary skill in the art to which the present invention pertains.

The infrared ray emitting unit 12 is composed of a single or a plurality of infrared elements, and emits infrared rays to the target. In one embodiment of the present invention, the infrared emitting unit 12 may be composed of an infrared element such as, for example, a laser diode (LD) or a light emitting diode (LED), but the infrared element is It is not limited thereto, and the application of any infrared device capable of radiating infrared light to the target is not excluded.

In one embodiment of the present invention, although the case in which one infrared emitting unit 12 is disposed is illustrated, the present invention is not limited thereto. That is, the more the number of infrared emitting units 12 is, the higher the output, which results in increasing the recognition range of infrared rays, so that even low illuminance and long distances can be recognized. number can be determined.

On the other hand, since the infrared emitting unit 12 composed of LD has straightness, the stereo matching device according to an embodiment of the present invention includes a hardware splitter (not shown) at the front end of the infrared emitting unit 12 . Not including), it is possible to spread the infrared rays emitted from the infrared radiation unit 12 in a predetermined pattern.

The left image acquisition unit 11 and the right image acquisition unit 13 according to the first embodiment of the present invention can simultaneously acquire not only the RGB image but also the IR image by the infrared ray emitted by the infrared ray emitting unit 12 . there is.

The stereo matching unit 2 of FIG. 1 includes a synchronization unit 21 , an alignment unit 22 , and a matching unit 23 .

The synchronization unit 21 receives the left image and the right image from the left image acquisition unit 11 and the right image acquisition unit 13 . The synchronization unit 21 synchronizes the received left and right images. Synchronization refers to aligning an image reception time equally. That is, the synchronizer 21 adjusts the received left and right images to be received at the same time.

The alignment unit 22 performs calibration so that the synchronized left and right images are positioned on the same epipolar line. In the alignment process of the alignment unit 22 , the encoder unit 3 receives calibration data from the left image acquisition unit 11 and the right image acquisition unit 13 of the camera unit 1 , and converts it into an alignment parameter. and delivered to the alignment unit 22 . The alignment unit 22 may perform calibration on the synchronized image received from the synchronizer 21 using the alignment parameter received from the encoder unit 3 .

The matching unit 23 extracts a depth map by performing stereo matching on the aligned left and right images.

Since the operations of the aligning unit 22 and the matching unit 23 are obvious to those of ordinary skill in the art to which the present invention pertains, a detailed description thereof will be omitted.

Hereinafter, the operation of the stereo matching apparatus according to the first embodiment of the present invention of FIG. 1 will be described.

In the stereo matching apparatus according to the first embodiment of the present invention, the infrared emitting unit 12 is always on, emitting infrared rays to the target, and the left image acquiring unit 11 and the right image acquiring unit 13 are Acquire RGB images and IR images, respectively.

Thereafter, the synchronization unit 21 synchronizes the left and right RGB and IR images, and outputs them to the alignment unit 22 . The alignment unit 22 aligns the left/right images, and the matching unit 23 performs 3D stereo matching by extracting the depth map.

As described above, according to the first embodiment of the present invention, the infrared emitting unit 12 emits infrared rays, and the left image acquisition unit 11 and the right image acquisition unit 13 acquire the RGB image and the IR image. By performing stereo matching using the , 3D stereo matching can be accurately performed by accurately extracting the depth map even in low light.

2 is a block diagram of a stereo matching apparatus according to a second embodiment of the present invention.

As shown in the drawing, the stereo matching apparatus according to the second embodiment of the present invention includes a camera unit 1 and a stereo matching unit 2, and the depth of the stereo image extracted from the stereo matching unit 2 is A map is provided to the encoder unit 3 .

The camera unit 1 according to the second embodiment of the present invention includes a left image acquisition unit 11 , an infrared emitting unit 12 , a light receiving unit 14 and a right image acquisition unit 13 , and a stereo matching unit 2 . includes a synchronization unit 21 , an alignment unit 22 , a matching unit 23 , a control unit 24 , and a switch unit 25 . The second embodiment of the present invention differs from the first embodiment in FIG. 1 in the configuration of the light receiving unit 14 of the camera unit 1 and the control unit 24 and the switch unit 25 of the stereo matching unit 2 . , differences from the first embodiment of FIG. 1 will be mainly described.

The left image acquisition unit 11 and the right image acquisition unit 13 of the camera unit 1 acquire left and right RGB images and/or IR images, respectively.

The infrared emitting unit 12 is turned off at normal illuminance, and when the light receiving unit 14 detects that the illuminance is equal to or less than a predetermined illuminance, it is changed to an ON state under the control of the controller 21 . This will be described in more detail later.

The light receiving unit 14 receives the light of the place where the camera unit 1 is disposed, and transmits the received light intensity to the control unit 24 of the stereo matching unit 2 to be described later. The light receiving unit 14 is, for example, a light receiving sensor or an illuminance sensor, but is not limited thereto.

The control unit 24 of the stereo matching unit 2 receives the illuminance from the light receiving unit 14, and controls the switch unit 22 to turn on the infrared emitting unit 12 when the received illuminance is equal to or less than a predetermined illuminance. . The switch unit 22 switches on/off of the infrared radiation unit 12 under the control of the control unit 21 .

The synchronization unit 21 according to the second embodiment of the present invention receives the RGB image from the left image acquisition unit 11 and the right image acquisition unit 13 when the illuminance is normal, and receives the RGB image when the illuminance is less than a predetermined illuminance. and an IR image, or only an IR image.

Hereinafter, the operation of the stereo matching apparatus according to the second embodiment of the present invention will be described.

In the normal illuminance state, that is, when the illuminance detected by the light receiving unit 14 is within the normal range, the stereo matching device according to the second embodiment of the present invention provides RGB acquired by the left image acquisition unit 11 and the right image acquisition unit 13. For the image, the synchronization unit 21 performs synchronization, the alignment unit 22 aligns the left/right images, and the matching unit 23 performs 3D stereo matching by extracting the depth map. do. At this time, the power of the infrared radiation unit 12 is in an off state.

However, when the illuminance of the light detected by the light receiving unit 14 is less than or equal to a predetermined illuminance, the control unit 24 controls the switch unit 25 to change the power of the infrared emitting unit 12 to an ON state.

When the infrared ray emitting unit 12 is turned on and irradiated with respect to the target, the left image acquisition unit 11 and the right image acquisition unit 13 acquire an RGB image and an IR image, or only an IR image, respectively. do.

Thereafter, the synchronizer 21 synchronizes the left and right RGB and IR images or IR images, the alignment unit 22 aligns the left/right images, and the matching unit 23 generates a depth map for this. 3D stereo matching is performed by extracting.

According to the second embodiment of the present invention, a normal operation is performed in a normal illuminance state, and when it is detected in low illuminance, an IR image is acquired in addition to the RGB image, or a depth map is accurately extracted using only the IR image, thereby 3D stereo matching can be performed accurately.

3 is a block diagram of a stereo matching apparatus according to a third embodiment of the present invention.

As shown in the drawing, the stereo matching apparatus according to the third embodiment of the present invention includes a camera unit 1 and a stereo matching unit 2, and the depth of the stereo image extracted from the stereo matching unit 2 is A map is provided to the encoder unit 3 .

The camera unit 1 according to the third embodiment of the present invention includes a left image acquisition unit 11 , an infrared radiation unit 12 and a right image acquisition unit 13 , and the stereo matching unit 2 includes a synchronization unit 21 . ), an alignment unit 22 , a matching unit 23 , a light intensity detection unit 26 and a switch unit 27 . The third embodiment of the present invention is different from the first embodiment of FIG. 1 in that the configuration of the light intensity detection unit 26 and the switch unit 27 of the stereo matching unit 2 is different from that of the first embodiment of FIG. The differences will be mainly explained.

The left image acquisition unit 11 and the right image acquisition unit 13 of the camera unit 1 acquire left and right RGB images and/or IR images, respectively.

The infrared emitting unit 12 is turned off at normal illuminance, and is turned on by switching of the switch unit 27 when the light intensity detecting unit 26 of the stereo matching unit 2 detects that the illuminance is less than or equal to a predetermined illuminance. ON) state.

The light intensity detection unit 26 of the stereo matching unit 2 receives the RGB image synchronized by the synchronizer 21, monitors the average of pixel values of the image, and when the average of the pixels is less than or equal to the threshold, low illuminance , and the switch unit 27 changes the infrared emitting unit 12 to an on state. The light intensity detection unit 26 is provided to the stereo matching unit 2, and may be provided in the form of software.

The switch unit 27 switches the on/off of the infrared radiation unit 12 under the control of the light intensity detection unit 26 .

The synchronizer 21 according to the third exemplary embodiment of the present invention receives an RGB image from the left image acquisition unit 11 and the right image acquisition unit 13 when the illuminance is normal, and receives the RGB image when the illuminance is less than a predetermined illuminance. and an IR image. In the third embodiment of the present invention, since the luminance detection unit 26 detects the average of luminance using pixels of the RGB image, the RGB image is received regardless of the illuminance.

A detailed operation of the stereo matching apparatus according to the third embodiment of the present invention will be described as follows.

In the normal illuminance state, that is, when the luminance detection unit 26 determines the normal illuminance from the average value of the pixels of the RGB image received from the synchronization unit 21, the stereo matching device according to the third embodiment of the present invention includes the left image acquisition unit 11 ) and the RGB image acquired by the right image acquisition unit 13, the synchronization unit 21 performs synchronization, the alignment unit 22 aligns the left/right images, and the matching unit 23 determines the depth 3D stereo matching is performed by extracting the map. At this time, the power of the infrared radiation unit 12 is in an off state.

However, when the light intensity detection unit 26 determines that the average value of the pixels of the RGB image received from the synchronization unit 21 is equal to or less than the threshold value, the light intensity detection unit 26 controls the switch unit 27 to control the infrared emission unit 12 . ) to turn on the power.

When the infrared ray emitting unit 12 is turned on and irradiated with respect to the target, the left image acquisition unit 11 and the right image acquisition unit 13 acquire an RGB image and an IR image, respectively.

Thereafter, the synchronizer 21 synchronizes the left and right RGB images and the IR image, the alignment unit 22 aligns the left/right images, and the matching unit 23 extracts the depth map. 3D stereo matching is performed by

According to the third embodiment of the present invention, normal operation is performed under normal illuminance, and when it is detected in low illuminance, an IR image is acquired in addition to the RGB image, the depth map is accurately extracted, thereby accurately performing 3D stereo matching. can be done

Although the embodiments according to the present invention have been described above, these are merely exemplary, and those of ordinary skill in the art will understand that various modifications and equivalent ranges of embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be defined by the following claims.

1: Camera unit 11: Left image acquisition unit
12: infrared radiation unit 13: right image acquisition unit
14: light receiving unit 2: stereo matching unit
21: synchronization unit 22: alignment unit
23: matching unit 24: control unit
25, 27: switch unit 26: light intensity detection unit
3: Encoder part

Claims (12)

a camera unit including an infrared radiation unit, a left image acquisition unit, and a right image acquisition unit; and
and a stereo matching unit for receiving an image from the left image acquisition unit and the right image acquisition unit,
The camera unit determines the illuminance by using the pixel values of the images received from the left image acquisition unit and the right image acquisition unit, and when it is determined that the illuminance is a low illuminance equal to or less than a predetermined illuminance, the infrared emitting unit is turned on. change it to
The left image acquisition unit and the right image acquisition unit acquire a left IR image and a right IR image,
The stereo matching unit stereo matching apparatus for extracting a depth map by stereo matching the left IR image and the right IR image.
According to claim 1,
The stereo matching unit,
and a synchronization unit for adjusting the left/right IR images received from the left image acquisition unit and the right image acquisition unit to be received at the same time.
3. The method of claim 2,
The stereo matching unit,
an alignment unit for correcting the synchronized left/right IR images to be positioned on the same epipolar line; and
and a matching unit configured to extract the depth map by performing matching on the aligned left/right IR images.
4. The method according to any one of claims 1 to 3,
The stereo matching unit,
and an encoder unit for receiving calibration-related data from the camera unit and converting it into an alignment parameter.
5. The method of claim 4,
The alignment unit,
A stereo matching device for correcting the left/right IR image by using the alignment parameter received from the encoder unit.
According to claim 1,
The stereo matching unit,
Stereo matching device including a switching unit for switching on/off the infrared emitting unit.
According to claim 1,
The infrared radiation unit,
A stereo matching device comprising at least one of a laser diode (LD) and a light emitting diode (LED).
According to claim 1,
The camera unit,
A stereo matching device comprising a splitter for spreading the infrared rays emitted from the infrared radiation unit in a predetermined pattern.
A camera module comprising the stereo matching device of claim 1.
A mobile terminal comprising the camera module of claim 9 .
According to claim 1,
The camera unit is a stereo matching device for determining the low illuminance when the average value of the pixels of the image received from the left image acquisition unit and the right image acquisition unit is less than or equal to a threshold value.
According to claim 1,
When it is determined by the camera unit as the normal illuminance rather than the low illuminance,
The stereo matching unit stereo matching apparatus for extracting a depth map by stereo matching the images received from the left image acquisition unit and the right image acquisition unit.

Images