KR102147120B1 - 3-dimensional image processing system - Google Patents

Abstract

A three-dimensional image processing system is disclosed. According to the 3D image processing system according to an embodiment of the present invention, by acquiring an RGB image and/or an IR image and using it to extract a depth map, stereo matching is performed accurately even in the case of low illumination, thereby accurately performing a 3D image. Images can be processed.

Description

3D image processing system {3-DIMENSIONAL IMAGE PROCESSING SYSTEM}

The present invention relates to a 3D image processing system.

As is well known, human vision is one of the senses to obtain information about the surrounding environment, and through both eyes, it is possible to perceive the position of an object and its proximity to each other. In other words, visual information coming through both eyes is synthesized into a single distance information and is freely active.

The stereo camera system is used when implementing this visual structure into a machine.

In a stereo camera system, stereo matching is performed on images obtained by using two cameras. In this stereo matching process, the stereo camera obtains a depth map using the binocular difference between the two cameras. This depth map data is used for recognition.

The technical problem to be solved by the present invention is to provide a 3D image processing system for acquiring an IR image and performing stereo matching, and outputting a 3D stereoscopic image in case of low illumination.

In order to solve the above technical problem, a 3D image processing system according to an embodiment of the present invention includes: a camera unit for acquiring left/right red, green, and blue (RGB) images and infrared (IR) images of a target; A stereo matching unit for performing stereo matching on left/right images received from the camera unit; And an image processing unit for synthesizing the left/right images received from the camera unit into a 3D image.

In an embodiment of the present invention, the camera unit includes: a left image acquisition unit for acquiring an RGB image and an IR image on the left side of the target; A right image acquisition unit that acquires a right RGB image and an IR image of the target; And at least one or more radiating units that emit infrared rays toward the target.

In an embodiment of the present invention, the stereo matching unit includes: a synchronization unit for adjusting the left/right images received from the left image acquisition unit and the right image acquisition unit to be received at the same time; An alignment unit correcting the synchronized left/right IR images to be positioned on the same epipolar line; And a matching unit for extracting a depth map by performing stereo matching on the aligned left/right IR images.

In an embodiment of the present invention, the stereo matching unit may further include an IR interpolation unit for interpolating the IR image received from the synchronization unit.

In an embodiment of the present invention, the image processing unit includes: a synthesis unit for synthesizing the RGB image received from the synchronization unit in a 3D format; And it may include a vergence control unit for controlling the vergence angle of the 3D format.

In an embodiment of the present invention, the image processing unit may further include an RGB interpolation unit that interpolates the RGB image received from the synchronization unit.

In addition, in order to solve the above technical problem, the 3D image processing system of an embodiment of the present invention includes a camera unit that acquires a left/right RGB image or an RGB image and an IR image (RGB/IR image) for a target. ; A stereo matching unit controlling the camera unit to acquire an RGB image or an RGB/IR image, and performing stereo matching on the left/right images received from the camera unit; And an image processing unit for synthesizing the left/right images received from the camera unit into a 3D image.

In an embodiment of the present invention, the camera unit includes: a first acquisition unit for acquiring an RGB image or an RGB/IR image on the left side of the target; A second acquisition unit that acquires an RGB image or an RGB/IR image on the right side of the target; A light receiving unit for detecting illuminance; And at least one or more first radiating units that emit infrared rays toward the target.

In an embodiment of the present invention, the stereo matching unit may include a control unit for controlling to change the first radiating unit to an on state when the illuminance detected by the light receiving unit is less than a predetermined illuminance; A first synchronization unit for adjusting the left/right images received from the first and second acquisition units to be received at the same time; A first alignment unit correcting the synchronized left/right RGB image or IR image to be positioned on the same epipolar line; And a first matching unit for extracting a depth map by performing stereo matching on the aligned left/right RGB images or IR images.

In an embodiment of the present invention, the stereo matching unit may further include a first interpolation unit interpolating the IR image received from the first synchronization unit.

In an embodiment of the present invention, the stereo matching unit may further include a first switching unit for switching on/off of the first radiating unit under the control of the control unit.

In an embodiment of the present invention, the image processing unit includes: a first synthesizing unit for synthesizing the RGB image received from the first synchronization unit into a 3D format; And a first visual angle control unit that controls the vergence angle of the 3D format.

In one embodiment of the present invention, the image processing unit may further include a second interpolation unit for interpolating the RGB image received from the first synchronization unit.

In an embodiment of the present invention, the control unit controls to change the first radiating unit to an ON state when the illuminance detected by the light receiving unit is less than a predetermined illuminance, and the first and second acquisition units generate an RGB/IR image. It can be controlled to acquire.

In an embodiment of the present invention, the camera unit includes: a third acquisition unit for acquiring an RGB image or an RGB/IR image on the left side of the target; A fourth acquisition unit that acquires an RGB image or an RGB/IR image on the right side of the target; And at least one or more second radiating units that emit infrared rays toward the target.

In an embodiment of the present invention, the stereo matching unit includes: a second synchronization unit for adjusting the left/right images received from the third and fourth acquisition units to be received at the same time; An alignment unit for correcting the synchronized left/right RGB image or IR image to be positioned on the same epipolar line; A matching unit for extracting a depth map by performing stereo matching on the aligned left/right RGB images or IR images; And when the average of the pixels of the left/right image is less than or equal to the threshold value, the second radiating unit may include a light intensity detection unit that controls to change to the on state.

In one embodiment of the present invention, the stereo matching unit may further include a third interpolation unit for interpolating the IR image received from the second synchronization unit.

In an embodiment of the present invention, the stereo matching unit may further include a second switching unit for switching on/off of the second radiating unit under the control of the light intensity detection unit.

In an embodiment of the present invention, the image processing unit includes: a second synthesizing unit for synthesizing the RGB image received from the second synchronization unit into a 3D format; And a second visual angle control unit that controls the visual angle of the 3D format.

In an embodiment of the present invention, the image processing unit may further include a fourth interpolation unit for interpolating the RGB image received from the second synchronization unit.

In one embodiment of the present invention, when the average of the pixels of the left/right image is less than or equal to a threshold value, the light intensity detection unit controls to change the second radiating unit to an on state, and the third and fourth acquisition units are RGB/ It can be controlled to acquire IR images.

By radiating infrared rays to a target, an RGB image and an IR image are simultaneously acquired, and a depth map is accurately extracted even in the case of low illumination, thereby accurately performing 3D stereo matching.

In addition, while performing a normal operation in a normal illumination state, it detects it in low-light conditions and acquires an IR image in addition to an RGB image, or acquires only an IR image, and accurately extracts a depth map using this, and thereby 3D stereo matching. It has the effect of performing correctly.

1 is a block diagram of a stereo matching device according to a first embodiment of the present invention.
2 is a block diagram of a stereo matching device according to a second embodiment of the present invention.
3 is a block diagram of a stereo matching device according to a third embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail in the detailed description. However, this is not intended to limit the present invention to a specific embodiment, it is to be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention.

Hereinafter, an embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a configuration diagram of a 3D image processing system according to a first embodiment of the present invention.

As shown in the drawing, a 3D image processing system according to an embodiment of the present invention includes a camera unit 1, a stereo matching unit 2 and an image processing unit 4, and a stereo matching unit 2 The depth map of the stereo image extracted from and the 3D image output from the image processing unit 4 are provided to the encoder unit 3. The encoder unit 3, for example, is for coding a stereo image received in H.264, and it is apparent to those of ordinary skill in the art to which the present invention belongs to the technology related to image coding. Therefore, a detailed description thereof will be omitted. In the following description of an embodiment of the present invention, it is the same.

In the description of an embodiment of the present invention, the camera unit 1, the stereo matching unit 2, the encoder unit 3 and the image processing unit 4 are illustrated as being configured as separate hardware, but are limited thereto. No, it may be implemented as a separate chip within the same hardware (eg, a camera, etc.), or may be implemented as a single chip.

The camera unit 1 according to an embodiment of the present invention includes a left image acquisition unit 11, an infrared radiation unit 12, and a right image acquisition unit 13, and red, green, and blue for a target ( Red-Green-Blue (RGB) images and Infrared (IR) images are acquired. In one embodiment of the present invention, the components of the camera unit 1 are not limited to being aligned in the manner shown in the drawings, it is obvious to those of ordinary skill in the art to which the present invention belongs. will be.

The left image acquisition unit 11 and the right image acquisition unit 13 are, for example, camera sensors such as a charge coupled device (CCD) and a complementary metal-oxide semiconductor (CMOS) sensor. As, the left and right RGB images and IR images are obtained, respectively. However, it does not mean that the left image acquisition unit 11 and the right image acquisition unit 13 of the present invention are limited to a CCD or CMOS sensor, and it does not exclude the application of other devices that perform similar functions. Is obvious to those of ordinary skill in the art to which the present invention pertains.

The infrared radiation unit 12 is composed of a single or a plurality of infrared elements, and emits infrared rays to the target. In one embodiment of the present invention, the infrared radiation unit 12 may be composed of an infrared element such as, for example, a laser diode (LD) or a light emitting diode (LED), but the infrared element is It is not limited thereto, and does not exclude the application of any infrared element capable of emitting infrared rays to the target.

In one embodiment of the present invention, a case in which one infrared radiating unit 12 is disposed is illustrated, but the present invention is not limited thereto. That is, as the number of infrared radiation units 12 increases, the output increases, which results in an increase in the recognition range of infrared rays, so that even low illumination and long distances can be recognized, and according to the appropriate recognition distance, the output of the infrared radiation unit 12 is increased. You can decide the number.

On the other hand, since the infrared radiating unit 12 composed of LD has a straightness, the 3D image processing system according to an embodiment of the present invention has a hardware splitter (splitter) at the front end of the infrared radiating unit 12 ( (Not shown) may be further included to spread the infrared rays emitted from the infrared radiation unit 12 in a predetermined pattern.

The left image acquisition unit 11 and the right image acquisition unit 13 according to the first embodiment of the present invention can simultaneously acquire not only an RGB image, but also an IR image by infrared rays emitted by the infrared radiation unit 12. have.

The stereo matching unit 2 of FIG. 1 includes a synchronization unit 21, an IR interpolation unit 28, an alignment unit 22, and a matching unit 23.

The synchronization unit 21 receives a left image and a right image from the left image acquisition unit 11 and the right image acquisition unit 13. The synchronization unit 21 synchronizes the received left and right images. Synchronization refers to the same alignment of the time at which the video is received. That is, the synchronization unit 21 adjusts the received left and right images to be received at the same time.

In addition, the synchronization unit 21 transmits each of the synchronized left and right images, the IR image to the IR interpolation unit 28, and the RGB image to the RGB interpolation unit 41 of the image processing unit 4, respectively. Interpolation can be performed by the executives 28 and 41, and the distribution unit includes a distribution unit (not shown) at the rear end of the synchronization unit 21, and the distribution unit distributes the corresponding RGB image and the IR image to each interpolation unit ( 41, 28).

The RGB interpolation unit 41 receives the RGB image from the synchronization unit 21 and interpolates it, and the IR interpolation unit 44 receives the IR image from the synchronization unit 23 and interpolates it. Interpolation is generally averaging surrounding pixels to create new pixels, and there are numerous interpolation methods, and as they are well known in the art to which the present invention pertains, a detailed description thereof will be omitted.

In this case, in the first embodiment of the present invention, the synchronization unit 21 transmits the IR image to the IR interpolation unit 28, so that the IR interpolation unit 28 interpolates the IR image, and further, the RGB image 41), and the RGB interpolation unit 41 interpolates the RGB image, but when the left/right image acquisition units 11 and 13 perform interpolation, a separate RGB interpolation unit 41 and IR interpolation The executive unit 28 may not be included, and in this case, the image interpolated by the left/right image acquisition units 11 and 13 is transmitted to the synchronization unit 21, and the synchronization unit 21 is a combination unit ( 42) and the alignment unit 22 will be able to directly transmit the interpolated RGB image and IR image.

The alignment unit 22 performs calibration so that the input left/right IR images are located on the same epipolarline. In the alignment process of the alignment unit 22, the encoder unit 3 receives calibration data from the left image acquisition unit 11 and the right image acquisition unit 13 of the camera unit 1, and converts it into alignment parameters. And transfer it to the alignment unit 22. The alignment unit 22 may calibrate the input image by using the alignment parameter received from the encoder unit 3.

The matching unit 23 performs stereo matching on the aligned left/right IR images, and extracts a depth map.

Since the operation of the alignment unit 22 and the matching unit 23 is obvious to those of ordinary skill in the art to which the present invention pertains, a detailed description thereof will be omitted.

Meanwhile, the image processing unit 4 according to the first embodiment of FIG. 1 includes an RGB interpolation unit 41, a synthesis unit 42, and a vergence control unit 43.

The function of the RGB interpolation unit 41 is as described above.

The synthesizing unit 42 synthesizes the left/right RGB images in a 3D format. It is widely known that the synthesis method includes a side by side method and a top and bottom method. The vergence control unit 43 controls the vergence angle of the 3D format image synthesized by the synthesis unit 42 so that the focus of the 3D image is not shaken and a three-dimensional effect is realized. Since the synthesis unit 42 and the vergence control unit 43 generate a 3D image as well known, a more detailed description will be omitted.

Hereinafter, the operation of the 3D image processing system according to the first embodiment of the present invention of FIG. 1 will be described.

In the 3D image processing system according to the first embodiment of the present invention, the infrared radiation unit 12 is always on, radiates infrared rays to the target, and the left image acquisition unit 11 and the right image acquisition unit 13 ) Acquire an RGB image and an IR image, respectively.

Thereafter, the synchronization unit 21 synchronizes the left and right RGB and IR images, and transmits the IR image to the IR interpolation unit 28 and the RGB image to the RGB interpolation unit 41. To this end, it is already described that the distribution unit for distributing the IR image and the RGB image may be disposed at the rear end of the synchronization unit 21.

The IR interpolation unit 28 interpolates the transmitted IR image and outputs it to the alignment unit 22. The alignment unit 22 aligns the left/right image, which is an infrared image, and the matching unit 23 extracts a depth map to perform 3D stereo matching.

Meanwhile, the RGB image transmitted from the synchronization unit 21 to the RGB interpolation unit 41 is interpolated by the RGB interpolation unit 41, and the interpolated image is provided to the synthesis unit 42 and converted into a 3D format, The vergence control unit 43 performs vergence control to output a 3D stereoscopic image.

This output is provided to the encoder section 3 together with the depth map which is the output of the matching section 23.

However, in the description of the first embodiment of the present invention, an IR image is provided to the stereo matching unit 2 to extract a depth map, but an RGB image and an IR image are simultaneously provided to extract the depth map. May be.

According to the first embodiment of the present invention, the infrared radiation unit 12 emits infrared rays, and the left image acquisition unit 11 and the right image acquisition unit 13 acquire an RGB image and an IR image and use the stereo matching. And by performing the image processing, it is possible to accurately extract the depth map even in the case of low illumination, thereby accurately performing 3D image processing.

2 is a block diagram of a 3D image processing system according to a second embodiment of the present invention.

As shown in the drawing, the 3D image processing system according to the second embodiment of the present invention includes a camera unit 1, a stereo matching unit 2, and an image processing unit 4, and a stereo matching unit 2 The depth map of the stereo image extracted from) and the 3D image output from the image processing unit 4 are provided to the encoder unit 3.

The camera unit 1 of the second embodiment of the present invention includes a left image acquisition unit 11, an infrared radiation unit 12, a light receiving unit 14, and a right image acquisition unit 13. The stereo matching unit 2 includes a synchronization unit 21, an IR interpolation unit 28, a multiplexer (MUX) 29, an alignment unit 22, a matching unit 23, a control unit 24, and a switch unit 25. Includes. In addition, the image processing unit 4 includes an RGB interpolation unit 41, a synthesis unit 42, and a vergence control unit 43.

The second embodiment of the present invention is the first embodiment of Fig. 1 and the control unit 24 and the switch unit 25 and the MUX 29 of the light receiving unit 14 and the stereo matching unit 2 of the camera unit 1 As the configuration of is different, the difference from the first embodiment of FIG. 1 will be mainly described.

The left image acquisition unit 11 and the right image acquisition unit 13 of the camera unit 1 acquire left and right RGB images and/or IR images, respectively.

The infrared radiation unit 12 is turned off at normal illuminance, and when the light receiving unit 14 detects that the illuminance is less than or equal to the predetermined illuminance, it is changed to an ON state under the control of the controller 21.

The light receiving unit 14 receives light from a place where the camera unit 1 is disposed, and transmits the received light illumination to the control unit 24 of the stereo matching unit 2 to be described later. The light receiving part 14 is, for example, a light receiving sensor or an illuminance sensor, but is not limited thereto.

The control unit 24 of the stereo matching unit 2 receives the illuminance from the light receiving unit 14, and controls the switch unit 22 to turn on the infrared radiation unit 12 when the received illuminance is less than a certain illuminance. . The switch unit 22 switches on/off of the infrared radiation unit 12 under the control of the control unit 21.

The synchronization unit 21 according to the second embodiment of the present invention receives an RGB image from the left image acquisition unit 11 and the right image acquisition unit 13 in the case of normal illuminance, and the RGB image in the case of a predetermined illuminance or less. And an IR image.

Therefore, the synchronization unit 21 performs synchronization to equally adjust the time when the left and right images are received, and in the case of normal illumination (when only the RGB image is received), the RGB image is transferred to the RGB interpolation unit 41. When the image is transmitted and less than a predetermined illuminance (when an RGB image and an IR image are received), the RGB image is transmitted to the RGB interpolation unit 41, and the IR image is transmitted to the IR interpolation unit 28. However, in the description of the second embodiment of the present invention, it has been described that the IR image and the RGB image are separated and the depth map is extracted and 3D image processing is performed, but the depth map is extracted by simultaneously providing the IR image and the RGB image. May be.

In addition, when interpolating the images acquired by the left image acquisition unit 11 and the right image acquisition unit 13 and providing them to the synchronization unit 21, the IR interpolation unit 28 and the RGB interpolation unit 41 Is not included, and in this case, the left/right images synchronized by the synchronization unit 21 may be directly provided to the synthesis unit 42 or the MUX 29. That is, in the case of normal illumination, the RGB image synchronized by the synchronization unit 21 is provided to the MUX 29 to perform a depth map extraction procedure, and is also provided to the synthesis unit 42 to perform image processing. . In the case of less than a predetermined illuminance, the synchronization unit 21 performs synchronization, the RGB image is provided to the synthesis unit 42, and the IR image is provided to the MUX 29 to perform image processing and depth map extraction, respectively. have.

Hereinafter, the operation of the 3D image processing system according to the second embodiment of the present invention will be described.

In the normal illuminance state, that is, when the illuminance detected by the light receiving unit 14 is within the normal range, the 3D image processing system of the second embodiment of the present invention includes the left image acquisition unit 11 and the right image acquisition unit 13 The synchronization unit 21 performs synchronization with respect to the acquired RGB image, and the RGB interpolation unit 41 performs RGB interpolation on the synchronized left and right images.

Among them, black and white data (eg, luminance (Y) data) among RGB images is provided from the RGB interpolation unit 41 to the MUX 29. For this black and white data, the alignment unit 22 provides the left/right image Is aligned, and the matching unit 23 performs 3D stereo matching by extracting a depth map. Since only black and white images are used in depth map extraction, stereo matching can be performed by this method.

In addition, the image interpolated by the RGB interpolation unit 41 is provided to the synthesis unit 42 and converted into a 3D format, and the vergence control unit 43 performs vergence control to output a 3D stereoscopic image. . This output is provided to the encoder section 3 together with the depth map which is the output of the matching section 23. At this time, the power of the infrared radiation unit 12 is turned off.

However, when the illuminance of light detected by the light receiving unit 14 is less than or equal to a predetermined illuminance, the control unit 24 controls the switch unit 25 to change the power of the infrared radiation unit 12 to the ON state.

When the infrared radiation unit 12 is turned on and radiates infrared rays to the target, the left image acquisition unit 11 and the right image acquisition unit 13 acquire an RGB image and an IR image, respectively.

Thereafter, the synchronization unit 21 synchronizes the left and right RGB images and the IR images, and transmits the IR image to the IR interpolation unit 28 and the RGB image to the RGB interpolation unit 41. To this end, it has already been described that the distribution unit for distributing the infrared image and the RGB image may be disposed at the rear end of the synchronization unit 21.

The IR interpolation unit 28 interpolates the transmitted IR image and outputs it to the MUX 29. The input of the MUX 29 is, in this case, an interpolated IR image, and the MUX 29 outputs it to the alignment unit 22. The alignment unit 22 may align the left/right images, which are IR images, and the matching unit 23 may receive them and extract the depth map.

In addition, the RGB image transmitted from the synchronization unit 21 to the RGB interpolation unit 41 is interpolated by the RGB interpolation unit 41, and the interpolated image is provided to the synthesis unit 42 and converted into a 3D format, The vergence control unit 43 performs vergence control to output a 3D stereoscopic image.

This output is provided to the encoder section 3 together with the depth map which is the output of the matching section 23.

According to the second embodiment of the present invention, a normal operation is performed in a normal illumination state, and then an IR image is acquired in addition to an RGB image, while performing a normal operation in a low-light condition, thereby accurately extracting a depth map, thereby accurately performing 3D image processing. Can be done.

3 is a block diagram of a 3D image processing system according to a third embodiment of the present invention.

As shown in the drawing, the 3D image processing system according to the second embodiment of the present invention includes a camera unit 1, a stereo matching unit 2, and an image processing unit 4, and a stereo matching unit 2 The depth map of the stereo image extracted from) and the 3D image output from the image processing unit 4 are provided to the encoder unit 3.

The camera unit 1 of the third embodiment of the present invention includes a left image acquisition unit 11, an infrared radiation unit 12 and a right image acquisition unit 13, and the stereo matching unit 2 is a synchronization unit 21 ), an IR interpolation unit 28, a MUX 29, an alignment unit 22, a matching unit 23, a light intensity detection unit 26, and a switch unit 27. In the third embodiment of the present invention, the configuration of the light intensity detection unit 26, the switch unit 27, and the MUX 29 of the stereo matching unit 2 from the first embodiment of Fig. 1 is different. The differences from the first embodiment will be mainly described. Further, descriptions overlapping with those of the second embodiment will be omitted.

The left image acquisition unit 11 and the right image acquisition unit 13 of the camera unit 1 acquire left and right RGB images and/or IR images, respectively.

When the infrared radiation unit 12 is turned off at normal illuminance, and when the light intensity detecting unit 26 of the stereo matching unit 2 detects that it is less than or equal to a predetermined illuminance, it is turned on by switching of the switch unit 27 ( ON).

The light intensity detection unit 26 of the stereo matching unit 2 receives the RGB image synchronized by the synchronization unit 21, monitors the average of the pixel values of the image, and when the average of the pixels is less than the threshold value, low illumination Is determined, and the switch unit 27 changes the infrared radiation unit 12 to the ON state. The light intensity detection unit 26 is provided to the stereo matching unit 2 and may be provided in a software format.

The switch unit 27 switches on/off of the infrared radiation unit 12 under the control of the light intensity detection unit 26.

The synchronization unit 21 according to the third embodiment of the present invention receives an RGB image from the left image acquisition unit 11 and the right image acquisition unit 13 in the case of normal illuminance, and the RGB image in the case of a predetermined illuminance or less. And IR images.

A detailed operation of the 3D image processing system according to the third embodiment of the present invention will be described as follows.

In the normal illuminance state, that is, when the luminous intensity detector 26 determines the normal illuminance from the average value of the pixels, the 3D image processing system of the third embodiment of the present invention includes the left image acquisition unit 11 and the right image acquisition unit 13 The synchronization unit 21 performs synchronization with respect to the RGB image acquired by A, and the RGB interpolation unit 41 performs RGB interpolation with respect to the synchronized left and right images.

Among them, black and white data (eg, luminance (Y) data) among RGB images is provided from the RGB interpolation unit 41 to the MUX 29. For this black and white data, the alignment unit 22 provides the left/right image Are aligned, and the matching unit 23 performs stereo matching by extracting a depth map. Since only black and white images are used in depth map extraction, stereo matching can be performed by this method.

In addition, the RGB image interpolated by the RGB interpolation unit 41 is provided to the synthesis unit 42 and converted into a 3D format, and the vergence control unit 43 performs vergence control to output a 3D stereoscopic image. do. This output is provided to the encoder section 3 together with the depth map which is the output of the matching section 23. At this time, the power of the infrared radiation unit 12 is turned off.

On the other hand, when the light intensity detection unit 26 determines that the average value of the pixels is less than or equal to the predetermined illuminance, the light intensity detection unit 26 controls the switch unit 27 to change the power of the infrared radiation unit 12 to the ON state.

When the infrared radiation unit 12 is turned on and radiates infrared rays to the target, the left image acquisition unit 11 and the right image acquisition unit 13 acquire an RGB image and an IR image, respectively.

Thereafter, the synchronization unit 21 synchronizes the left and right RGB images and the IR images, and transmits the IR image to the IR interpolation unit 28 and the RGB image to the RGB interpolation unit 41. To this end, it has already been described that the distribution unit for distributing the infrared image and the RGB image may be disposed at the rear end of the synchronization unit 21.

The IR interpolation unit 28 interpolates the transmitted IR image and outputs it to the MUX 29. The input of the MUX 29 is, in this case, an interpolated IR image, and the MUX 29 outputs it to the alignment unit 22. The alignment unit 22 may align the left/right images, which are IR images, and the matching unit 23 may receive them and extract the depth map.

In addition, the RGB image transmitted from the synchronization unit 21 to the RGB interpolation unit 41 is interpolated by the RGB interpolation unit 41, and the interpolated image is provided to the synthesis unit 42 and converted into a 3D format, The vergence control unit 43 performs vergence control to output a 3D stereoscopic image.

This output is provided to the encoder section 3 together with the depth map which is the output of the matching section 23.

According to the third embodiment of the present invention, while performing a normal operation in a normal illumination state, detecting it in a low-light condition, acquiring an IR image in addition to an RGB image, and accurately extracting a depth map, thereby accurately performing 3D image processing. Can be done.

In the above second and third embodiments, the MUX 29 receives an image from one of the RGB interpolation unit 41 and the IR interpolation unit 28 and transmits the image to the alignment unit 22. In this case, when an image to be transmitted to the alignment unit 22 is determined in a step prior to the MUX 29, the image may be directly transmitted to the alignment unit 22.

In addition, in the embodiment of the present invention, in the case of outputting an image interpolated by the left/right image acquisition units 11 and 13, both the IR interpolation unit 28 and the RGB interpolation unit ( 41) may not be included as described above.

Although the embodiments according to the present invention have been described above, these are merely exemplary, and those of ordinary skill in the art will understand that various modifications and equivalent ranges of embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention should be determined by the following claims.

1: camera unit 11: left image acquisition unit
12: infrared radiation unit 13: right image acquisition unit
14: light receiving unit 2: stereo matching unit
21: synchronization unit 22: alignment unit
23: matching unit 24: control unit
25, 27: switch unit 26: light intensity detection unit
3: encoder unit

Claims (17)

A left image acquisition unit that acquires an RGB image and an IR image on the left side of the target;
A right image acquisition unit for acquiring a right RGB image and an IR image for the target; And
A camera unit including at least one radiating unit that emits infrared rays toward the target;
A stereo matching unit for synchronizing the image received from the left image acquisition unit and the image received from the right image acquisition unit, and then performing stereo matching based on the synchronized images to output a depth map;
An image processing unit which receives the synchronized images from the stereo matching unit and synthesizes them into a 3D image; And
And an encoder that performs coding based on a depth map output from the stereo matching unit and a 3D image output from the image processing unit,
The camera unit includes a light receiving unit that detects the illuminance of the RGB image,
The stereo matching unit,
3 extracting the depth map using the RGB image and the IR image when the illuminance is less than a predetermined illuminance, and extracting the depth map using black and white data from the RGB image when the illuminance exceeds the predetermined illuminance Dimensional image processing system.
The method of claim 1,
The stereo matching unit,
3D image processing system comprising a synchronization unit for adjusting the left / right image received from the left image acquisition unit and the right image acquisition unit to be received at the same time.
The method of claim 2,
The stereo matching unit,
3D image processing system comprising; an alignment unit for correcting the synchronized left and right IR images to be positioned on the same epipolar line.
The method of claim 2,
The stereo matching unit,
3D image processing system comprising an IR interpolation unit for interpolating the IR image received from the synchronization unit.
The method of claim 2,
The image processing unit,
A synthesizing unit for synthesizing the RGB image received from the synchronization unit in a 3D format; And
A 3D image processing system including a vergence control unit for controlling the vergence angle of a 3D format.
The method of claim 2,
The image processing unit,
3D image processing system further comprising an RGB interpolation unit for interpolating the RGB image received from the synchronization unit.
delete The method of claim 1,
The stereo matching unit,
And a control unit configured to control the camera unit to acquire the RGB image and the IR image by changing the radiating unit to an on state when the illuminance is less than a predetermined illuminance.
delete delete delete delete The method of claim 1,
The radiating part,
3D image processing system including a splitter (splitter) to emit the infrared rays in a predetermined pattern.
A camera module comprising the 3D image processing system of claim 1.
A mobile terminal comprising the camera module of claim 14.




A left image acquisition unit that acquires an RGB image and an IR image on the left side of the target;
A right image acquisition unit for acquiring a right RGB image and an IR image for the target; And
A camera unit including at least one radiating unit that emits infrared rays toward the target;
A stereo matching unit for synchronizing the image received from the left image acquisition unit and the image received from the right image acquisition unit, and then performing stereo matching based on the synchronized images to output a depth map;
An image processing unit which receives the synchronized images from the stereo matching unit and synthesizes them into a 3D image; And
And an encoder for performing coding based on a depth map output from the stereo matching unit and a 3D image output from the image processing unit; and
The stereo matching unit includes a light intensity detection unit detecting pixel values of the synchronized left/right image and a radiating unit radiating infrared rays to the target,
When the average of the pixel values detected by the light intensity detector is less than or equal to the threshold value, the depth map is extracted using the RGB image and the IR image, and the average of the pixel values detected by the light intensity detector exceeds the threshold value. And 3D image processing system for extracting the depth map using black and white data from the RGB image.
The method of claim 16,
The stereo matching unit,
3D image processing system for controlling the camera to acquire the RGB image and the IR image by changing the radiating unit to an on state when the average of the pixel values of the left and right images is less than or equal to a threshold value.

Images