KR101823256B1 - Image processing apparatus and method for image processing - Google Patents

Abstract

An image processing apparatus according to an embodiment of the present invention may include a first image sensor for obtaining a plurality of single color images having different exposure values; a second image sensor for obtaining a plurality of multicolor images having different exposure values; a first synthesis part for performing a high dynamic range (HDR) synthesis operation on the plurality of single color images; a second synthesis part for performing a high dynamic range synthesis operation on the plurality of multicolor images; and a third synthesis part for synthesizing an image synthesized by the first synthesis part and an image synthesized by the second synthesis part. It is possible to reduce image quality degradation.

Description

TECHNICAL FIELD The present invention relates to an image processing apparatus and an image processing method,

The present invention relates to an image processing apparatus and an image processing method.

In order to correct the backlight of an image in general, a plurality of images having different exposure values are combined to obtain an image with an increased dynamic range, which can be defined as high dynamic range (HDR) synthesis.

Japanese Unexamined Patent Application Publication No. 2014-023062

An embodiment of the present invention provides an image processing apparatus and an image processing method capable of reducing image deterioration occurring in a process of enlarging a dynamic range.

An image processing apparatus according to an embodiment of the present invention includes a first image sensor for obtaining a plurality of single color images having different exposure values; A second image sensor for obtaining a plurality of multi-color images having different exposure values; A first combining unit for performing a high dynamic range (HDR) combining operation on the plurality of single color images; A second combining unit for performing a high dynamic range combining operation on the plurality of multi-color images; A third synthesizing unit synthesizing an image synthesized by the first synthesizing unit and an image synthesized by the second synthesizing unit; And the third synthesizing unit may apply the luminance information of the region corresponding to the region in the image synthesized by the first synthesizing unit to the region where saturation has occurred in the image synthesized by the second synthesizing unit have.

An image processing method according to an embodiment of the present invention includes: obtaining a plurality of single color images having different exposure values; Obtaining a plurality of multi-color images having different exposure values; Obtaining a single color image having an extended dynamic range from the plurality of single color images; Obtaining a multi-color image having an extended dynamic range from the plurality of multi-color images; Synthesizing a single color image having the dynamic range expanded and a multi-color image having the dynamic range expanded; And the combining step may apply the luminance information of the region corresponding to the region in the single color image in which the dynamic range is extended to the region in which saturation has occurred in the multi-color image in which the dynamic range is extended.

The image processing apparatus according to an exemplary embodiment of the present invention can reduce image deterioration occurring in the process of enlarging the dynamic range.

1 is a diagram illustrating an image processing apparatus according to an embodiment of the present invention.
2 is a diagram illustrating an image processing apparatus according to an embodiment of the present invention.
3 is a view illustrating an image sensor and a synthesizer included in an image processing apparatus according to an embodiment of the present invention.
4 is a flowchart illustrating an image processing method according to an embodiment of the present invention.
5 is a diagram illustrating an exemplary computing environment in which one or more embodiments disclosed herein may be implemented.

The following detailed description of the invention refers to the accompanying drawings, which illustrate, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different, but need not be mutually exclusive. For example, certain features, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with an embodiment. It is also to be understood that the position or arrangement of the individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is to be limited only by the appended claims, along with the full scope of equivalents to which such claims are entitled. In the drawings, like reference numerals refer to the same or similar functions throughout the several views.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily carry out the present invention.

1 is a diagram illustrating an image processing apparatus according to an embodiment of the present invention.

1, an image processing apparatus according to an exemplary embodiment of the present invention includes a first image sensor 110, a second image sensor 120, a first combining unit 130, a second combining unit 140, And may include a third combining unit 150.

The first and second image sensors 110 and 120 can obtain a plurality of images having different exposure values. Here, the exposure value of the image may be defined as an image sensor exposure value at the time of obtaining the image. The larger the exposure value of the image sensor is, the larger the amount of light entering the image sensor is, and thus the image with a large exposure value may be brighter than the image with a small exposure value.

For example, the first and second image sensors 110 and 120 may repeatedly obtain a plurality of images having different exposure values at predetermined intervals. If the plurality of images have exposure values of -2, 0, and 2, respectively, the first and second image sensors 110 and 120 generate three Images can be obtained.

The first image sensor 110 may obtain a plurality of single color images having different exposure values. Here, a single color image is an image having luminance information per pixel and no color information per pixel.

For example, the first image sensor 110 senses light of first, second, and third colors for each pixel, adds the luminance information of the first, second, and third colors to each pixel, A single color image can be obtained.

That is, the first image sensor 110 can detect the color of the light without discriminating the color of the light. For example, the first image sensor 110 may include a pixel array having a wide light receiving frequency range to improve light absorption rate. Therefore, the first image sensor 110 may have high sensitivity characteristics.

The second image sensor 120 may obtain a plurality of multi-color images having different exposure values. Here, the multi-color image is an image having color information per pixel.

For example, the second image sensor 120 may sense light of the first, second, and third colors for each pixel, and may detect the light of the plurality (e.g., The color of the multi-color image of the image can be determined.

That is, the second image sensor 120 includes a plurality of pixel arrays each sensing light of first, second, and third colors, respectively, to independently detect light of the first, second, and third colors . For example, the pixel array may have different light-receiving frequency ranges to detect lights of different colors, respectively. The color information of the multi-color image can be calculated based on the detection result difference between the pixel arrays.

On the other hand, the first color may be red, the second color may be blue, and the third color may be white or green. The multicolor image is an RWB-based image when the third color is white, and an RGB-based image when the third color is green.

The first combining unit 130 may perform a high dynamic range (HDR) combining operation on a plurality of single color images. For example, the first synthesizer 130 obtains an image in which a dynamic range is enlarged by obtaining an image in which a bright region of an image having a large exposure value and a dark region of an image having a small exposure value are combined. Accordingly, the first combining unit 130 can obtain a single color image in which the backlight is corrected.

The second combining unit 140 may perform a high dynamic range combining operation on a plurality of multicolor images. For example, the second combining unit 140 may perform a combining operation using the same principle as the combining principle of the first combining unit 130. Accordingly, the second combining unit 140 can obtain a multi-color image in which the backlight is corrected.

The third combining unit 150 may combine the image synthesized by the first synthesizing unit 130 and the image synthesized by the second synthesizing unit 140. That is, the third combining unit 150 can utilize both the high sensitivity characteristic of a single color image and the color information of a multi-color image. Accordingly, the image processing apparatus according to an embodiment of the present invention can reduce the occurrence of saturation in the process of enlarging the dynamic range, thereby reducing image deterioration.

For example, the third combining unit 150 may generate an image obtained by adding color information of an image synthesized by the second combining unit 140 to an image synthesized by the first synthesizing unit 130 . If the multi-color image is an RWB-based image, the third combining unit 150 adds the luminance information of the first and second colors of the multi-color image to the single color image, 1 and the luminance information of the second color.

For example, the third combining unit 150 may calculate the luminance information of the corresponding region of the image synthesized by the first combining unit 130 in the area where saturation has occurred in the image synthesized by the second combining unit 140, Can be applied.

2 is a diagram illustrating an image processing apparatus according to an embodiment of the present invention.

2, an image processing apparatus according to an exemplary embodiment of the present invention includes a first image sensor 210, a first image processing unit 215, a second image sensor 220, a second image processing unit 225, A first synthesizer 230, a second synthesizer 240, and a third synthesizer 250.

The first image sensor 210 may obtain a first single color image 211 having a positive first exposure value and a second single color image 212 having a negative exposure value of the first exposure value.

The first image processing unit 215 can match and correct the first single color image 211 and the second single color image 212 with each other.

The second image sensor 220 may obtain a first multi-color image 221 having a second positive exposure value and a second multi-color image 222 having a negative exposure value of the second exposure value.

The second image processing unit 225 may match and correct the first multi-color image 221 and the second multi-color image 222 with each other.

Depending on the design, the first and second image sensors 210 and 220 may further obtain a single color image having an exposure value of 0 and a multi-color image having an exposure value of 0, respectively.

The first and second image processing units 215 and 225 can perform a more accurate correction operation as the number of exposure values of the image obtained by the first and second image sensors 210 and 220 increases, 210 and 220, the faster the matching and the correcting operation can be performed.

The first and second composing units 230 and 240 may control exposure values of the first and second image sensors 210 and 220, respectively.

On the other hand, an exposure value of +1 has a shutter speed value of 0.5, an exposure value of 0 has a shutter speed value of 1, and an exposure value of -1 has a shutter speed value of 2, but is not limited thereto. Also, an exposure value of +1, an exposure value of 0, and an exposure value of -1 may all have an aperture value of 1 and an ISO value of 100, but are not limited thereto.

3 is a view illustrating an image sensor and a combining unit included in an image processing apparatus according to an embodiment of the present invention.

3, the image sensor 300 may include a pixel array 310, a digital block 320 and a register 330, and the combining unit 400 may include a demosaic 410, an 3A engine 420 A color processor 430, a YUV processor 440, an HDR control unit 450, an image compositing unit 460, a video encoder 470, and a Jpeg encoder 480.

The image sensor 300 can perform the same operations as the first and second image sensors described above with reference to Figs.

The pixel array 310 can sense light on a pixel-by-pixel basis.

The digital block 320 includes a digital process 321 operation that computes luminance information and / or color information based on the detection result of the pixel array 310, an output interface 322 operation that generates an image based on the calculation result Can be performed.

The register 330 receives the exposure value control signal from the combining unit 400 and can adjust the opening / closing speed and / or the analog gain of the shutter included in the image sensor 300.

The combining unit 400 may perform the same operations as the first through third combining units described above with reference to FIGS.

The demosaic 410 may provide an arithmetic environment for the image.

3A engine 420 may perform an arithmetic process of matching and correcting images and obtaining luminance information.

The color processor 430 and the YUV processor 440 may sequentially perform operations on the color information. Here, the YUV processor 440 may perform an operation on the color information according to the YUV method.

The HDR control unit 450 may generate an exposure value control signal and transmit the generated exposure value control signal to the image sensor 300. [ The HDR controller 450 may determine the final exposure value of the image sensor based on the luminance information of the image synthesized by the synthesizer 400. [

The image combining unit 460 may combine the luminance information and / or the color information.

The video encoder 470 and the Jpeg encoder 480 may generate the synthesized image from the synthesis result of the image synthesizer 460. [ Here, Jpeg is a standard of a technique for compressing a still image such as a photograph for use in communication.

4 is a flowchart illustrating an image processing method according to an embodiment of the present invention.

Referring to FIG. 4, an image processing method according to an exemplary embodiment of the present invention includes a step S10 of obtaining a single color image, a step S20 of obtaining a multi-color image, a step S30 of HDR- (Step S40) of synthesizing a multi-color image and a multi-color image (step S50) of synthesizing a HDR-synthesized single color image and an HDR-synthesized multi-color image.

For example, the steps S10, S20, S30, S40, and S50 may be performed by the first and second image sensors shown in FIG. 1 and the first through third composing units, respectively. Therefore, the same or corresponding contents to those of the above-described image processing apparatus relating to the image processing method are not described redundantly.

The image processing apparatus in step S10 may obtain a plurality of single color images having different exposure values.

The image processing apparatus in step S20 may obtain a plurality of multi-color images having different exposure values.

The image processing apparatus in step S30 may obtain a single color image having a dynamic range extended from the plurality of single color images.

The image processing apparatus in step S40 may obtain a multi-color image in which the dynamic range is extended from the plurality of multi-color images.

The image processing apparatus in step S50 may combine the single color image in which the dynamic range is expanded and the multi-color image in which the dynamic range is expanded.

For example, the steps S10, S20, S30, S40, and S50 may be performed by the computing environment shown in FIG.

FIG. 5 is a diagram illustrating an exemplary computing environment in which one or more embodiments disclosed herein may be implemented, and is illustrative of a system 1000 including a computing device 1100 configured to implement one or more of the embodiments described above. / RTI > For example, the computing device 1100 may be a personal computer, a server computer, a handheld or laptop device, a mobile device (mobile phone, PDA, media player, etc.), a multiprocessor system, a consumer electronics device, A distributed computing environment including any of the above-described systems or devices, and the like.

The computing device 1100 may include at least one processing unit 1110 and memory 1120. [ The processing unit 1110 may include, for example, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array And may have a plurality of cores. The memory 1120 can be a volatile memory (e.g., RAM, etc.), a non-volatile memory (e.g., ROM, flash memory, etc.) or a combination thereof.

In addition, the computing device 1100 may include additional storage 1130. Storage 1130 includes, but is not limited to, magnetic storage, optical storage, and the like. The storage 1130 may store computer readable instructions for implementing one or more embodiments as disclosed herein, and other computer readable instructions for implementing an operating system, application programs, and the like. The computer readable instructions stored in storage 1130 may be loaded into memory 1120 for execution by processing unit 1110.

In addition, computing device 1100 may include input device (s) 1140 and output device (s) Here, input device (s) 1140 may include, for example, a keyboard, a mouse, a pen, a voice input device, a touch input device, an infrared camera, a video input device or any other input device. Also, output device (s) 1150 can include, for example, one or more displays, speakers, printers, or any other output device. In addition, computing device 1100 may use an input device or output device included in another computing device as input device (s) 1140 or output device (s) 1150. [

In addition, computing device 1100 may include communication connection (s) 1160 that enable communication with other devices (e.g., computing device 1300) via network 1200. [ (S) 1160 may include a modem, a network interface card (NIC), an integrated network interface, a radio frequency transmitter / receiver, an infrared port, a USB connection or other Interface. Also, the communication connection (s) 1160 may include a wired connection or a wireless connection.

Each component of the computing device 1100 described above may be connected by various interconnects (e.g., peripheral component interconnect (PCI), USB, firmware (IEEE 1394), optical bus architecture, etc.) And may be interconnected by a network.

As used herein, terms such as "component," "module," "system," "interface," and the like generally refer to a computer-related entity that is hardware, a combination of hardware and software, software, or software in execution. For example, an element may be, but is not limited to being, a processor, an object, an executable, an executable thread, a program and / or a computer running on a processor. For example, both the application running on the controller and the controller may be components. One or more components may reside within a process and / or thread of execution, and the components may be localized on one computer and distributed among two or more computers.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, Those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims.

110, 210: a first image sensor
120, 220: Second image sensor
130, and 230:
140, 240: a second synthesis unit
150, 250: third synthesis section

Claims (9)

A first image sensor for obtaining a plurality of single color images having different exposure values;
A second image sensor for obtaining a plurality of multi-color images having different exposure values;
A first combining unit for performing a high dynamic range (HDR) combining operation on the plurality of single color images;
A second combining unit for performing a high dynamic range combining operation on the plurality of multi-color images; And
A third synthesizing unit synthesizing an image synthesized by the first synthesizing unit and an image synthesized by the second synthesizing unit; Lt; / RTI >
Wherein the third combining unit applies the luminance information of the area corresponding to the area in the image synthesized by the first combining unit to the area where saturation has occurred in the image synthesized by the second synthesizing unit.
The method according to claim 1,
Wherein some of the plurality of single color images have a positive first exposure value,
Wherein the other of the plurality of single color images has a negative exposure value of the first exposure value,
Wherein some of the plurality of multicolor images have a positive second exposure value,
And the other of the plurality of multicolor images has a negative exposure value of the second exposure value.
3. The method of claim 2,
Wherein the plurality of single color images further comprise a single color image having an exposure value of zero,
Wherein the plurality of multi-color images further comprises a multi-color image having an exposure value of zero,
Wherein the first exposure value and the second exposure value are the same.
The method according to claim 1,
Wherein the first image sensor comprises a first pixel array for sensing light of first, second and third colors for each pixel,
The second image sensor includes a second pixel array for sensing light of the first color for each pixel, a third pixel array for sensing light of the second color for each pixel, and a third pixel array for sensing light of the third color for each pixel, Comprising a four pixel array,
Wherein the second image sensor determines the color of the plurality of multi-color images based on the luminance information of the second through fourth pixel arrays.
5. The method of claim 4,
Wherein the third synthesizing unit adds the luminance information of the first and second colors of the image synthesized by the second synthesizing unit to the image synthesized by the first synthesizing unit, And obtains an image of multiple colors based on the brightness information of the image and the brightness information of the first and second colors of the image synthesized by the second compositing unit.
delete The method according to claim 1,
Wherein the third synthesizing unit synthesizes the image synthesized by the first synthesizing unit and the synthesized image obtained by the second synthesizing unit based on the luminance information of the synthesized image, An image processing apparatus for determining a final exposure value of a sensor.
The method according to claim 1,
Wherein the first image sensor adjusts an exposure value of the plurality of single color images by adjusting a shutter opening /
Wherein the second image sensor adjusts an exposure value of the plurality of multi-color images by adjusting a shutter opening / closing speed at predetermined intervals.
Obtaining a plurality of single color images having different exposure values;
Obtaining a plurality of multi-color images having different exposure values;
Obtaining a single color image having an extended dynamic range from the plurality of single color images;
Obtaining a multi-color image having an extended dynamic range from the plurality of multi-color images; And
Synthesizing the single color image with the extended dynamic range and the multi-color image with the extended dynamic range; Lt; / RTI >
Wherein the synthesizing step applies luminance information of a region corresponding to the region in a single color image in which the dynamic range is extended to an area in which saturation occurs in the multi-color image in which the dynamic range is extended.

Images