KR20070105028A - Apparatus for processing image signal - Google Patents

Abstract

An apparatus for processing an image signal is provided to obtain plural images having different gains with respect to the same subject, divide the images into plural areas, measure image information with respect to the same area of each image, select only an area having largest image information and synthesize the areas, thereby providing a large dynamic area. An imaging device(310) photographs an arbitrary object and provides an image signal corresponding to the photographed object. A timing generator(320) adjusts an exposure time of the imaging device. A first frame memory(330) stores an initial image signal photographed by the imaging device. A second frame memory(340) stores an image signal newly photographed by the imaging device for the exposure time adjusted by the timing generator. A histogram/entropy calculator(350) calculates a histogram, cumulative density function and image information(entropy) of each block after dividing the image, stored in the first and second frame memories, into blocks. An exposure/gain controller(360) compares entropies of each block obtained by the histogram/entropy calculator and calculates an exposure time. A demultiplexer(370) separately distributes an initial image signal obtained by the imaging device and an image signal after the exposure time adjustment to the first and second frame memories. A mixing unit(380) mixes the images stored in the first and second memories so that image information can be maximized.

Description

Apparatus for processing image signal

BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a diagram illustrating a response characteristic and a dynamic range with respect to the incident light amount of a general electronic image pickup device.

2 is a portrait photograph obtained from an image pickup device having a limited dynamic range in a shooting environment requiring a wide dynamic range.

3 is a diagram illustrating a configuration of an image signal processing apparatus according to a first embodiment of the present invention.

4 is a diagram illustrating a configuration of an image signal processing apparatus according to a second embodiment of the present invention.

5 is a diagram showing a histogram of an original image, an image processed by the histogram equalization technique, and an image processed by the image signal processing apparatus of the present invention.

6 is a view showing an image taken at 20lux illuminance.

7 is a diagram illustrating an image processed by the histogram equalization technique of the image of FIG. 6.

8 is a view showing an image processed by the image signal processing apparatus of the present invention the image of FIG.

9 is a view showing average (total brightness), standard deviation (contrast), and information amount of an original image, an image processed by the histogram equalization technique, and an image processed by the image signal processing apparatus of the present invention.

FIG. 10 is a view showing an image obtained by processing 1.2 times the brightness of the image of FIG. 2; FIG.

11 is a view showing an image processed by the image signal processing apparatus of the present invention the image of FIG.

<Explanation of symbols for main parts of the drawings>

310,410 imaging device 320,420 timing generator

330,430 ... first frame memory 340,440 ... second frame memory

350,450 ... histogram / entropy calculator

360,460 ... exposure / gain control 370 ... demultiplexer

470 ... automatic gain adjuster 380,480

The present invention relates to an image signal processing apparatus, and more particularly, to obtain a plurality of images having different image gains for the same subject with respect to a limited dynamic region in an existing image pickup apparatus, and to divide them into various regions. The present invention relates to an image signal processing apparatus capable of providing a wide dynamic range by measuring image information on the same region of each image, selecting only the region having the largest image information, and synthesizing them.

In most imaging devices, as shown in FIG. 1, the output voltage increases linearly with light intensity, but linearly due to noise at low light intensity and saturation at high light intensity. The increasing portion is constrained to have a finite dynamic range. As shown in FIG. 2, the image obtained by the image capturing apparatus having the limited dynamic range generates an image in which bright portions and dark portions coexist, that is, a portion 201 that is not clear for a subject having a wide dynamic region. Done. Therefore, in order to obtain a clear image of a subject having a large dynamic range, it is necessary to increase the gain by selecting only a dark part.

One of the conventional methods for extending such dynamic range is histogram equalization. In this method, the cumulative density function is calculated from the histogram of the input image, and the luminance of each pixel is defined using the cumulative density function, thereby making the luminance distribution of the image uniform. However, when the frequency is leveled, as the contrast of the luminance band having a large number of distributions is greatly changed, the quantization noise increases and the portion having the luminance having a low frequency has a relatively low contrast in the overall image. In order to improve this, Korean Patent Publication No. 10-2000-0050571 (name of the invention: "Image signal processing apparatus and method having histogram equalization and color enhancement function") separates the image areas of the low light and high light parts, respectively. Applying the appropriate transform function to the result in the dynamic range extension and contrast improvement. However, this method is effective for an image with a relatively narrow dynamic range, but since the exposure time of the sensor cannot be controlled for a subject having a wide dynamic range, the contrast may be reduced due to the saturation region and the noise region. In addition, discontinuity may occur at the boundary where the low light and high light areas meet, which may cause block of the image.

As another method, U.S. Patent No. 5,144,442 synthesizes an input image with a plurality of different exposures to have a wider dynamic range. However, this approach can be complicated by the need to have multiple input channels.

The present invention was created in view of the above problems, and according to a unique Smart Auto Exposure (Smart Auto Exposure) algorithm for a sensor, two images having different exposure times are obtained, and the information is divided into regions. By selecting and synthesizing a large region, it is possible to provide a clear image without deterioration of contrast even for a subject having a large dynamic region, and to process the image signal that can prevent blocking by synthesizing the divided region using Rational Gaussian. The object is to provide a device.

In order to achieve the above object, the image signal processing apparatus according to the first embodiment of the present invention,

An imaging device which photographs an arbitrary object and provides an image signal according to the object;

A timing generator for adjusting the exposure time of the imaging device;

A first frame memory for storing an initial video signal photographed by the imaging device;

A second frame memory for storing a video signal newly captured by the imaging device by the exposure time adjusted by the timing generator;

A histogram / entropy calculator for dividing blocks from the images stored in the first frame memory and the second frame memory to calculate a histogram, cumulative density function, and image information (entropy) of each block;

An exposure / gain control unit for calculating an exposure time by comparing the entropy of each block obtained by the histogram / entropy calculation unit;

A demultiplexer for separately dividing the initial image signal obtained by the image pickup device and the image signal after exposure time adjustment into the first frame memory and the second frame memory; And

It is characterized in that it comprises a mixing unit for mixing the image stored in the first frame memory and the second frame memory in a direction in which the image information is maximized.

In addition, in order to achieve the above object, a video signal processing apparatus according to a second embodiment of the present invention,

An imaging device which photographs an arbitrary object and provides an image signal according to the object;

A timing generator for adjusting the exposure time of the imaging device;

A first frame memory for storing an initial video signal photographed by the imaging device;

A histogram / entropy calculator for dividing blocks from the images stored in the first frame memory and the second frame memory to calculate a histogram, cumulative density function, and image information (entropy) of each block;

An exposure / gain control unit configured to calculate a gain by comparing the entropy of each block obtained by the histogram / entropy calculation unit;

An automatic gain controller for producing an image having a gain calculated by the exposure / gain control unit;

A second frame memory for storing an image newly obtained by the automatic gain adjuster; And

It is characterized in that it comprises a mixing unit for mixing the image stored in the first frame memory and the second frame memory in a direction in which the image information is maximized.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

3 is a block diagram of a video signal processing apparatus according to a first embodiment of the present invention.

Referring to FIG. 3, the image signal processing apparatus 300 according to the first exemplary embodiment of the present invention includes an imaging device 310, a timing generator 320, a first frame memory 330, and a second frame memory 340. ), A histogram / entropy calculator 350, an exposure / gain controller 360, a demultiplexer 370, and a mixer 380.

The imaging device 310 photographs an arbitrary object and provides an image signal accordingly. As the imaging device 310, a charge coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) image sensor may be used.

The timing generator 320 (hereinafter abbreviated as TG) is used to adjust the exposure time of the imaging device 310.

The first frame memory 330 is for storing an initial image signal photographed by the imaging device 310.

The second frame memory 340 is for storing an image signal newly photographed by the imaging device 310 by the exposure time adjusted by the timing generator 320.

The histogram / entropy calculator 350 divides blocks from images stored in the first frame memory 330 and the second frame memory 340 to calculate a histogram, cumulative density function, and image information (entropy) of each block. . Here, the histogram / entropy calculator 350 calculates entropy, which is image information, for a block having the same position in the first frame memory 330 and the second frame memory 340 by Equation 4 to be described later.

The exposure / gain control unit 360 calculates an exposure time by comparing the entropy of each block obtained by the histogram / entropy calculation unit 350. The exposure / gain control unit 360 calculates an exposure time by using Equations 1, 2, and 3 described below.

The demultiplexer 370 (abbreviated as Demux in the drawing) may display the first image signal obtained by the image pickup device 310 and the image signal after adjusting the exposure time of the first frame memory 330 and the second frame memory 340. ) And distribute them separately.

The mixer 380 mixes the images stored in the first frame memory 330 and the second frame memory 340 in a direction in which image information is maximized. The mixing unit 380 mixes the images stored in the first frame memory 330 and the second frame memory 340 by Equation 5 to be described later.

4 is a diagram illustrating a configuration of an image signal processing apparatus according to a second embodiment of the present invention.

Referring to FIG. 4, the image signal processing apparatus 400 according to the second exemplary embodiment of the present invention includes an imaging device 410, a timing generator 420, a first frame memory 430, and a second frame memory 440. , A histogram / entropy calculator 450, an exposure / gain controller 460, an automatic gain adjuster 470, and a mixer 480. As described above, the video signal processing apparatus 400 of the second embodiment is similar in structure to the video signal processing apparatus 300 of the first embodiment. Therefore, the description of the components having the same function as the first embodiment will be omitted, and only other portions will be described.

That is, in the case of this second embodiment, the point that the automatic gain adjuster 470 (abbreviated as AGC in the drawing) is employed in place of the demultiplexer 370 in the first embodiment is characteristically different, and therefore The related signal input / output circuit is also slightly different from the first embodiment, as can be seen in FIG.

Here, the automatic gain adjuster 470 serves to make an image having a gain calculated by the exposure / gain control unit 460 which will be described later.

The second frame memory 440 and the exposure / gain control unit 460 have the same name as the first embodiment, but differ in the function and role of the first embodiment.

That is, the exposure / gain control unit 460 calculates a gain by comparing the entropy of each block obtained by the histogram / entropy calculation unit 450.

The second frame memory 440 stores the image newly obtained by the automatic gain adjuster 470.

Next, an operation of the image signal processing apparatus according to the present invention having the above configuration will be described with reference to FIGS. 3 and 4.

First, the image output from the imaging devices 310 and 410 is controlled by the exposure / gain control unit 360 and 460 and the timing generators 320 and 420 until the mixing unit 380 and 480 obtain the necessary input image. Here, the definition of "saturated" is defined as when pixels having a certain ratio or more have a certain contrast level or more in the entire image. In this case, each threshold is set as 2% of the entire pixel of the input image, and the largest contrast level. For example, it means 256 levels in contrast with 8 bit depth. If the value corresponding to 98% of the cumulative density function of the input image is 256 steps or less, the exposure / gain control unit 360 adjusts the exposure time to compensate for the difference, and the ratio of pixels distributed in 256 steps is 98%. If it is above, the exposure time is adjusted to a value of 75% of the current exposure time so that the ratio of saturated and saturated pixels is 2% or less. If this is expressed as an expression, it is as follows.

, l_sat 값이 255이하일 때

l when _sat is less than 255

, l_sat 값이 255보다 클 때

l when _sat is greater than 255

Here, t _exp , t ₀ , and l _sat denote the contrast level when the final exposure time, the initial exposure time, and the cumulative density are 98%, respectively.

As described above, in the initial stage, the exposure / gain control units 360 and 460 obtain an input image (hereinafter, referred to as an initial input image) having an optimal dynamic range within the dynamic range of a given sensor. The reason for minimizing the saturation in the initial stage is to prevent the bright and delicate portions of the subject from being damaged by the saturation in the input image received from the imaging devices 310 and 410. The reason for the threshold is to adjust the degree of saturation according to the sensor or subjective judgment.

Once the initial stage is completed, the exposure / gain control unit 360 obtains an input image having a different exposure time to increase the brightness of the dark portion and stores the input image in the second frame memory 340 through the demultiplexer 370, or as shown in FIG. As described above, the gain of the initial input image is adjusted through the automatic gain adjuster 470 to generate a new input image and store it directly in the second frame memory 440. In this case, another exposure time or gain is programmed to be automatically determined according to the average value (hereinafter, average brightness) of the histogram calculated from the initial input image. The function that determines this value is the logarithmic function with the average brightness as an independent variable. The reason for determining the exposure time or gain by using the logarithmic function is that when the two input images are mixed, when the average brightness between the two images differs considerably, spots appear in the mixed images. Therefore, rather than maximizing the average brightness, it is reasonable to determine logarithmically according to the average brightness of the input image, which is in good agreement with human visual characteristics. Equation 2 below is a fitting function employed in the present invention, and each constant is empirically obtained by several experiments.

Here, l _ave and G are gains for obtaining the input image to correct the average brightness and the dark portion of the input image, respectively. And, a ₀ , a ₁ , a ₂ are constants to be determined by experiment. In addition, the exposure time required to obtain the input image to correct the dark portion from Equation 2 can be expressed as Equation 3 below.

Here, t _exp and t _{exp (0)} mean an exposure time and an initial exposure time required for the correction of the dark portion, respectively, and G is a gain calculated in Equation 2 above.

As described above, the input images stored in the first frame memories 330 and 430 and the second frame memories 340 and 440 are divided into M × N. If each of the equalized small images is defined as a block, the histogram / entropy calculator 350 or 450 next calculates entropy, which is image information, for blocks having the same position in the first frame memory 330 and 430 and the second frame memory 340 and 440. It is calculated by the equation (4).

Here, I _n denotes the entropy of the n-th block, and P _i ^R , P _i ^G , and P _i ^B represent the probability density for the contrast level i in red, green, and blue, respectively.

As such, the entropy calculated from each input image is compared with each other for the same block, and the weighting function is multiplied with the larger block (this process is called 'block gain expansion'). The blending process is performed by normalizing the sum of the block gain-enlarged images for the block. This process can be expressed by the following equations (5), (6) and (7).

Here, Y _HDRR and X _{i , j} are input images of frame memory j including a synthesized image and a block having a large entropy for the i-th block, respectively, and x _i and y _i are coordinates of the center pixel of the i-th block. , B _i means the effective area of the i-th block. At this time, the effective area and the number of blocks are obtained experimentally.

Meanwhile, FIG. 5 is a diagram showing a histogram of an original image taken at a low light (20lux) as shown in FIG. 6, an image obtained by using a histogram equalization method, and an image obtained by the image signal processing apparatus of the present invention.

As shown in FIG. 5, in the original image, most pixels are distributed at low light, while in the image using the histogram equalization technique, the pixels are distributed at most contrast levels. In addition, as shown in FIG. 9, the contrast (standard deviation) and the overall brightness (average) are improved, but many pixels are distributed in the saturated region, so that the image of the bright part is damaged as shown in FIG. 7. On the other hand, when the image signal processing apparatus of the present invention is used, as shown in FIG. 8, a clear image is obtained with almost no pixels saturated with the improvement of contrast and brightness. Comparing this to the video information amount, it can be seen that the video information amount according to the present invention is the largest as shown in FIG.

10 and 11 show that the image signal processing apparatus of the present invention is also useful for an image obtained in a backlight state as shown in FIG. FIG. 10 shows that the gain is 1.2 in the original image. The face of the child is slightly brighter, but the background is already saturated, and thus it is not possible to provide a clear background. However, in the case of FIG. 11 using the image signal processing apparatus of the present invention, it can be seen that a clear child's face and a clear background can be obtained at the same time.

As described above, the image signal processing apparatus according to the present invention obtains a plurality of images having different image gains for the same subject with respect to the limited dynamic region in the conventional imaging apparatus, and divides them into various regions. By measuring image information of the same region of each image, only the region having the largest image information is selected, and the combination thereof is advantageous in that a wide dynamic region can be provided.

Claims (5)

An imaging device which photographs an arbitrary object and provides an image signal according to the object; A timing generator for adjusting the exposure time of the imaging device; A first frame memory for storing an initial video signal photographed by the imaging device; A second frame memory for storing a video signal newly captured by the imaging device by the exposure time adjusted by the timing generator; A histogram / entropy calculator for dividing blocks from the images stored in the first frame memory and the second frame memory to calculate a histogram, cumulative density function, and image information (entropy) of each block; An exposure / gain control unit for calculating an exposure time by comparing the entropy of each block obtained by the histogram / entropy calculation unit; A demultiplexer for separately dividing the initial image signal obtained by the image pickup device and the image signal after exposure time adjustment into the first frame memory and the second frame memory; And And a mixing unit which mixes the images stored in the first frame memory and the second frame memory in a direction in which image information is maximized. The method of claim 1, And the histogram / entropy calculation unit calculates entropy, which is image information, for a block having the same position in the first frame memory and the second frame memory by the following equation.

Where I _n is the entropy of the nth block, P _i ^R , P _i ^G , and P _i ^B are the probability densities for the contrast level i in red, green, and blue, respectively. The method of claim 1, The exposure / gain control unit calculates an exposure time from the following equations.

l when _sat is less than 255

l when _sat is greater than 255 Where t _exp , t ₀ , and l _sat represent the final exposure time, initial exposure time, and contrast level when the cumulative density is 98%, respectively.

(Where a _ave and G are gains to obtain the input image to correct the average brightness and the dark parts of the input image, respectively, and a ₀ , a ₁ , and a ₂ are constants to be determined by the experiment)

Where t _exp and t _{exp (0)} are the exposure time and initial exposure time required for the compensation of dark areas, respectively. The method of claim 1, And the mixing unit mixes the images stored in the first frame memory and the second frame memory by the following equations.

(Where Y _HDRR and X _{i , j} are input images of frame memory j including a large entropy block for the synthesized image and the i-th block, respectively, and x _i and y _i are coordinates of the center pixel of the i-th block) B _i means the effective area of the i th block) An imaging device which photographs an arbitrary object and provides an image signal according to the object; A timing generator for adjusting the exposure time of the imaging device; A first frame memory for storing an initial video signal photographed by the imaging device; A histogram / entropy calculator for dividing blocks from the images stored in the first frame memory and the second frame memory to calculate a histogram, cumulative density function, and image information (entropy) of each block; An exposure / gain control unit configured to calculate a gain by comparing the entropy of each block obtained by the histogram / entropy calculation unit; An automatic gain controller for producing an image having a gain calculated by the exposure / gain control unit; A second frame memory for storing an image newly obtained by the automatic gain adjuster; And And a mixing unit which mixes the images stored in the first frame memory and the second frame memory in a direction in which image information is maximized.

Images