JPH06269009A - Color wide dynamic range camera - Google Patents

Abstract

PURPOSE: To improve the picture quality of video images by acquiring plural color video images of scenes of different exposure levels and processing the components of the color video images to generate a synthetic image including the emphasized details. CONSTITUTION: A color wide dynamic range camera 300 generally consists of a two-dimensional sensor array. However, a color camera 104 that can contain a linear sensor array sends the individual red, green and blue components to the monochromatic dynamic range systems 1021 to 1023 respectively. Then these components are individually processed as if they were the monochromatic signals.

Description

Detailed Description of the Invention

[0001]

[Industrial applications]

The present invention relates to apparatus and techniques for providing enhanced video images, especially color video images.

[0003]

[Prior art]

To date, various video image quality improving devices and technologies have been proposed. In order to reduce the dynamic range of the video signal by subtracting the DC level of the video signal,
In particular, automatic gain control (AGC) technology has been used for video signals. Such a technique is described in US Pat. No. 4,719,350, in which linear AGC is applied to the video image pixel by pixel.
No. This technique is only applicable to plain areas bounded by lines parallel to the scan direction.

Generally speaking, automatic gain control is only used in video processing on an interframe basis, not an intraframe basis.

[0006] Furthermore, Stockham's "Image Processing in the Context of Visual Model", IEEE Proceedings Volume 60 (7)
No.), July 1972, pages 828-842, and X.
IE, Stockham, “Toward the unification of the three laws of brightness perception and two visual models”, IEEE Trans System, Artificial Brain Science 19 (2), March 1989.
Mon / April pp. 379-382 (Gonzalez's "Digital Image Processing", Second Edition, Addison-We
Sley, pp. 185-186), the isomorphic filter shown in FIG. 3 of this document is used to improve the quality of black and white images. The logarithm of the input is calculated by the logarithmic module 501. Logarithmic module 501
Output is linear or non-linear, reducing low frequencies,
Received by the filter 502, which has a frequency dependent gain that tends to amplify high frequencies. The output of filter 502 is received by exponential (or antilogarithm) module 503. This method divides the illumination component and the reflectance component. The illumination component of an image generally has a slow spatial variation, whereas the reflectance component tends to change rapidly, especially at seams of dissimilar objects. Therefore, isomorphic filters tend to emphasize details at the expense of low frequencies. High spatial frequency (detail) of image is 2
A typical result of such a method is a two-fold enhancement and a DC component attenuated by half the original value.

[0007]

[Problems to be Solved by the Invention]

The present invention seeks to provide an improved video quality enhancing apparatus for color images which overcomes the limitations of the prior art apparatus and techniques described above.

[0009]

[Means for Solving the Problems]

Means for solving the problems of the present invention are as follows.

First, in a color wide dynamic range video imager, sensor means for providing a plurality of color video images of a scene at different exposure levels; dividing each color video image into components. Generating a composite color video image containing image information from the components of each of the plurality of color video images by processing at least one of the components of each of the plurality of color video images. Means for processing the component of each of the plurality of color video images to obtain.

Secondly, in the color wide dynamic range video image apparatus according to the first aspect, the means for processing processes each one of the plurality of color video images.
First means for converting one achromatic component and two chromatic components; for each of said achromatic and chromatic components thereby generating one treated achromatic component and two treated chromatic components Apparatus for converting the processed achromatic and chromatic components into at least three chromatic output components of each of the plurality of video images.

Thirdly, in the color wide dynamic range video image device according to the second aspect, the first means for the transformation uses the first matrix multiplier and the second means for the transformation is used. The means of using a matrix multiplier, the second matrix multiplier being the inverse of the first matrix multiplier.

Fourth, in the color wide dynamic range video image device described in the third, the first matrix multiplier has first and second elements of 0.2684, 0.6749 and 0.0567. Column and; element is 0.6762,-
A device comprising a second row of 0.5766 and -0.0997; a third row of elements 0.1604, 0.2434 and -0.4038;

Fifth, in the color wide dynamic range video system of the second aspect, one of the means for processing each of the plurality of video images; and an output of each of the processing means. Apparatus having adder means for combining.

Sixth, in the color wide dynamic range video apparatus according to the fifth aspect, an apparatus further comprising a histogram means for improving the output of the apparatus.

Seventh, in the color wide dynamic range video image device according to the first aspect, the means for processing processes one of each of the plurality of color video images.
First means for converting one achromatic component and two chromatic components; neighbor processing means for said achromatic component thereby generating one processed achromatic component; and applying a gain to said chromatic component Gain multiplying means for generating an amplified chromatic component, and converting the processed achromatic and the amplified chromatic component into at least three chromatic output components of each of the plurality of video images. A second means for:

Eighth, in the color wide dynamic range video apparatus described in the seventh, the gain multiplication means comprises a look-up table.

Ninth, in the color wide dynamic range video system of the seventh aspect, the processing means for each of the plurality of video images; and the output of each of the processing means are combined. Apparatus having adder means for the.

Tenth, the color wide dynamic range video apparatus of the ninth aspect, further comprising histogram means for improving the output of the apparatus.

Eleventh, in the color wide dynamic range video imager according to the first aspect, the component of each of the plurality of color video images is chromatic.

Twelfthly, in the color wide dynamic range video camera described in the eleventh aspect, the processing means receives each of the chromatic components and generates an intensity signal therefrom; Means for processing; means for synthesizing the processed intensity signal into the chromatic component; memory means for storing previous information for each of the chromatic components; and the chromatic component for the previous information for each of the chromatic components. And a means for synthesizing the chromatic components and an output lookup table means for converting the chromatic component.

Thirteenth, the color wide dynamic range video camera according to the twelfth aspect, further comprising second processing means for receiving the output of the memory means.

Fourteenth, in the color wide dynamic range video camera described in the twelfth, the processing means further exposes between the means for synthesizing the processed intensity signal into the chromatic component and the memory means. A camera characterized by having a dependent lookup table.

Fifteenth, in the color wide dynamic range video camera of the fourteenth aspect, the processing means applies a non-linear look-up table to the intensity signal from the image taken at an even higher exposure level. A camera characterized by being equipped with.

Sixteenth, in the color wide dynamic range video camera described in the fifteenth, the exposures are taken in different fields and then combined.

Seventeenth, the color wide dynamic range video camera described in the sixteenth, wherein the look-up table means includes a gamma correction means and a histogram correction means.

Eighteenth, in the color wide dynamic range video camera described in the eleventh aspect, the processing means further calculates a luminance component from the three chromatic components for each of the plurality of color video images. Means for calculating a convolutional luminance component from the luminance component; means for calculating a weighting coefficient from at least the luminance component; means for multiplying the three chromatic components by the weighting coefficient: And the camera.

Nineteenth, in the color wide dynamic range video camera described in the eighteenth, the calculating means divides the convolutional luminance component by the luminance component.

Twentyth, in the color wide dynamic range video camera described in the eleventh aspect, the processing means further calculates a luminance component from the chromatic component for each of the plurality of color video images. Means for dividing said chromatic component by said luminance component to generate a normalized chromatic signal; means for calculating a weighting factor from at least said luminance component of each of said plurality of color video images; said weighting factor A means for multiplying the normalized chromatic component by:

Twenty-first, in the color wide dynamic range video camera according to the twentieth, the processed luminance components from each of the plurality of color video images are added and an output from the sum is obtained. A camera comprising means for multiplying the composite color video image.

Twenty-second, in the color wide dynamic range video camera described in the first, the sensor means comprises a two-dimensional sensor array.

Twenty-third, in the color wide dynamic range video camera described in the first, the sensor means comprises a linear sensor array.

Therefore, in accordance with a preferred embodiment of the present invention, means for providing a plurality of video color images of a scene at different exposure levels, each color image being separated into at least three different components; A video imaging device having means for processing a plurality of color image components to produce a composite color video image containing image information from the individual color video images and including enhanced details of local areas therein. Provided.

Further in accordance with a preferred embodiment of the present invention,
A device for processing a plurality of video images comprises a device for locally enhancing the dynamic range of a portion of a composite video image.

Still further in accordance with a preferred embodiment of the present invention an apparatus for processing a plurality of color video images comprises an apparatus for storing boundary pointing information within the composite video image.

Further in accordance with a preferred embodiment of the present invention,
A device for processing a plurality of color video images comprises a device for applying neighborhood transformations to the components of a plurality of color video images.

Still further in accordance with a preferred embodiment of the present invention, means for providing a plurality of video images of a scene with different exposure levels, each image comprising at least three components; Video image enhancement device comprising a plurality of video images to produce a composite video image containing the image information of, and the enhanced details of local areas therein.

Still further in accordance with a preferred embodiment of the present invention the processing device may include an image quality enhancement device such as a histogram equalizer.

[0040]

【Example】

An embodiment of the device of the present invention will be described below with reference to the drawings.

FIG. 1 is a schematic diagram of a monochrome wide dynamic range camera.

In FIG. 1, 10 is a camera, 12 is a camera timing, 14 is an A / D, 16 is an input LUT, 17 is an exposure selector, 18 is a host CPU, 20 is NTP, 22 is timing control, and 24 is a synthesizer. , 26 is a frame buffer, 28 is an output LUT, 30 is an RT histogram equalizer, 32 is a frame memory, and 34 is a frame memory L.
UT, 36 is a video generator, 37 is a sync generator, and 38 is a video monitor.

FIG. 2 is a schematic diagram of an exposure selector forming part of the circuit of FIG.

In FIG. 2, 14 is an A / D and 16 is an input LU.
T, 18 are host CPUs, 22 is timing control, 20
6 is a counter and 208 is the Nth.

FIG. 3 is a schematic diagram of a prior art isomorphic filter.

In FIG. 3, reference numeral 501 is a logarithm, 502 is a filter, and 503 is exposure.

FIG. 4 shows a first embodiment of a color wide dynamic range camera.

In FIG. 4, 104 is a color camera and 102 is a WDR system.

FIG. 5 shows the prior art Fauger.
as) algorithm.

FIG. 6 is for obtaining the required plurality of exposure levels which can replace the parallel channels of all subsequent embodiments, especially those shown in FIGS. 7, 8, 12, 13 and 14. FIG. 6 is a schematic diagram showing the use of a single detector.

In FIG. 6, 119 is a memory buffer and 121
Indicates a selector, and 120 indicates long exposure and short exposure.

FIG. 7 is a schematic diagram of a second embodiment of a color wide dynamic range camera showing two separate branches, one for each exposure level.

In FIG. 7, reference numeral 120 denotes long exposure, short exposure, and 126.
Indicates a histogram.

FIG. 8 is a schematic diagram of a third embodiment of a color wide dynamic range camera.

In FIG. 8, reference numeral 120 indicates long exposure and short exposure.

FIG. 9 is a schematic view of the fourth embodiment of the color wide dynamic range camera.

In FIG. 9, 300 is an image taken with short / long exposure, 132 is (A ⁺ -gain * A) / gain, 134 is R memory, 136 is G memory, 138 is B memory, and 140 is an output LUT. Is shown.

FIG. 10 is a schematic view of the fifth embodiment of the color wide dynamic range camera.

In FIG. 10, 300 is an image photographed with short / long exposure, 132 is (A ⁺ -gain * A) / gain, and 134 is R.
Memory, 136 is G memory, 138 is B memory, 140
Is an output LUT, 142 is LUT1 (long exposure), LUT2
(Short exposure) is shown.

FIG. 11 is a schematic diagram of a sixth embodiment of a color wide dynamic range camera.

In FIG. 11, 300 is an image taken with short / long exposure, 132 is (A ⁺ -gain * A) / gain, and 134 is R.
Memory, 135 convolution, 136 G memory, 137 convolution, 138 B memory, 139 convolution, 140 output LUT, 142 LUT1 (long exposure), LUT2
(Short exposure) is shown.

FIG. 12 is a schematic view of the seventh embodiment of the color wide dynamic range camera.

FIG. 13 is a schematic diagram of a first modification of the seventh embodiment of the color wide dynamic range camera.

FIG. 14 is a schematic diagram of a second modification of the seventh embodiment of the color wide dynamic range camera.

With particular reference to the drawings in which the same elements are numbered the same in a number of figures, it can be seen that FIG. 1 is a monochrome wide range dynamic system of the present application. Monochrome wide dynamic range camera 1
00 is a visible or invisible light sensor of any type that enables the exposure time to be changed by an externally supplied control pulse such as CCD, CID, photodiode array or other type of external exposure control. The camera 10 is provided. Additional components of the monochrome wide dynamic range camera 100 other than the camera 10 will be collectively referred to as a monochrome wide dynamic range system 102 hereinafter.

The monochrome wide dynamic range system 102 further includes a camera timing circuit 12 that provides timing pulses to the camera 10. The timing circuit 12 can include a conventional clock, counter and frequency divider. The timing pulse supplied to the camera 10 initiates photo-electrical accumulation of charge in the sensor array to change the selectable duration while at the same time sensing, preferably via a preamplifier circuit built into the camera 10. Of the signal current generated by
It also serves to control the reading to the D converter 14. The control of photo-electrical storage of charge is generally accomplished in two ways, either by operating a shutter, such as an electronic shutter to control the light input, or by controlling the integration time of the sensor array.

The digitized video data from the A / D converter 14 is supplied in parallel to two systems, a look-up table (LUT) 16 and an exposure selector 17. The exposure selector is arranged in parallel as shown in FIG. 2 and comprises first and second comparators 200 and 202 which output to an AND gate 204. Comparator 200 is 0-2
The signal from the A / D converter 14 in the 8-bit range of 55 is set to a low threshold level I such as the signal level of 20.
Compare with (L). Comparator 202 compares the signal from A / D converter 14 to a high threshold level I (H), such as the signal level at 235. The traffic light is above 20, 23
When less than 5, the two comparators together generate a logical "true" signal, which is ANDed by AND gate 204.

The output of the AND gate 204 is the counter 20.
6 and the counter is incremented when AND gate 204 receives two "true" signals.

The counter 206 is reset at the beginning of each frame. When the image in the current frame is nearly saturated, i.e. when many pixels are white (e.g. having a digital value of 255 or close to it), the counter value at the end of the frame will be a very low number. The same is true when the image is almost cut off, that is, when many pixels are black (that is, the digital value is 20 or less). Conversely, for a “normal” image with some value variation, the majority of pixels are between 20 and 235.
With the value of. In the case of such an image frame, the value of the counter 206 at the end of each frame becomes a large number.

The output of the counter 206 is supplied to the comparator 208. At the end of each frame, counter 206
Is compared with a threshold N (th) by a comparator 208. This threshold is selected to determine if the image of the frame is "normal" for images that are nearly saturated or cut off. When the output value of the counter 206 is higher than N (th), the value of the comparator 208 is logical “true”. This output is supplied to both the timing control circuit 22 and the host CPU 18 (FIG. 1).

Measuring and determining whether a particular frame at an exposure level should be combined with multiple frames at different exposure levels can be performed in at least two ways.

According to one embodiment of the monochrome wide dynamic range camera 100, the measurement is performed by the host CPU1.
It can be done on a relatively sporadic basis determined by 8. In such a case, a complete series of exposures is performed that covers the full range of exposures the system is capable of. At the end of each exposure, the output of comparator 208 is received by host CPU 18.

The host CPU uses the parameters I (L), I
(H) and N (th) can be controlled and these can be freely modified. The information collected assists the host CPU 18 in determining which exposure to take or use until the next measurement. For another embodiment of the monochrome wide dynamic range camera 100, briefly, the monochrome wide dynamic range camera provides two or more exposures, one long exposure, one short exposure, and perhaps an intermediate exposure. It works by Each image is, for example, 1 + epsilon (where ε
Is between 0 and 1 and is generally equal to approximately 0.2)
And minus 1/8 of the remaining 8 peripheral points (-1 /
8, which in other equivalent formulas can also be normalized to -1) and The convolved images are then added pixel by pixel. The results can be further processed, including, for example, histogram transformations.

Color wide dynamic range camera 3
A first embodiment of 00 is shown in FIG. Generally 2
A color camera 104, which consists of a dimensional sensor array, but may also include a linear sensor array, sends separate red, green, and blue components to the monochrome wide dynamic range systems 102 ₁ , 102 ₂ , 102 ₃ , respectively. Each component is processed separately as if it were a monochrome signal. However, this embodiment is computationally intensive and produces distortion in the output color.

In FIG. 5, the logarithm is applied, and the red, green and blue signals are achromatic (A) and chromatic (C1, C).
2) A prior art color process or algorithm developed by Faugeras that is transformed into a signal (similar to Y, I and Q signals in color television respectively).
Faugeras, PhD Thesis, University of Utah, Comput
er Science 1976, see also: Fau.
geras, "Color Model and Pe"
rception ”, IEEE Transactio
ns ASSP, August 1979). Subsequently, a high pass filter or other filter is applied to each of these three components separately, and the modified R _out , G _out and B _out signals are reproduced by inverse transformation and indexing as shown in FIG. . FIG. 5 shows a conversion module 106 for converting an R-G-B signal into an L-M-S signal. The logarithmic conversion module 108 and the P conversion module 110 convert the LMS signal into the Faugeras A-C1-C2 signal. Filters 112 ₁ , 112 ₂ , 112 ₃ are A-C1-C2
It is applied to each of the signals to generate A ⁺ -C1 ⁺ -C2 ⁺ . The filter 112 ₁ is particularly preferably a high pass filter. The P-inverse module 114, the exponent module 116 and the U-inverse module 118 are A ⁺ -C1 ⁺ -
C2 ⁺ is converted to R _out -G _out -B _out.

As seen in the above-mentioned Ph.D. dissertation of Faugeras,

[0078]

[Formula 1]

[0079]

[Formula 2]

U ^-1 and P ^-1 correspond to the inverse matrix derived by the standard matrix algebra method.

Color wide dynamic range camera 3
The second example of 00 is as shown in FIG. This embodiment removes the logarithmic module 108 and the exponential module 116 and combines the U conversion module 106 and the P conversion module 110 into a single M conversion module that multiplies the input matrix by the matrix algebraic product M of U and P.

[0082]

[Formula 3]

Similarly, the M-inverse module 124 multiplies the input matrix by the matrix M ^-1 , which is, of course, the inverse of the M matrix described above. Note that the first column of the matrix M ^-1 is all 1's.

Neighbor processing modules 122 ₁ , 122 ₂ , 1
22 ₃ generally contains convolutional open nuclei, which are all central elements 1 + epsilon (ε) and eight peripheral elements minus 1/8 (-1/8). ) Is preferably the aforementioned 3 × 3 matrix.

Furthermore, this process is applied to two or more separate exposures and the results are adders 125 ₁ , 125 ₂ , 125 _3.
Are added together for each pixel. The histogram transform module 126 or other operation can then be applied. From multiple aiming detectors synchronized together and set to different exposure levels, or as shown in FIG. 6, the exposure level varies from frame to frame, field to field, or scan lines for linear arrays. Multiple exposures can be taken from a single detector, which varies from time to time. In the case of a single detector, the input memory buffer is provided by the buffer 119 so that its output has a suitable short exposure for the two input channels of FIG. 7 (see also FIGS. 8, 12, 13, and 14). Selected by selector 121 to provide a long exposure. "Detector"
The term single CCD or other sensor array combined with appropriate color filters or groups of two or three arrays aligned on a beam splitting prism such that each sensor array detects a distinct color component. Pointing to. Color component division methods using optical filters and beam splitting prisms are well known to those skilled in the art. Figures and preferred embodiment described in this application is RGB color - but as the input and output _{components, Y / C R / C B} , Y / C, Cyan / Magenta / Yellow / green complementary mosaic or other color expression Can be used in exactly the same way.

Color wide dynamic range camera 3
The third embodiment of No. 00 is as shown in FIG. Unlike the embodiment shown in FIG. 7, dynamic range reduction is achieved only by applying a point transform to the chromatic component (eg, a look-up table or fixed gain applied per pixel) to the achromatic signal component (A). This is accomplished by performing a convolution of More specifically, the neighbor processing modules 122 ₂ , 122 ₃ of FIG. 7 typically use fixed gains using multiples between 1.0 and 1.5 (which may be different from each other or all the same). Operator modules 123 ₁ , 123 ₂ , 123 ₃ , 123 ₄
Has been replaced by. The higher the value of the multiplication, the generally deeper the color saturation.

The possible programming of the look-up table 128 allows for histogram scaling to expand the contrast area of the output image. Alternative programming can also provide gamma correction, which is well known in video systems, for the purpose of enhancing dark areas in an image, for example:

I _out = k _gamma * I _in ^gamma

In this equation, for example, gamma is approximately equal to 0.7 and k _gamma is in the range 5.0 to 11.0.

Color wide dynamic range camera 3
The fourth embodiment of 00 is a simplified hardware version of the third embodiment of the present invention. The derivations below relate to the matrix multiplication M ⁻¹ , where <R _out , G _out , B _out > considers only one channel, ignoring the output look-up table.

[0091]

[Formula 4]

Next, in FIG. 9 where the above formula has been implemented, the image or set of images is divided into red, green and blue components. The multiplier / adder block 130 is Niblac
k's formula ("Introduction to Image Processing" by Niblack, 59-
(See page 60), that is, A = 0.3 * R + 0.6 *
Calculate the function of achromatic strength using G + 0.1 * B. In the neighborhood processing block 132, the central element is (1
+ Ε−gain) / gain, and the remaining elements are −1 / (8
* Perform equation (A ⁺ -gain * A) / gain with a 3x3 open nucleus equal to (gain). Applicants have found that ε is 0.2 and 0.
We have empirically found that the useful region of gain must be between 1.0 and 1.35. The output of the neighborhood processing block 132 is the adders 133 ₁ , 133 ₂ , 1
Each of these signals is absorbed by the R.G.B. signal by 33 ₃ .

Memory blocks 134, 136, 138
Stores the R, G, and B signals for each frame in order to obtain the summation with the next frame.

The output lookup table 140 implements the function: I _out = k * (I _in * gain) ^gamma , which may include multiplication by the “gain” required by the above formula. Gamma is 0.7
Is approximately equal to and k is between 5 and 10.

The fifth embodiment of the color wide dynamic range camera 300 shown in FIG. 10 realizes a higher dynamic range (wider histogram) in the bright area of the image than that of the previous embodiments. This embodiment is accomplished by applying different correction look-up tables for short and long exposure images.

Elements 125, 130, 132, 13 of FIG.
3, 134, 136, 138, 140 are substantially the same as the correspondingly numbered elements shown in FIG. However, an exposure dependent lookup table 142 has been added downstream of the intersection of the neighborhood processing block 132 and the R, G, B components of the image. The exposure-dependent look-up table 142 is a first look-up table (LUT) that operates on a long-exposure image.
1) is provided. A typical function performed by LUT1 (I) is LUT1 (I) = 5 * I ^0.6 .
Similarly, the exposure-dependent look-up table 142 is a second look-up table (LUT) that operates on short-exposure images.
It also has 2). A typical function performed by LUT2 is LUT (I) = I ^1.1 . The use of the exposure dependent look-up table 142 thereby enhances the dynamic range (histogram) of bright areas, thereby improving contrast and adding saturation of the color in bright areas by adding background DC levels from long exposure frames. Correct the loss. In addition, LUT1 and LUT2
Can be modified to include a gain factor, thereby simplifying the 3 × 3 convolution open core of the neighborhood processing block 132 with one central element epsilon (ε) and minus one eighth (−1). It is also possible to include eight peripheral elements of / 8).

Therefore, the output LUT is multiplied by a constant (by the display monitor "not shown") if necessary to expand only the final histogram.

The function outputs: LUT (I) = minimum (limit, 255 / max (R95, G
95, B95) * I) automatically determined by looking at the histogram.

In this formula, R95, G95, and B95 are red,
95% of the green and blue pixels do not exceed the respective values. The limit is the highest value that can be displayed on the monitor, which is typically between 200 and 255 on an 8-bit per pixel monitor.

An alternative useful LUT function is: LUT (I) = minimum (limit, limit / max (R95, G9
5, B95) * I).

These LUT functions are typical examples,
Other functions can be selected as well. The three parameters--the sharpness coefficient epsilon (ε) and the contrast and brightness (output histogram coefficient)-can be changed by the user of the system. The contrast and brightness factor can be set to different values for each of the three color components to "white balance" the system.

However, the embodiments described thus far have the following drawbacks:

1. A narrow dynamic range is generated for bright areas of the image. Within these regions, frames with long exposures or open apertures (or at high exposure levels) are saturated or the signal is very high and relatively uniform. After processing, these areas in the frame with the open aperture will have uniform signal levels. Within the combined image, this uniform, or DC, signal increases the amount of white in the image, but does not add detail.

2. Moving objects in the image produce "movement artifacts." This is 2 when combined
Significant displacement between two frames causing double contours and bleeding effects.

3. Digital noise occurs in the gamma correction table. When the slope of the gamma correction function is larger than 1 (abrupt than 45 degrees), the step between adjacent pixel values becomes large. Therefore, a small difference that is almost inconspicuous leads to the appearance of a large step or is converted into a steeper difference, so that a boundary appears.

These drawbacks are corrected by the following modification as shown in the sixth embodiment of the color wide dynamic range camera 300 of FIG. In FIG. 11, FIG.
In addition to including some components corresponding to 0 components, the following additional elements are also included.

1. Prior to the convolution by the neighborhood processing block 132, the non-linear look-up table is LUT-A 1
31 applies to the A signal of the open aperture frame only.

The useful functions are: LUT (A) = (A ^1.8 / 256 ^1.8 ) * 256, which emphasizes high intensities such that they are more strongly attenuated by the subsequent convolution.

Thus, the spatial average of the brighter areas in the image tends to have a lower DC whitening level than the previously described embodiments.

2. To address motion artifacts, a field mode is introduced whereby two fields are captured and combined at two different apertures (or exposure levels) instead of two frames. Since the field velocity is twice as fast as the frame velocity in the interlaced scanning camera, the subject moves only half as much as between fields as it does when moving between frames, so the motion artifacts contained in the synthesized result are The number and degree are greatly reduced.

However, since the two fields are interlaced and scanned, the pixels included in the fields are not the same. Therefore, the synthesis algorithm synthesizes pairs of neighboring pixels rather than pairs of the same pixels. Therefore, the resolution is lost to some extent. In this embodiment, this effect is due to the frame memory block 134 storing the previous field,
Module 135 added after 136, 138,
137, 139, implemented in the form of a convolution, generally corrected by low-pass filter means. Neighbor operations on memory images allow different processing of the image's direct and delayed paths (via memory). Alternatively, the open core of convolver 132 can be modified to include spatial filtering functionality, so that convolvers 135, 1
37 and 139 are unnecessary.

This spatial filter method can also be effectively used in reducing motion artifacts in the frame mode.

3. The output LUT 140 is used for gamma correction and histogram correction. A sigmoid curve may be used instead of the mathematically calculated correction described above to remove digital noise in the output LUT 140. This can reduce the slope of the gamma correction function in the problem area.

While the embodiments described so far, which are based on the additive algorithm, achieve the desired result of capturing and reproducing high contrast color images, they do so in and around saturated areas in the picture. Tends to produce unusual, unnatural color effects and distortion. In order to maintain the visual sensation of true color in the picture, it is necessary to maintain the ratio of R / G and B / G to the original value even if the absolute values of R, G, and B change.
The examples described so far, including the additive algorithm, were not able to maintain these ratios in all cases.

In order to avoid the above-mentioned deficiencies of the additive algorithm, a multiplication algorithm was introduced in the seventh embodiment shown in FIG. The additive algorithm of the previous embodiment (in which case, for example, the result of the convolution, (A ⁺ -gain *
A) / gain, which is added to the original R, G, and B values), the multiplying algorithm of FIG. 12 maintains luminance ( Multiply a function of the achromatic signal. The brightness function is set to 2 in order to give a large weight to the information-bearing exposure at the pixel.
R-G at each pixel of each exposure before summing the two exposures (these two exposures can be obtained by two individual detectors or a single switched detector arrangement shown in FIG. 6) Selected to weight the B chromatic values. More specifically, as shown in FIG.
Blocks 143 and 144 calculate luminance values Y ₁ and Y ₂ , respectively, from the input RGB signals from each of the two exposure levels. The intensity values are then processed by the Neighbor Processing (NP) blocks 146, 148 to obtain the boundary enhancement intensity values Y ₁ ⁺ , Y ₂ ^+.
(Similar to block 132 of the previous example). The purpose of the Neighbor Processing (NP) block is to give more weight to the part of the image within a particular exposure that is rich in boundary information (Y ⁺ is significantly different from Y) and is essentially boundary information-less (Y ⁺ is Y. (Similar to)) to suppress the saturation or cutoff region. The weighting coefficient is the convolution brightness value Y ₁ ⁺ ,
It is subsequently calculated by dividing Y ₂ ⁺ by the unconvoluted luminance values Y ₁ , Y ₂ by the respective split blocks 150, 152. The resulting weighting coefficients Y ₁ ⁺ / Y ₁ , Y ₂
⁺ / Y ₂ is multiplied by each color component for each exposure by multiplying blocks 154, 156. The resulting image is composited by block 158 to obtain the desired composite output.

Another method for weighting the color components is shown in FIG. The two intensity channels are treated as before. Two pre-computed weighting functions (added to 1.0 for each pixel) are provided by block 160, for example by means of a look-up table or the like. The six color channels are first divided or normalized by the corresponding intensities, then weighted with W ₁ and W ₂ by blocks 154 and 156, then added together by block 158 and finally by block 164. Y ⁺ is multiplied by (Y ⁺ is calculated by an adder 162 for adding two convolution intensity from the block 146, 148). The purpose of normalizing the chromatic signal by the corresponding intensities is to maintain the desired intensity relationship in the output image by adding Y to the resulting colors.
It is possible to multiply by ⁺ .

FIG. 14 shows a modified version of the proposed architecture, which is somewhat more efficient. R ⁺ G for Y ⁺
Instead of multiplying by B (3 products), W ₁ and W ₂ are multiplied by Y ⁺ by blocks 168 and 170, respectively, to save one multiplication.

In this way, the above-mentioned objects and advantages of the present invention are most effectively achieved. While the present application discloses and describes preferred embodiments of the present invention, it is not intended that the invention be limited thereto, and the scope of the invention should be determined by the appended claims.

[0119]

【The invention's effect】

According to the present invention, it is possible to improve the image quality of a video image, especially a color video image.

[Brief description of drawings]

FIG. 1 is a schematic diagram of a monochrome wide dynamic range camera.

2 is a schematic diagram of an exposure selector forming part of the circuit of FIG. 1. FIG.

FIG. 3 is a schematic diagram of a prior art isomorphic filter.

FIG. 4 is the first color wide dynamic range camera.
It is an example of.

FIG. 5 Prior Art Faugeras
It is an algorithm.

FIG. 6 All embodiments, in particular FIGS. 7, 8, 12, 13
15 is a schematic diagram showing the use of a single detector to obtain the required multiple exposure levels, which can replace the parallel channels of those shown in FIGS.

FIG. 7 is a schematic diagram of a second embodiment of a color wide dynamic range camera showing two separate branches, one for each exposure level.

[FIG. 8] Third color wide dynamic range camera
It is a schematic diagram of an example of.

FIG. 9 is a fourth color wide dynamic range camera.
It is a schematic diagram of an example of.

FIG. 10 is a schematic diagram of a fifth embodiment of a color wide dynamic range camera.

FIG. 11 is a schematic view of a sixth embodiment of a color wide dynamic range camera.

FIG. 12 is a schematic view of a seventh embodiment of a color wide dynamic range camera.

FIG. 13 is a schematic view of a first modification of the seventh embodiment of the color wide dynamic range camera.

FIG. 14 is a schematic view of a second modification of the seventh embodiment of the color wide dynamic range camera.

[Explanation of symbols]

10 camera 12 camera timing 14 A / D 16 input LUT 17 exposure selector 18 host CPU 20 NTP 22 timing control 24 synthesizer 26 frame buffer 28 output LUT 30 RT histogram equalizer 32 frame memory 34 frame memory LUT 36 video generator 37 synchronization generation 38 Video Monitor 102 WDR System 104 Color Camera 119 Memory Buffer 120 (Long Exposure), (Short Exposure) 121 Selector 126 Histogram 132 (A ⁺ -Gain * A) / Gain 134 R Memory 135 Convolution 136 G Memory 137 Convolution 138 B memory 139 convolution 140 output LUT 142 LUT1 (long exposure), LUT2 (short exposure) 206 counter 208 Nth 300 image taken with short / long exposure 5 01 Logarithm 502 Filter 503 Exposure

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Noam Solek Israel, Kiryat Harochette (no address) (72) Inventor Tamar Jinosaar Israel, Haifa, Fitchmans Treat 20 (72) Inventor Yehoshua Wai. Zeevi Israel, Haifa, Alexander Yannai Street 48 (72) Inventor Daniel Jay. Crigler Israel Moshav Zippoli (No house number)

Claims (23)

[Claims] 1. In a color wide dynamic range video imager, sensor means for providing a plurality of color video images of a scene at different exposure levels; and means for dividing each color video image into components. When;
Processing the at least one of the components of each of the plurality of color video images to produce a composite color video image containing image information from the components of each of the plurality of color video images. Means for processing said component of each of a plurality of color video images. 2. The color wide dynamic range video imager of claim 1, wherein the means for processing divides each of the plurality of color video images into an achromatic component and two chromatic components. First means for converting; a neighboring processing means for each of the achromatic and chromatic components thereby generating one treated achromatic component and two treated chromatic components; Second means for converting the chromatic and chromatic components into at least three chromatic output components of each of said plurality of video images. 3. The color wide dynamic range video imager of claim 2, wherein the first means for converting uses a first matrix multiplier and the second means for converting. Using a matrix multiplier, wherein the second matrix multiplier is the inverse of the first matrix multiplier. 4. The color wide dynamic range video imager of claim 3, wherein the first matrix multiplier comprises a first column whose elements are 0.2684, 0.6749 and 0.0567. The elements are 0.6762, -0.5766 and-
The second column is 0.0997; the element is 0.1604,
A third column which is 0.2434 and -0.4038; 5. A color wide dynamic range video apparatus according to claim 2, wherein one of said means for processing each of said plurality of video images; and an output of each of said processing means is combined. And an adder means for the device. 6. The color wide dynamic range video device of claim 5, further comprising a histogram means for enhancing the output of the device. 7. The color wide dynamic range video imager of claim 1, wherein the means for processing divides each of the plurality of color video images into an achromatic component and two chromatic components. First means for transforming; neighborhood processing means for said achromatic component thereby generating one processed achromatic component; and applying a gain to said chromatic component, thereby amplifying the chromatic component. Apparatus for generating gain multiplication means; and second means for converting the processed achromatic and amplified chromatic components into at least three chromatic output components of each of the plurality of video images. . 8. The color wide dynamic range video device of claim 7, wherein the gain multiplication means comprises a look-up table. 9. The color wide dynamic range video apparatus of claim 7, further comprising: the processing means for each of the plurality of video images; and further combining the outputs of each of the processing means. An apparatus having adder means. 10. The color wide dynamic range video device of claim 9 further comprising histogram means for enhancing the output of the device. 11. The color wide dynamic range video imager of claim 1, wherein the component of each of the plurality of color video images is chromatic. 12. A color wide dynamic range video camera according to claim 11, wherein said processing means receives each of said chromatic components and produces an intensity signal therefrom; Means;
Means for synthesizing the processed intensity signal into the chromatic component; memory means for storing previous information for each of the chromatic components; means for synthesizing the previous information for each of the chromatic components into the chromatic component A camera having output look-up table means for converting the chromatic component. 13. The color wide dynamic range video camera of claim 12, further comprising second processing means for receiving the output of the memory means. 14. A color wide dynamic range video camera according to claim 12, wherein said processing means further comprises an exposure dependent look between said means for combining said processed intensity signal into said chromatic component and said memory means. A camera characterized by having an up table. 15. The color wide dynamic range video camera of claim 14, wherein the processing means comprises a non-linear look-up table applied to the intensity signal from an image taken at an even higher exposure level. A camera characterized by being. 16. The color wide dynamic range video camera of claim 15, wherein the exposures are taken in different fields and then combined. 17. The color wide dynamic range video camera according to claim 16, wherein the look-up table means includes a gamma correction means and a histogram correction means. 18. The color wide dynamic range video camera of claim 11, wherein the processing means further calculates, for each of the plurality of color video images, a luminance component from the three chromatic components. Means for calculating a convolutional luminance component from the luminance component; means for calculating a weighting coefficient from at least the luminance component; means for multiplying the three chromatic components by the weighting coefficient: camera. 19. The color wide dynamic range video camera of claim 18, wherein the calculating means divides the convolutional luminance component by the luminance component. 20. The color wide dynamic range video camera according to claim 11, wherein the processing means further calculates a luminance component from the chromatic component for each of the plurality of color video images;
Means for dividing the chromatic component by the luminance component to generate a normalized chromatic signal; means for calculating a weighting factor from at least the luminance component of each of the plurality of color video images; A camera comprising: means for multiplying the normalized chromatic component. 21. The color wide dynamic range video camera of claim 20, wherein the processed luminance components from each of the plurality of color video images are added and the output from the composite is combined. A camera equipped with means for multiplying color video images. 22. The color wide dynamic range video camera of claim 1, wherein said sensor means comprises a two-dimensional sensor array. 23. The color wide dynamic range video camera of claim 1, wherein the sensor means comprises a linear sensor array.