KR102182696B1 - Image processing apparatus and image processing method - Google Patents

Abstract

An image processing apparatus 1A, comprising: a used image selection unit 30 for generating use-image selection information for selecting a long exposure image or a short exposure image to be used for each area of the image; An averaging processing unit 50 for obtaining averaged use-image selection information by averaging the past use-image selection information and the current time use-image selection information; And a WDR synthesizing unit 70 for generating a synthesized image by synthesizing a long exposure image and a short exposure image according to the averaged use-image selection information.

Description

Image processing apparatus and image processing method TECHNICAL FIELD

The present invention relates to an image processing apparatus and method, and more particularly, to an image processing apparatus and method for performing a WDR (Wide Dynamic Range) or HDR (High Dynamic Range) photographing function.

Recently, a shooting function called WDR (Wide Dynamic Range) or HDR (High Dynamic Range) has been increasing. According to this photographing function, an image of short exposure (hereinafter referred to as "short exposure image") and an image of long exposure (hereinafter referred to as "long exposure image") are continuously photographed and synthesized, so that the image sensor can shoot dynamics. You can get a dynamic range out of range. Such a shooting function is particularly effective in scenes with very high contrast ratios, such as backlit compositions.

In general, a criterion for selecting a region of an image, that is, a long exposure image or a short exposure image to be used for each pixel is determined with reference to pixel value information or motion detection information. In addition, it is common to combine a pair of short-exposure images and long-exposure images in successive image groups independently.

Here, let's look at a scene in which the camera vibrates slightly. For example, a case where a fixed camera installed outdoors vibrates slightly due to the passage of a vehicle (or by the wind), or a case where a portable camera vibrates due to slight hand shake will be described. When attention is paid to a specific area in the WDR composite image in such a scene, the selection of the short exposure image and the long exposure image fluctuates in the time direction, so that the difference in quality between the short exposure image and the long exposure image is like flicker. It may be seen.

The difference in quality between the short exposure image and the long exposure image will be described. In general, when WDR image synthesis is performed, a short exposure image is normalized to an exposure ratio and converted to the same brightness as a long exposure image, followed by synthesis. In a region where the long-exposure image is saturated, the short-exposure image and the long-exposure image cannot have the same brightness, but if the long-exposure image is not saturated, the brightness of the long-exposure image and the normalized short-exposure image may match. However, it is not easy to make the picture quality of both sides exactly the same.

For example, in the normalization by the exposure ratio, there are cases where the ratio of the intended exposure ratio and the actually photographed pixel value is not the same. This is because the shutter time setting accuracy is insufficient or the exposure characteristics of the image sensor are affected. When the short exposure image is normalized, the arithmetic precision may be problematic.

In addition, even if it is assumed that the brightness is matched, a short exposure image generally has a lot of noise. Therefore, if the short exposure image and the long exposure image showing the same subject frequently fluctuate, a difference in the amount of noise may be detected as flicker.

As a technique for the purpose of reducing the influence on the WDR synthesized image by such external vibration, there is a technique disclosed in the following patent documents.

The technique described in Japanese Patent Application Laid-Open No. 2003-143484 (hereinafter referred to as Patent Document 1) is a technique for synthesizing after absorbing the position shift of the entire screen by coordinate transformation, and the position of the short exposure image and the position of the long exposure image Even if it is shifted by vibrations such as hand shake, the effect can be excluded and synthesized.

The technology described in Japanese Patent Application Laid-Open No. 2012-165213 (hereinafter referred to as Patent Document 2) refers to the long-exposure image when determining which of the short-exposure image and the long-exposure image to use, and if the pixel value is greater than or equal to the threshold value. An algorithm using short exposure images is disclosed.

Since the long-exposure image is saturated, the image selection smoothly transitions and a natural composite image is created in the high-contrast area where the short-exposure image is used and the long-exposure image is properly exposed and the long-exposed image is adjacent. . However, since the pixel values of the saturated long exposure image are mixed, a phenomenon in which the color becomes lighter in the saturated portion of the high contrast region occurs. Therefore, in the technique of Patent Document 2, an image in which a pixel value region in which a long exposure image is bright is enlarged by a vibration amplitude, and a selection signal is determined with reference to this image, thereby expanding a region using a short exposure image.

The technology disclosed in Japanese Patent Application Laid-Open No. 2011-135152 (hereinafter referred to as Patent Document 3) is a technology for controlling bracket photography with a short shutter time to minimize misalignment caused by vibration.

The technique described in Japanese Patent Application Laid-Open No. 2013-066142 (hereinafter referred to as Patent Document 4) is a technique for suppressing synthetic artifacts and shaking images by using a short exposure image in a motion region. In this technique, when a long exposure image is photographed by shaking due to movement of a subject, a short exposure image having a small amount of shaking is used.

In the case of using the technique described in Patent Document 1, there will be an effect while observing one WDR composite image. However, the effect of vibration cannot be excluded when continuously observing the WDR composite image. For example, when a position of a short exposure image is moved to fit based on a long exposure image, if the reference long exposure image is affected by vibration, the effect of vibration remains in the composite image group. That is, according to the technique described in Patent Document 1, the selection of the short exposure image and the long exposure image may be changed in the time direction due to vibration, so that a part of the image may look like flicker.

In the technique described in Patent Document 2, since the saturated long-exposure image is not used for synthesis, the occurrence of unnatural color due to the use of the pixel values of the saturated long-exposure image for synthesis may be eliminated. However, according to this technique, the position of the transition region between the short exposure image and the long exposure image is only spatially moved. Accordingly, when paying attention to a specific region of the temporally continuous WDR composite image, a phenomenon in which a short exposure image and a long exposure image frequently fluctuate due to vibration may occur.

According to the technique described in Patent Document 3, the influence of rapid vibration cannot be excluded. Further, even assuming that the influence of vibration can be excluded, noise can be increased by a short shutter time.

The technique described in Patent Document 4 does not mention the flicker included in the continuous WDR synthesis result when vibration is present. However, when the vibration of the entire screen is detected as motion, a short exposure image is selected over the entire screen, and noise increases throughout the screen. In addition, when vibration is not detected as motion, the image used for synthesis is selected from the short exposure image and the long exposure image according to the pixel value, but the image used for synthesis changes in time, causing flicker. There is a possibility.

Japanese Patent Laid-Open No. 2003-143484 Japanese Patent Publication No. 2012-165213 Japanese Patent Laid-Open No. 2011-135152 Japanese Patent Laid-Open No. 2013-066142

An embodiment of the present invention provides an image processing apparatus and method capable of suppressing a phenomenon in which the selected images are frequently fluctuated by minute vibrations when a short exposure image and a long exposure image are selected for use in synthesis. I want to.

An image processing apparatus according to an aspect of the present invention includes a used image selection unit, an averaging processing unit, and a synthesis unit.

The use image selection unit generates use-image selection information for selecting a long exposure image or a short exposure image to be used for each area of the image.

The averaging processing unit obtains averaged use-image selection information by averaging the past use-image selection information and the current time use-image selection information.

The synthesis unit generates a composite image by synthesizing the long exposure image and the short exposure image according to the averaged use-image selection information.

According to the above configuration, when a short exposure image and a long exposure image are selected to be used for synthesis, the used-image averaged by the averaging processing unit is not the use-image selection information itself generated by the use image selection unit. Optional information is referenced. Accordingly, it is possible to suppress a phenomenon in which the selected images are frequently fluctuated by a minute vibration.

Preferably, the image processing apparatus comprises: a motion detector for detecting motion and obtaining motion detection information; And an update unit for updating the averaged use-image selection information according to the motion detection information. Here, the synthesizing unit generates a composite image by synthesizing the long exposure image and the short exposure image according to the updated use-image selection information.

According to this configuration, since the motion detected by the motion detection unit (for example, a large vibration or a moving object) is reflected in the synthesis process, artifacts generated by the synthesis are reduced.

In addition, preferably, the image processing apparatus includes a reduction processing unit for reducing the long exposure image and the short exposure image to obtain a reduced long exposure image and a reduced short exposure image. Here, the use image selection unit generates the use-image selection information according to the reduced long exposure image and the reduced short exposure image.

According to this configuration, use-image selection information is generated according to the reduced short exposure image and the reduced long exposure image, and the generated use-image selection information is transmitted to the averaging processing unit. In addition, in the averaging processing unit, the averaged use-image selection information is written into the memory as past use-image selection information. Accordingly, it is possible to suppress an increase in the memory area or an increase in the amount of data transfer.

In addition, preferably, the motion detector detects the motion according to the reduced long exposure image and the reduced short exposure image.

According to this configuration, noise is reduced during the reduction process, so that motion detection is possible using an image of good quality. Further, since motion detection with reference to a reduced image is equivalent to motion detection with reference to a certain area, a spatially robust detection result can be obtained.

In addition, preferably, the update unit updates the averaged use-image selection information so that a short exposure image is used in the motion region.

According to this configuration, artifacts in which the outline of the composite image is doubled can be reduced.

According to the image processing method of another aspect of the present invention,

Generating use-image selection information for selecting a long exposure image or a short exposure image to be used for each area of the image;

Obtaining averaged use-image selection information by averaging the past use-image selection information and the current time use-image selection information; And

And generating a composite image by synthesizing the long exposure image and the short exposure image according to the averaged use-image selection information.

According to the method, when a short exposure image and a long exposure image are selected to be used for synthesis, the averaged use-image selection information is referred to, not the use-image selection information itself. Accordingly, it is possible to suppress a phenomenon in which the selected images are frequently fluctuated by a minute vibration.

As described above, according to the present invention, when a short exposure image and a long exposure image are selected for use in synthesis, it is possible to suppress a phenomenon in which the selected images are frequently fluctuated by a minute vibration.

1 is a diagram showing the configuration and operation of an image processing apparatus according to a first embodiment of the present invention.
2 is a diagram showing a state in which a pair of a long exposure image and a short exposure image is continuously photographed.
3 is a block diagram showing a detailed configuration of an averaging processing unit.
4, 5, 6A, 6B, and 6C are diagrams for explaining examples of effects exerted by the first embodiment of the present invention.
7 is a diagram showing the configuration and operation of an image processing apparatus according to a second embodiment of the present invention.

The following description and the accompanying drawings are for understanding the operation according to the present invention, and parts that can be easily implemented by a person skilled in the art may be omitted.

In addition, the specification and drawings are not provided for the purpose of limiting the present invention, and the scope of the present invention should be determined by the claims. Terms used in this specification should be interpreted as meanings and concepts consistent with the technical idea of the present invention so that the present invention can be most appropriately expressed.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the present specification and drawings, elements having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions are omitted.

(Example 1)

First, the configuration and operation of the image processing apparatus 1A according to the first embodiment of the present invention will be described.

1 is a diagram showing the configuration and operation of an image processing apparatus 1A according to a first embodiment of the present invention. As shown in FIG. 1, the image processing apparatus 1A includes an image sensor 10, a frame memory 20A, a used image selection unit 30, a motion detection unit 40, an averaging processing unit 50, and an update unit ( 60), a WDR synthesis unit 70, and a gradation compression unit 80. Hereinafter, the operation of each functional block included in the image processing apparatus 1A will be described in detail in order.

The image processing apparatus 1A continuously photographs two images by changing the exposure setting of the image sensor 10. Here, it is assumed that short exposure photographing is performed first and then long exposure photographing is performed. However, you may want to take a long exposure first and then a short exposure.

The short-exposure image and the long-exposure image taken in this way are written in pairs in the frame memory 20A. The long-exposure image and the short-exposure image are photographed, and the photographed long-exposure image and the short-exposure image are written in the frame memory 20A continuously.

2 is a diagram showing a state in which a pair of a long exposure image and a short exposure image is continuously photographed. 2, a short exposure image (t-1), a long exposure image (t-1), a short exposure image (t), a long exposure image (t), a short exposure image (t+1), and a long exposure image ( It is shown that t+1) is photographed over time. In FIG. 2, the photographing time of each image is indicated in parentheses, but in the same way, the photographing time of each image is sometimes indicated in parentheses.

In this embodiment, at least the long exposure image at the current time and the short exposure image at the current time need only remain in the frame memory 20A. Therefore, the interval at which the long exposure image and the short exposure image are photographed is not particularly limited. Further, the long exposure image and the short exposure image may be written in the frame memory 20A at a cycle of every frame, or may be written to the frame memory 20A at a cycle of a plurality of frames.

In more detail, as in the example shown in FIG. 1, when there is an area for writing a pair of a long exposure image and a short exposure image, the image processing apparatus 1A refers to the pair of the long exposure image and the short exposure image. You just need to write consecutively for. For example, the short exposure image t-1 and the long exposure image t-1 written in the frame memory 20A may be overwritten by the short exposure image t and the long exposure image t. .

In addition, in the example shown in Fig. 1, the image processing device 1A is a system for outputting a long exposure image and a short exposure image, and the image sensor 10 outputs a long exposure image and a short exposure image in time division. However, the long exposure image and the short exposure image may be simultaneously output. In this case, the image processing apparatus 1A may have two systems, a system for outputting a long exposure image from the image sensor 10 and a system for outputting a short exposure image. Each shutter time is determined by the dynamic range of the subject to be photographed or the specifications of the image sensor.

In the exemplary embodiment of the present invention, the terms short exposure image and long exposure image are used, but these terms are not limited to the absolute exposure time of each of the two captured images. Accordingly, when two images having different exposure times are captured, an image having a relatively short exposure time among the two images corresponds to a short exposure image, and an image having a relatively long exposure time corresponds to a long exposure image.

The image sensor 10 is configured to image light from outside on a light-receiving plane of the imaging device, photoelectrically convert the imaged light into an amount of charge, and convert the amount of charge into an electric signal. The type of the image sensor is not particularly limited, and may be, for example, Charge Coupled De-Vice (CCD) or Complementary Metal Oxide Semiconductor (CMOS).

The used image selection unit 30 detects the saturation state or movement of the long exposure image and the short exposure image with respect to the short exposure image and the long exposure image read out from the frame memory 20A, and the short exposure image and the long exposure image Use-image selection information for selecting one of the exposure images as a use image is generated. Various algorithms are assumed as an algorithm for selecting either a short exposure image or a long exposure image.

For example, since a region saturated in a long exposure image is highly likely not to be saturated in a short exposure image, a short exposure image may be selected as the image used for the region. However, with this process alone, artifacts in which the contours are doubled may occur in areas with large movement. Accordingly, a process may be performed to detect the motion and reduce the phenomenon that the outline becomes doubled according to the motion. Including such processing, an algorithm for selecting either one of a short exposure image and a long exposure image is not particularly limited.

In addition, as described above, the use-image selection information may be a set of binary data indicating which one of a short exposure image and a long exposure image to be selected, but the long exposure image and the short exposure image are mixed at a certain ratio It may be a set of mixing ratios indicating whether or not. For example, the used image selection unit 30 may increase the mixing ratio of the short exposure image as the degree of saturation of the long exposure image increases. Further, the used image selection unit 30 may increase the mixing ratio of the short exposure image as the movement of the short exposure image or the long exposure image increases. The algorithm for calculating the mixing ratio of the short exposure image and the long exposure image is also not particularly limited.

The motion detection unit 40 detects motion and generates motion detection information. The method of motion detection is not limited, but when motion is detected from a short exposure image and a long exposure image, it is preferable to multiply and normalize any one image by a gain corresponding to the exposure ratio, and then calculate the difference.

In addition, the method of generating the motion detection information is not particularly limited. For example, the motion detection unit 40 may generate motion detection information according to a relationship between a threshold value and a motion detected according to a short exposure image and a long exposure image.

Specifically, the motion detection unit 40 may detect a difference between pixel values or gradients of corresponding regions in the short exposure image and the long exposure image, and detect a region having a difference greater than a threshold value as a motion region. Of course, the motion detection unit 40 may detect an area in which the difference is less than a threshold value as a non-motion area. The region in which the difference is equal to the threshold value may be detected as any region. The motion detection unit 40 may generate such a detection result as motion detection information.

The averaging processing unit 50 averages the past use-video selection information and the current time use-video selection information to obtain averaged use-video selection information. The averaging performed by the averaging processing unit 50 is not particularly limited, but may be an additive average. An example of the configuration of the averaging processing unit 50 will be described in detail as follows.

3 is a diagram showing an example of a detailed configuration of the averaging processing unit 50. As shown in FIG. 3, the averaging processing unit 50 may include an Infinite Impulse Response (IIR) filter 51 and a past use-image selection information memory 52.

For example, the IIR filter 51, the use of the current time input from the used image selection unit 30-image selection information and past use-past use read from the image selection information memory 52-image selection The information is added and averaged according to Equation 1 below.

In Equation 1, Out represents the use-image selection information output from the IIR filter 51 to the rear end, Sprv represents the past use-image selection information, Scur represents the current time use-image selection information, and COEF. Respectively indicate the mixing coefficient. The mixing coefficient may be set in advance or may be set by the user. The use-image selection information output from the IIR filter 51 to the later stage is also written in the memory 52 of the past use-image selection information, and is used for subsequent averaging by the IIR filter 51.

In addition, FIG. 3 shows an example in which the IIR filter 51 is used as a configuration for flattening the past use-image selection information and the current time use-image selection information. However, the averaging configuration is not limited to the IIR filter 51. For example, instead of the IIR filter 51, a Finite Impulse Response (FIR) filter may be used, or another filter may be used.

The update unit 60 updates the use-video selection information averaged by the averaging processing unit 50 according to the motion detection information generated by the motion detection unit 40. Specifically, the updater 60 refers to the motion detection information and updates the averaged use-image selection information so that a short exposure image is selected as a use image in the motion region. Doing so can reduce artifacts in which the outline of the composite image is doubled. On the other hand, the update unit 60 refers to the motion detection information and operates so as not to update the use-image selection information averaged by the averaging processing unit 50 in the non-motion area.

The WDR synthesis unit 70 synthesizes a long exposure image and a short exposure image according to the use-image selection information averaged by the averaging processing unit 50 to generate a WDR synthesis image. Specifically, the WDR synthesizing unit 70 uses a short exposure image for a short exposure image use area and a long exposure image for a long exposure image use area, according to the averaged use-image selection information. Create

The synthesis method by the WDR synthesis unit 70 is not particularly limited.

For example, a case where the value indicating the content of selecting a long exposure image is "0" and the value indicating the content of selecting a short exposure image is "1".

The WDR synthesizing unit 70 sets “α × (pixel value of the short exposure image) + (1-) for the corresponding pixels in the long exposure image and the short exposure image by setting the mixing ratio constituting the use-image selection information to α. α) × (pixel value of a long exposure image)" can be calculated, and the result of the calculation can be output as an image after synthesis (WDR image).

In this way, in the first embodiment of the present invention, when the short-exposure image and the long-exposure image are selected for use in synthesis, the averaged, not the use-image selection information itself generated by the use-image selection unit 30 The usage-image selection information averaged by the processing unit 50 is referred to. Accordingly, a phenomenon in which the selected images are frequently fluctuated by a minute vibration can be suppressed.

In addition, as described above, the WDR synthesis unit 70 may directly refer to the averaged use-image selection information output from the averaging processing unit 50, but the updated use-image selection information by the update unit 60 You can refer to it. That is, the WDR synthesizing unit 70 may generate a synthesized image by synthesizing the long exposure image and the short exposure image according to the use-image selection information updated by the update unit 60. According to this configuration, since the motion detected by the motion detection unit 40 (for example, a large vibration or a moving object) is reflected in the synthesis process, artifacts generated by the synthesis are reduced.

The gradation compression unit 80 performs compression processing for accommodating a bit range of a video signal having a wide dynamic range in a predetermined bit range on the WDR image generated by the WDR synthesis unit 70. As the compression processing, tone mapping according to a look-up table (LUT) may be used, but any method may be used.

The rear end of the gradation compression unit 80 is connected to an image processing engine including, for example, a demosaic unit for generating an RGB frame from Bayer data, an outline enhancement unit, and color management. Therefore, it is preferable that the data amount of the output signal from the grayscale compression unit 80 is adjusted (for example, about 12 bits) to suit the size of the input data to the image processing engine. Since it is converted into a dark image simply by reducing the data size, it is good if the high luminance side is strongly compressed so as to approximate the human visual characteristics.

In the example described above, an example of photographing and combining two types of images (short exposure image and long exposure image) with different exposure amounts has been described. However, the kind of images used for synthesis is not particularly limited. For example, three or more types of images with different exposure amounts may be captured and synthesized.

More specifically, when using a short exposure image, a medium exposure image, and a long exposure image, a motion detection unit for detecting motion from the short exposure image and the heavy exposure image, and motion detection from the medium exposure image and the long exposure image It is sufficient to provide a motion detection unit for. In addition, an area for writing a heavy exposure image may be provided in the frame memory. The format of the use-image selection information may be a format for selecting a use image from three types of a short exposure image, a medium exposure image, and a long exposure image.

Subsequently, examples of the effects exhibited by the first embodiment of the present invention will be described.

4 to 6 are diagrams for explaining examples of effects exhibited by the first embodiment of the present invention.

Referring to FIG. 4, a short exposure image and a long exposure image are shown. Each video captures a scene from indoors to bright outdoors. In the long exposure image, the window area is completely saturated and the interior is dark. In the short exposure image, clouds outside the window were captured, but the interior was almost black. In such a scene, a case of selecting an image to be used is considered by using the logic of "a short exposure image is used when a pixel value of a long exposure image is equal to or greater than a threshold value".

Fig. 5 shows a state in which the average value of the evaluation region R1 in the long exposure image is calculated by paying attention to the evaluation region R1 surrounded by a dotted line in the long exposure image. The window area is saturated with a 12-bit signal, so the pixel value is 4095, the pixel value is set to 0 for simplicity in the dark room, and the size of the evaluation area R1 of the long exposure image is 16 pixels X. Let it be 16 pixels. Here, when the camera vibrates horizontally, pixel values in the dark room in the evaluation area R1 repeat intrusion and escape.

For example, if two dark pixel values in the room infiltrate the evaluation region R1, the average pixel value of the long exposure image in the region is 3584, which significantly falls. When 4 pixel values are intruded, it becomes a much smaller value as 3072. That is, when the camera vibrates with the amplitude of 4 pixels, the average pixel value of the long exposure image in the corresponding region fluctuates between 4095 and 3072.

When evaluating the pixel value of a long-exposed image near an area with such a high-contrast edge, the evaluation value rises and falls significantly due to slight vibrations of several pixels, near the threshold for determining image selection. There is a case of going up and down. In Fig. 6A, a state in which the pixel value of the long exposure image changes so as to rise or fall near the threshold value is shown. When a use image is selected according to this evaluation value, a variation in which a short exposure image is selected or a long exposure image is selected for a corresponding region occurs in the time direction.

FIG. 6B is a diagram illustrating movement of use-image selection information in the time direction when the use of the short exposure image is defined as 256 and the use of the long exposure image is defined as 0. FIG. According to the first embodiment of the present invention, averaging is performed between the past use-image selection information and the current time use-image selection information.

6C is a diagram showing the movement of averaged use-video selection information in the time direction. Referring to FIG. 6C, since the use-video selection information is averaged, temporal variation of the use-video selection information is suppressed. When the use-image selection information is averaged, since the mixing ratio of the short exposure image and the long exposure image is accommodated in a small range, the difference in quality between the short exposure image and the long exposure image cannot be easily detected.

However, when the use-image selection information is added and averaged in the time direction, the synthesis process for a moving object becomes a problem. In order to reduce the artifacts that occur when synthesizing a moving object, a technique for selecting a short exposure image in the motion area is known. Using a short exposure image selected in the motion area-If the image selection information is added and averaged in the time direction, The exposure images are mixed, resulting in composite artifacts. For this problem, in the first embodiment of the present invention, it is possible to reduce synthesis artifacts of a moving object by adding and averaging the use-image selection information in the time direction and then integrating it with the motion information.

If the entire screen performs a wide range of movements that can be viewed as panning and tilting, the entire screen is detected as movement. In this case, the WDR synthesis process is turned off or the coordinates are converted. It is sufficient to synthesize while matching, and the flicker due to temporal variation in the selection of the short exposure image and the long exposure image, which is noted by the first embodiment of the present invention, will not be noticeable.

However, a slight vibration of a few pixels, sometimes about 2 pixels, is not detected as a motion, and yet in a high contrast area, it causes the use-image selection information to fluctuate in time, which may cause flicker in the composite image. have. In particular, as the number of pixels increases, there is a concern that even minute vibrations are easily photographed and the phenomenon may surface. In the first embodiment of the present invention, caution is required in that it is a technique to reduce the influence of a minute vibration, which is difficult to detect by such motion detection, on a WDR synthesized image.

(Example 2)

Subsequently, a second embodiment of the present invention will be described. In the first embodiment of the present invention, compared to the prior art, it is necessary to preserve and maintain the past use-image selection information in a memory, and an increase in a memory area or an increase in data transmission amount may become an obstacle in terms of hardware. For example, when the use-video selection information format shown in FIG. 6B is used, information as much as "9 bits x image size" is preserved and maintained. In the second embodiment of the present invention, a description will be given of a configuration capable of suppressing an increase in the memory area or data transfer amount while simultaneously exhibiting the same effects exerted by the first embodiment of the present invention.

7 is a diagram showing the configuration and operation of the image processing apparatus 1B according to the second embodiment of the present invention. As shown in Fig. 7, the image processing apparatus 1B of the second embodiment of the present invention has a reduction processing unit 15 at the rear end of the image sensor 10 compared to the image processing apparatus 1A of the first embodiment of the present invention. ). Hereinafter, the reduction processing unit 15 will be mainly described in detail.

The reduction processing unit 15 reduces the long exposure image and the short exposure image. The method of generating the reduced image is not particularly limited, but a set of average values for each rectangular area constituting the image may be generated as a reduced image. Referring to FIG. 7, each reduced image is written in the frame memory 20B as a reduced long exposure image and a reduced short exposure image.

It is preferable that the use image selection unit 30 generates use-image selection information according to the reduced short exposure image and the reduced long exposure image. According to the above configuration, the size of the past use-image selection information generated by the use image selection unit 30 than the case of generating the use-image selection information according to the original short exposure image and the original long exposure image ) Becomes smaller. Accordingly, it is possible to suppress an increase in the amount of data transmission or an increase in the use-image selection information memory 52 in the past.

The motion detection unit 40 may generate motion detection information according to the reduced short exposure image and the reduced long exposure image. According to this configuration, it is possible to suppress an increase in the amount of data transmission.

Of course, the motion detection unit 40 may generate motion detection information according to the original short exposure image and the original long exposure image. In addition, the short exposure image and the long exposure image used by the WDR synthesizing unit 70 may not be reduced. Referring to FIG. 7, the unreduced short exposure image and the long exposure image are written in the frame memory 20B as the original short exposure image and the original long exposure image.

The effect of the second embodiment of the present invention will be described with a specific example. For example, if the reduction ratio is set to 1/10 for each vertical and horizontal, the data amount of the reduced image becomes 1/100 of the data amount of the original image. Therefore, the data amount of the reduced image 2 frames (long exposure image and short exposure image) is only 1/50 (2%) of the data amount of the original image, and the data amount of the past use-image selection information is also It may be 1/100 of the data amount of the use-image selection information generated from the original image. Therefore, compared with the prior art, in the second embodiment of the present invention, it is possible to significantly suppress an increase in the amount of data.

When a reduced image is used for motion detection, it is considered that the spatial resolution of the motion detection information is also 1/100. However, when motion detection is performed with reference to a reduced image, noise is reduced during the reduction process, so that motion detection can be performed using a high-quality image. In addition, since motion detection with reference to a reduced image is equivalent to motion detection with reference to a certain area, there is an advantage of obtaining a spatially robust detection result. Further, even if the spatial resolution of the motion detection information is slightly lowered, it is possible to obtain motion detection information with sufficient precision to apply a motion adaptation process to a motion object having a predetermined size. Therefore, it can be seen that there are few practical disadvantages of this configuration using a reduced image.

In the above, the first and second embodiments of the present invention have been described. According to a first embodiment of the present invention, a used image selection unit 30 for generating use-image selection information by selecting which image to use from a long exposure image and a short exposure image by region, and a past use-image selection A composite image by synthesizing the long-exposure image and the short-exposure image according to the average use-image selection information and the averaged use-image selection information by averaging the information and the current time usage-image selection information An image processing apparatus 1A is provided, including a WDR synthesizing unit 70 that generates a.

According to the above configuration, when the short-exposure image and the long-exposure image are selected to be used for synthesis, the use-image selection information generated by the use image selection unit 30 is not itself but averaged by the averaging processing unit 50 Reference is made to use-picture selection information. Therefore, it is possible to suppress a phenomenon in which the selected image is frequently fluctuated by a minute vibration.

In addition, according to the second embodiment of the present invention, it includes a reduction processing unit 15 for obtaining a reduced long exposure image and a reduced short exposure image by reducing a long exposure image and a short exposure image, and a used image selection unit 30 ) Is provided with an image processing apparatus 1B for generating use-image selection information according to the reduced long exposure image and the reduced short exposure image.

According to the above configuration, use-image selection information is generated according to the reduced short exposure image and the reduced long exposure image, and the generated use-image selection information is transmitted to the averaging processing unit 50. In addition, in the averaging processing unit 50, the averaged use-image selection information is written into the memory as past use-image selection information. Accordingly, it is possible to suppress an increase in the memory area or an increase in the amount of data transfer.

In the field of network cameras where the resolution is increasingly increasing, a technology for temporally stabilizing a composite image is expected to become more important in the future. The embodiment of the present invention is to reduce the phenomenon that flicker occurs in the synthesized image due to minute vibrations, which was difficult to solve in the prior art, and while stabilizing the quality of the synthesized image, it is possible to reduce synthesis artifacts of moving objects or realize a realistic circuit scale. Excellent in that you can.

So far, we have looked at the center of the preferred embodiments for the present invention. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiment should be considered from an illustrative point of view, not a limiting point of view. The scope of the present invention is shown in the claims rather than the above description, and the invention claimed by the claims and the inventions equivalent to the claimed invention should be construed as being included in the invention.

It may also be used in the field of general video synthesis.

1(1A,1B): image processing device, 10: image sensor,
15: reduction processing unit, 20 (20A, 20B): frame memory,
30: use image selection unit, 40: detection unit,
50: averaging processing unit, 51: IIR filter,
52: memory, 60: update unit,
70: WDR synthesis unit, 80: gradation compression unit.

Claims (6)

Information for selecting long-exposure images or short-exposure images to be used for each area of the image, or as information of the blending ratio indicating how much of the long-exposure image and the short-exposure image to be mixed-Use to generate image selection information An image selection unit;
An averaging processing unit for obtaining averaged use-image selection information by averaging past use-image selection information and current time use-image selection information;
A motion detector for detecting motion and obtaining motion detection information;
According to the motion detection information, the averaged use-image selection information is updated in the motion region in which the motion is detected, and the averaged use-image selection information is not updated in the non-motion region in which the motion is not detected. part; And
Including, a synthesis unit for generating a composite image by synthesizing the long exposure image and the short exposure image according to the updated use-image selection information. delete The method according to claim 1, wherein the image processing device,
A reduction processing unit for reducing the long exposure image and the short exposure image to obtain a reduced long exposure image and a reduced short exposure image,
The use image selection unit,
An image processing apparatus for generating the use-image selection information according to the reduced long exposure image and the reduced short exposure image. The method of claim 3, wherein the motion detection unit,
The image processing apparatus for detecting the motion according to the reduced long exposure image and the reduced short exposure image. The method of claim 1, wherein the update unit,
An image processing apparatus for updating the averaged use-image selection information so that a short exposure image is used in a motion region. Information for selecting a long exposure image or a short exposure image to be used for each area of the image, or use as information of a blending ratio indicating the ratio to mix each of the long exposure image and the short exposure image-generating image selection information ;
Obtaining averaged use-image selection information by averaging the past use-image selection information and the current time use-image selection information;
Detecting motion to obtain motion detection information;
Updating the averaged use-image selection information in the motion region in which the motion is detected, and not updating the averaged use-image selection information in the non-motion region in which the motion is not detected according to the motion detection information ; And
Generating a composite image by synthesizing the long exposure image and the short exposure image according to the updated use-image selection information.

Images