WO2015040731A1

WO2015040731A1 - Image processing device and image processing method

Info

Publication number: WO2015040731A1
Application number: PCT/JP2013/075436
Authority: WO
Inventors: 山口　宗明; 佑一郎小宮
Original assignee: 株式会社日立国際電気
Priority date: 2013-09-20
Filing date: 2013-09-20
Publication date: 2015-03-26
Also published as: JPWO2015040731A1

Abstract

A problem with conventional art is that, while it is possible to obtain a blur correction effect, a haze arises in an image. With the present invention, before time smoothing, pixels in the same location are grouped into a plurality of groups using brightness and color difference of the pixels. Frequencies of pixel numbers of the respective groups are computed. If, using the information of the group having the greatest frequency, the frequency exceeds a threshold, the time smoothing is carried out using only the information of the said group.

Description

Image processing apparatus and image processing method

The present invention relates to an image processing apparatus, and more particularly, to an image processing apparatus and an image processing method capable of reducing image degradation due to image fluctuations such as a hot flame.

Fluctuation correction is one of image processing techniques used for monitoring systems and the like. The purpose of fluctuation correction is to reduce the influence of fluctuation received by a monitoring video represented by a hot flame.
In the fluctuation correction technique, for example, the influence of fluctuation is reduced by time smoothing processing.

FIG. 1 is a diagram illustrating an example for explaining the effect of temporal smoothing processing of pixel values on an image.

As shown in the upper side of FIG. 1, the input image to be corrected is input to the image processing apparatus at predetermined time intervals according to a time series flowing from left to right. In FIG. 1, only the

input images

101, 102, and 103 are shown. As shown in FIG. 1, the

input images

101, 102, and 103 all look like the same image. That is, the subject image 402 reflected in the input image 102 and the subject image 403 reflected in the input image 103 are originally moving objects in a car or the like, but the time when the images 101 to 103 were photographed. Are not moved and are at the same pixel position as the subject image 401 reflected in the input image 101. Of course, the

subject images

201 and 301 reflected in the input image 101 are the

subject images

202 and 302 reflected in the input image 102 and the

subject images

203 and 303 reflected in the input image 103. It is a still body that does not move, and is at approximately the same pixel position. However, the stationary moving object images 401 to 403 in the

input images

101, 102, and 103 and the still object images 201 to 203 and 301 to 303 all appear to fluctuate due to the influence of the heat.

The lower smoothed image 104 in FIG. 1 is a smoothed image of n input images including the

input images

101, 102, and 103 (n is a natural number). As shown in the smoothed image 104, the still

object images

204 and 304 and the moving object image 404 of the moving object in the image are corrected for fluctuations due to the influence of the heat.
However, FIG. 1 shows the case of ideal processing conditions. Actually, the influence of fluctuation is not completely eliminated by only the time smoothing process (see Patent Document 1).

FIG. 2 is a diagram for explaining that the contour of the subject image of the image is blurred when only the time smoothing process is performed. The image 111 includes a subject image 211 that is a still life and does not fluctuate. For easy understanding, in the image 111, the subject image 211 is monotone, the luminance value is minimum (black), and the luminance values of other portions are maximum (white). Similarly, in the images 121 to 123, the subject images 221 to 223 are monotone and have the minimum luminance (black) and the other portions have the maximum luminance (white).
The subject image 211 is affected by fluctuations such as a heat wave, and becomes subject images that fluctuate like subject images 221 to 223 in n images 121 to 123. The images 121 to 123 in FIG. 2 are also input to the image processing apparatus at predetermined time intervals in accordance with the time series flowing from left to right, like the input images 101 to 103 in FIG.
When the time smoothing process is performed on the n images 121 to 123, the subject image 211 in the image 111 has a luminance corresponding to the region a2 of the subject image at the center as in the subject image 224 shown in the image 124. Becomes the minimum value (black), and the luminance of the images a1 and a3 at the ends (in this case, left and right) becomes the maximum and minimum intermediate value (gray) and blurs. In addition, the pixel size is larger than the original subject image 211. That is, the subject image 224 appears blurred.

JP 2009-059202 A JP 2010-263552 A

Japanese Patent Laid-Open No. 2004-133867 discusses (1) a plurality of types of model images for a multifunction device having a printer function, a scanner function, and a color copy function, and those constituent pixels are divided into six regions according to hue values. A first feature amount that represents a comprehensive feature of a plurality of types of model images that is classified and specified with a mean value of constituent pixels belonging to each region as a representative value is specified. (2) The constituent pixels of the correction target image are classified into six regions according to the hue value, and the average value of the constituent pixels belonging to each region is specified as the second feature amount representing the feature of the correction target image. (3) A technique is disclosed in which color conversion processing is performed so that each representative value of the second feature value approaches each representative value of the first feature value.
In Patent Document 2, a dark area in an output image is extracted, pixels that are outside the color reproduction range and are in the dark area are classified into one or more clusters, and whether each pixel of the output image is noise or not. In the classified cluster, the feature for the pixel determined to be noise is extracted, whether or not noise is present is determined, and the color conversion method of the cluster that is determined to be noise is present as noise. The color conversion method is different from the color conversion method when it is determined not to be performed, and the color conversion process is performed according to the determined color conversion method, so that the color conversion method is different depending on whether noise is present or not. Has disclosed a technique for determining a mapping direction in which noise is reduced in a corresponding cluster by analyzing the noise included in the cluster and making the noise inconspicuous.
However, Patent Literature 1 and Patent Literature 2 do not disclose that the correction target image is not a moving image, and the subject in the image is classified into a still life and a moving body for correction.
Non-Patent Document 1 discloses a technique for reducing noise by fusing images from a plurality of different sensors with a gradient pyramid. In addition, the fusion by the gradient pyramid is more effective than the fusion by the Laplacian pyramid and can be applied to the video sequence. However, with this technique, noise due to a moving object can be removed from the background, but the moving object itself is also removed. Therefore, it cannot be applied to image processing for monitoring a moving object.
As described above, the conventional fluctuation correction technique has a problem that although the fluctuation correction effect is obtained, the image is blurred.
In view of the above-described problems, an object of the present invention is to provide an image processing apparatus and an image processing method capable of correcting fluctuations due to the influence of a hot flame and the like and reducing image blur.

In order to achieve the above object, an image processing apparatus according to the present invention stores an input image input along a time series, and stores a correction target image and a plurality of images stored along the time series from the stored images. The pixel at the same position is extracted from the image memory for outputting the correction image, the input image or the correction target image, and the plurality of correction images, grouped according to the luminance, and the pixel information of the correction processing corresponding position And a pixel grouping unit for calculating pixel frequency of the grouped group, outputting pixel information indicating the calculated pixel frequency and the position of the group, and a group of maximum frequency using the input pixel frequency A pixel group to be used for the time smoothing process by comparing the pixel frequency with a predetermined first threshold and selecting whether or not to perform time smoothing by all pixels. A pixel group selection unit that outputs pixel information of the selected group, a fluctuation correction unit that corrects fluctuation by temporal smoothing using the pixel information of the selected group, and each element constituting the device It is a first feature of the present invention that includes a control unit for controlling and correcting the input image or the correction target image.

In the image processing device according to the first aspect of the present invention, the pixel grouping unit classifies into a plurality of groups based on the luminance and the first threshold, and further divides into a plurality of groups based on the color difference and the second threshold. Classification is a second feature of the present invention.

In the image processing device according to the first or second aspect of the present invention, the pixel grouping unit groups the luminance value information or luminance value information and color difference information of the grouped group together with the pixel frequency. Outputting as information is a third feature of the present invention.

In order to achieve the above object, the image processing method of the present invention stores an input image input along a time series, and stores the correction target image and the time series from the stored image. The image input step for outputting a plurality of correction images, the pixels at the same position are extracted from the input image or the correction target image, and the plurality of correction images, grouped according to luminance, and the correction processing corresponding position An image grouping step for calculating the pixel information of the group and the pixel frequency of the grouped group, outputting pixel information indicating the calculated pixel frequency and the position of the group, and using the input pixel frequency An image to be used for time smoothing processing by selecting a frequency group, comparing the pixel frequency with a predetermined threshold value, and selecting whether or not to perform time smoothing for all pixels. A group selection step of selecting a group and outputting pixel information of the selected group; and a fluctuation correction step of correcting fluctuation by temporal smoothing using the pixel information of the selected group, the input image or A corrected image is generated from the correction target image.

According to the present invention, it is possible to reduce blurring that occurs in the contour of a subject while correcting fluctuations.

It is a figure which shows an example for demonstrating the effect which a time smoothing process has on an image. It is a figure for demonstrating that the outline of the to-be-photographed object's object blurs only in the time smoothing process. It is a figure which shows an example for demonstrating the effect which the time smoothing of a pixel value has on an image | video. It is a figure which shows an example for demonstrating the effect which the time smoothing of a pixel value has on an image | video. It is a figure for demonstrating one Example of the image processing method of this invention. It is a figure for demonstrating one Example of the image processing method of this invention. It is a block diagram which shows the structure of one Example of the image processing apparatus of this invention. It is a functional block diagram of one Example of the imaging device 1500 which concerns on Example 2 of this invention. It is a functional block diagram of one Example of the monitoring system 1600 which concerns on Example 3 of this invention. It is a functional block diagram of one Example of the code decoding system 1700 which concerns on Example 4 of this invention. It is a flowchart for demonstrating one Example of the image processing method of this invention.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In each drawing, the same code | symbol was provided to the common component.
In addition, this invention is not limited to the Example demonstrated below, Various modifications are included. The following embodiments have been described in detail for the purpose of illustrating the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. Also, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment. Moreover, the structure of another Example can also be added to the structure of a certain Example. Further, with respect to a part of the configuration of each embodiment, another configuration can be added, deleted, or replaced.

The applicant of this patent has applied for a fluctuation correction technique using time smoothing as Japanese Patent Application No. 2012-107470 (hereinafter referred to as “prior application”). The prior application suppresses the influence of fluctuation by temporally smoothing a plurality of frames (hereinafter referred to as “correction images”) that are close in time to the correction target image. At that time, the similarity for each pixel is obtained between the correction target image and the correction image, or between the images obtained by mixing the correction target image and the correction image, and the strength of time smoothing is adjusted according to the result. It is said.

An outline of conventional fluctuation correction will be described with reference to FIGS. 3 and 4. FIG. 3 and 4 are diagrams illustrating an example for explaining the effect of temporal smoothing of pixel values on an image. In this description, for the sake of simplification, as in FIG. 1, description will be made using a monotone image in which the luminance value is divided into the maximum (white) and the luminance value is minimum (black). As in FIG. 1, the input image to be corrected is input to the image processing apparatus at a predetermined time interval according to a time series flowing from left to right.
FIG. 3 is a diagram for explaining the effect of conventional time smoothing processing on an image. The subject in the correction target image is corrected for the influence of fluctuation by time smoothing. However, as shown in FIG. 4, in the process of only the time smoothing, the outline of the subject is blurred due to the influence of fluctuation.
FIG. 3 is a diagram illustrating an example for explaining the effect of temporal smoothing of pixel values on an image. The upper input image 101 in FIG. 3 is the same as the image described in FIG. However, the subject image 401 as a moving body moves like 402 ′ and 403 ′ in the input images 102 ′ and 103 ′.
That is, in the region 800 where the subject of the correction target image is stationary like the subject images 204 ′ and 304 ′ of the smoothed image 104 ′ on the lower side of FIG. 3, distortion due to the influence of air fluctuation is caused by time smoothing. It is corrected. However, if time smoothing is performed on the region 801 where the subject of the correction target image is moving like the subject image 404 ′, not only the displacement of the pixel due to distortion but also the displacement of the pixel due to movement of the subject is smoothed. As a result, blurring occurs around the moving subject. That is, the time smoothing of the pixel value is accompanied by deterioration of the image when a moving subject exists in the image.

The outline of the conventional fluctuation correction will be further described with reference to FIG. For simplicity, FIG. 4 will be described in monotone as in FIG. However, description will be made using an image whose luminance value is represented by a numerical value of 256 gradations of “0 to 255”.
In FIG. 4, the images 141 to 146 fluctuate like images 151 due to the heat, etc., with respect to the images 141 to 146 without fluctuation. Similarly, image 152 is an image with fluctuation with respect to image 142, image 153 is an image with fluctuation with respect to image 143, image 153 is an image with fluctuation with respect to image 143, image 154 is an image with fluctuation with respect to image 144, image 155 Is an image with fluctuation with respect to the image 145, and an image 156 is an image with fluctuation with respect to the image 146. In this way, it is assumed that each pixel value is changed with the passage of time in the non-fluctuating images 141 to 146 like the images 151 to 156 having the fluctuations due to the fluctuations. Here, the cells in the images 140 to 146, 151 to 156, and 257 each indicate one pixel. In addition, the white square pixels of the images 141 to 146 and 151 to 156 have a luminance value of “255”, and the black square pixels have a luminance value of “0”.

In FIG. 4, when time smoothing processing is performed on n temporally continuous images with fluctuations such as the images 151 to 156 with fluctuations, the numerical value “213” shown in each pixel shown in the image 257 is obtained. It is corrected to gray pixels such as “,” “128” and “43”. That is, in the conventional correction process, all pixels are used for time smoothing, and as a result, as shown in FIG. 4, black pixels or white pixels due to fluctuations are added to the original white portion and black portion, resulting in gray.

For each pixel of the six images (images 151 to 156), the number of white pixels or black pixels is examined. Then, the region B1 (pixels in the first and second columns from the left) is a white pixel for all six images, and the region B5 (pixels in the sixth and seventh columns from the left) is a black pixel in all six images. It is. In the region B2 (each pixel in the third column from the left), the black pixel is 1 and the white pixel is 5, so the pixel value after time smoothing is 213 (rounded off after the decimal point), and the region B4 (the fifth column from the left) Each pixel) is 1 for white pixels and 5 for black pixels, so the pixel value after time smoothing is 43 (rounded off after the decimal point), and area B3 (each pixel in the fourth column from the left) is a white pixel Is 3 and the black pixel is 3, the pixel value after time smoothing is 128 (rounded off after the decimal point).
As described above, a pixel having a luminance value of “213” is a pixel in which black is actually mixed although it is actually white. A pixel having a luminance value of “43” is a pixel in which white is mixed slightly although it is actually black. Further, a pixel having a luminance value of “128” is a pixel in which white and black are equally mixed. For this reason, the gray portion is widened and the outline of the subject image is blurred.
In this case, as in b1 to b5 shown in the image 140, a gradation that gradually increases from left to right and from white to black appears. Here, b1 is a region having a luminance value of “255”, b2 is a region having a luminance value of “213”, b3 is a region having a luminance value of “128”, b4 is a region having a luminance value of “43”, and b5 is a luminance. This is an area having a value of “0”. Thus, the image looks blurred in the areas b2 to b4.

An outline of fluctuation correction according to the present invention will be described with reference to FIGS. In this description, for the sake of simplicity, description will be made using an image that is divided into white and black. 5 and 6 are diagrams for explaining an embodiment of the image processing method of the present invention. 5 and 6, as in FIGS. 3 and 4, the input image is input to the image processing apparatus at predetermined time intervals according to a time series flowing from left to right.
In one embodiment of the image processing method of the present invention, as shown in FIGS. 5 and 6, before performing the time smoothing process, the pixels at the same position are divided into a plurality of groups using the luminance and color difference of the pixels. . Then, the frequency of the number of pixels in each divided group is calculated. After that, when the information of the group having the highest frequency is used and the frequency exceeds a predetermined threshold, time smoothing is performed using only the information of the group. Other than that, time smoothing is performed using all pixels.

In the embodiment of FIG. 5, each pixel has a luminance value of “0 (black)” to “255 (white)”, is divided into two groups with “128” as a threshold, and the frequency threshold is “4”. An example will be shown. The images 141 to 146 without fluctuation and the images 251 to 256 with fluctuation in FIG. 5 are the same as the images 141 to 146 and 251 to 256 in FIG.
Of the two groups, the group A is white and the group B is black. The group to which the pixel at the pixel position indicated by each cell belongs is as shown in the diagram shown as grouping.
In each of the pixels in the left to first three regions 511 to 516, the left and second left pixels belong to the group A because all the images 501 to 506 are white. The pixels in the third column from the left belong to group A because the frequency of white is “5” in any of the images 501 to 506.
The pixels in the regions 521 to 526 in the fourth column from the left are “3” for both the black and white frequencies, so they are not in either group.
In each of the pixels in the regions 531 to 536 in the left 5 columns to the left 7 columns, the fifth column from the left belongs to the group B because the frequency of black is “5”. The remaining two columns (regions 6 to 7 on the left) are all black and belong to group B.
As a result, the pixels used for the time averaging can be selected from the pixels 560 that are not subjected to the smoothing process, as in the pixel selection images 551 to 556.

The result of the time averaging process based on the embodiment of FIG. 5 of the present invention will be described with reference to FIG. In FIG. 6, images 141 to 146 and 151 to 156 are the same as those shown in FIGS. Also, the image selection images 551 to 556 in FIG. 6 are the same as those in FIG.
In FIG. 6, the image 647 obtained by smoothing the fluctuating images 151 to 156 will be described.
As described with reference to FIG. 5, first, in each of the pixels in the region G1 in the left first column to the third column, the pixels in the left first column and the left two columns are all white in any of the images 501 to 506. Therefore, the pixel value is “255”. In addition, since the pixel in the third column from the left has a high frequency of white pixels, the time smoothing process is performed using only the white pixels, and the pixel value becomes “255”.
In addition, as described with reference to FIG. 5, the pixels in the region G3 in the sixth to seventh columns on the left are all black in any of the images 501 to 506, and thus the pixel value is “0”. In addition, since the frequency of black pixels is high in the pixels in the fifth column from the left, the time smoothing process is performed using only the black pixels, and the pixel value becomes “0”.
Further, in each pixel in the region G2 in the fourth column from the left, the frequency of the black pixels is equal to the frequency of the white pixels. Therefore, time smoothing is performed using all the pixels, and the pixel value becomes “128”. .
Therefore, in the final correction result image 640 after the input image or an image close in time to the input image is a correction target image and the correction target image is subjected to correction processing, the white pixel region g1 and the black pixel The region g2 in which the pixels excluding the region g3 are gray is only the region in the fourth column from the left, and blurring is reduced.

Next, an embodiment of the fluctuation correction according to the present invention will be described with reference to the image processing apparatus of FIG. FIG. 7 is a block diagram showing the configuration of an embodiment of the image processing apparatus of the present invention.
In the image processing apparatus 70 of FIG. 7, the input image 10 is input to the image grouping unit 1 and the image memory 2 via the input unit 4i. The image memory 2 stores n + 1 input images 10 (n is a natural number of 2 or more). The image memory 2 extracts a desired image from the stored image as a correction target image 12 and n correction images 11, and groups the extracted correction target image 12 and n correction images 11 into an image group. Output to part 1. Here, the correction target image 12 is an input image 10 or an image temporally close to the input image 10. The number of images before the current input image 10 to be used as the correction target image 12 is set in advance by the operator or appropriately selected.
The input image 10 is, for example, a video imaged at a predetermined time interval along a time series by one of imaging devices used in the monitoring system.

The control unit 5 includes elements (image grouping unit 1, image memory 2, image group selection unit 3, fluctuation correction unit 4, input unit 4i, output unit 4o, and operation input unit 5io) constituting the image processing apparatus 70. And each element in the apparatus is controlled according to a processing program stored in a built-in or external storage unit (not shown) connected to the control unit 5.
The operator performs an operation of selecting a correction target image by operating from the operation input unit 5io, an operation of changing a threshold value, and the like.

The pixel grouping unit 1 extracts pixels at the same position from a plurality of images, and groups them by luminance and color difference. That is, the pixel grouping unit 1 uses the input image 10 (not essential) input from the input unit 4 i, the correction target image 12 and the correction image 11 input from the image memory 2, and pixel information on the correction processing target position. 13 is taken out. The pixel information 13 is basically a pixel value for one pixel in a predetermined color space expression format (RGB, YC _b C _r , HSV, etc.) similar to that in the input image 10.
The pixel grouping unit 1 performs grouping using 1 + n pieces of image information 13 extracted for one pixel position, threshold values such as luminance values and color difference values that are fixedly or adaptively set, and classifies each group. The number of pixels (frequency of appearance of pixels in the time domain) is calculated. Then, the pixel grouping unit 1 sequentially outputs group information 14 including the calculated pixel frequency and 1 + n pixel information 13 at the position where the group information is obtained to the image group selection unit 3.
The group information 14 preferably holds not only the pixel frequency in a strict sense but also information indicating which pixel information 13 is classified into which group, or immediately after grouping without counting. The result of

The pixel group selection unit 3 selects the group with the highest frequency using the pixel frequency of each group included in the group information 14 sequentially input for each pixel position. When multiple groups have the same maximum frequency, select them all. Further, the pixel frequency is compared with the threshold value _Tf, and if it is equal to or greater than the threshold value, the pixel information 15 of the selected pixel group is output to the fluctuation correction unit 4, and if it is less than the threshold value, all 1 + n pixel information 15 are output. Output to the fluctuation correction unit 4.

The fluctuation correction unit 4 has a function of correcting fluctuations using only pixel information from the pixel group selection unit 3.
As a fluctuation correction algorithm, a well-known algorithm can be arbitrarily used. However, in this example, since the number of pixel information changes depending on the pixel position, it is necessary to appropriately handle a gap of a pixel that has not been subjected to fluctuation correction. . A method of obtaining a time-smoothed image by simply averaging pixel values at different times can easily cope with a change in the number of pixels.
The fluctuation correction unit 4 sequentially performs fluctuation correction processing for each pixel position to generate a corrected image, and outputs the corrected image generated via the output unit 4o as an output image 20.

The details of the operation of the image grouping unit 1 are as follows.
Pixel grouping is classified based on information of each pixel. Here, the pixel information is the brightness and color of the pixel.
When expressing pixels by luminance and color difference, classification is performed using luminance and color difference. That is, it is classified into a plurality of groups according to luminance, and the groups are further subdivided according to color differences. For example, if four threshold values are used to classify into five groups according to luminance values, and one threshold value is used for each of two pieces of color difference information, the information is classified into 20 groups.
Regarding the definition, since the luminance value is generally important, the classification may be performed using only the luminance.
These threshold values are not fixed values, but are preferably set adaptively for each pixel. For example, for each pixel, 1 + n frames of the average value of the pixel values (can be substituted by the output image fluctuation correcting unit 4) and determine the dispersion, the upper side relative to the average value for each component of the YC _b C _r, the lower Set a threshold on the side. The variance may be common to all screens without being calculated for each pixel, or a value obtained by multiplying the average value by a predetermined coefficient may be substituted. When such an adaptive threshold value is used, grouping may be performed in two ways: whether all components are close to the average value.

Details of the operation of the fluctuation correction unit 4 are as follows.
Pixel information 15 used for fluctuation correction is input from the image group selection unit 3 to the fluctuation correction unit 4. The pixel information 15 is obtained by selecting pixel information used for temporal smoothing from pixel information at the same position in the image and having different times. Therefore, smoothing is performed using the input pixel information 15.
After smoothing, the pixel information at the corresponding position of the image to be output is obtained, so that the fluctuation correction unit 4 outputs it as a corrected image via the output unit 4o.

According to the above embodiment, it is possible to reduce blurring that occurs in the contour of the subject while correcting the fluctuation.
In FIGS. 1 to 6, for the sake of easy understanding, the gray image is used. However, it is obvious that the same can be said for the color image as shown in FIG. 7.
Furthermore, in the above embodiment, the pixels used for grouping and temporal smoothing of images are selected for each pixel, but may be performed for each of a plurality of pixel blocks. Further, not only the frequency in the time domain but also the frequency in time and space can be used. For example, for a pixel at a certain correction processing corresponding position, a histogram (frequency) including a pixel spatially adjacent to the pixel. You may ask for.

FIG. 8 is a functional block diagram of the imaging apparatus 1500 according to the embodiment of the present invention. The imaging device 1500 includes an imaging unit 1501, an image processing device 70, and an image display unit 1502. The imaging unit 1501 is an imaging device that receives light emitted from a subject and converts the received optical image into image data. The image processing apparatus 70 is an image processing apparatus according to the first embodiment, and receives image data captured by the imaging unit 1501 and corrects distortion caused by air fluctuation due to generation of a hot flame or the like. The image display unit 1502 is a device that displays a corrected image output from the image processing apparatus 70.

The image display unit 1502 switches the image to be displayed according to the operation mode. For example, when the mode is 1, a corrected image in which air fluctuation due to the occurrence of a hot flame is reduced is displayed, and when the mode is 0, an uncorrected input image is displayed.

According to the imaging apparatus 1500 according to the second embodiment, there is provided an imaging apparatus that displays to the photographer a corrected image in which air fluctuation due to generation of a hot flame or the like of the entire image including both a stationary region and a moving object is reduced. can do.

FIG. 9 is a functional block diagram of the monitoring system 1600 according to the third embodiment of the present invention. The monitoring system 1600 includes an imaging device 1601, an image processing device 70, a server 1602, and a display device 1603. The imaging device 1601 is an imaging device such as one or more surveillance cameras that capture image data. The image processing apparatus 70 is the image processing apparatus according to the first embodiment, and receives image data captured by the imaging apparatus 1601 and corrects air fluctuation caused by the occurrence of a hot flame or the like. The server 1602 is a computer equipped with the image processing apparatus 70.
The display device 1603 is a device that displays a corrected image output from the image processing device 70.

The imaging device 1601 and the server 1602 and the server 1602 and the display device 1603 can be connected via a network such as the Internet, for example, depending on the physical arrangement between the monitored location and the monitoring operator. it can.

The monitoring system 1600 according to the third embodiment provides a monitoring system that displays to the monitoring operator a corrected image in which air fluctuation due to generation of a heat flame of the entire image including both the stationary region and the moving object is reduced. can do.

FIG. 10 is a functional block diagram of a code decoding system 1700 according to the fourth embodiment of the present invention. The code decoding system 1700 includes an encoding device 1710, a decoder 1721, and a display device 1722. The encoding device 1710 includes an imaging device 1711, an image processing device 70, and an encoder 1712.

The imaging device 1711 is an imaging device such as a surveillance camera that captures image data. An image processing apparatus 1704 is an image processing apparatus according to the first embodiment, and receives image data captured by the imaging apparatus 1711 and corrects the heat. The encoder 1712 encodes the corrected image data output from the image processing apparatus 70 and transmits the encoded image data to the decoder 1721 via the network. The decoder 1721 decodes the corrected image data that has been transmitted. The display device 1722 displays the image decoded by the decoder 1721.

According to the code decoding system 1700 according to the fourth embodiment, it is possible to provide a code decoding system that displays a decoded image in which fluctuation of air due to generation of a hot flame or the like of an entire image including both a stationary region and a moving object is reduced. it can. Furthermore, by reducing air fluctuations due to the occurrence of a hot flame in the image, the difference between images to be transmitted by the encoder 1712 is reduced, and the encoding efficiency is improved.

A fifth embodiment of the present invention will be described with reference to FIG. FIG. 11 is a flowchart for explaining an embodiment of the image processing method of the present invention.
In the process of FIG. 11, for example, the control unit 5 of FIG. 7 controls each element in the apparatus.
The control unit 5 has a processing program for performing image processing of the present invention, and performs image processing based on the processing program. It is assumed that the threshold value of the processing program is set to a predetermined value in advance.

In FIG. 11, first, in an image input step S01, an input image input from the outside in time series is input and stored in the image memory, and the correction target image and the time series are stored from the stored image. A plurality of correction images are output.

Next, in an image grouping step S02, pixels at the same position are extracted from the input image or the correction target image and a plurality of correction images, grouped according to luminance, pixel information at the correction processing corresponding position, and the grouped group. And the pixel information indicating the calculated pixel frequency and the group position is output.
That is, in the image grouping step S02, the pixel information at the position corresponding to the correction process is extracted from the input image that has been input or the stored correction target image and the stored correction image. Then, using the extracted image information, the pixel frequency of each grouped group set in advance using the luminance value of the pixel or the like is calculated. Further, group information including the calculated pixel frequency and pixel information indicating the processing corresponding position (position of the group) are output.
In addition to the pixel frequency, the group information is, for example, luminance value information or luminance values and color differences of the grouped groups.

Next, in the group selection step S03, a group having the maximum frequency is selected using the input pixel frequency, and the pixel frequency is compared with a predetermined threshold value to select whether to perform time smoothing for all pixels. To select a pixel group to be used for the time smoothing process, and output pixel information of the selected group.
That is, in the group selection step S03, the group having the maximum frequency is selected using the pixel frequency of each group included in the input group information. Further, the pixel frequency is compared with the threshold value, and whether or not to perform time smoothing by all pixels is selected, and a pixel group to be used for the time smoothing process is selected. Then, the pixel information of the selected pixel group is output.

Next, in the fluctuation correction step S04, fluctuation correction is performed by time smoothing using the pixel information of the selected group.
That is, in the fluctuation correction step S04, fluctuation correction by time smoothing is performed using the pixel information of the input group. Then, fluctuation correction processing is performed on all pixels in the processing target image to generate a corrected image and output it.

As described above, by generating a corrected image from the input image or the correction target image, it is possible to reduce blurring generated in the contour of the subject while correcting the fluctuation.

In addition, according to the image correction apparatus and the image correction method of the above-described embodiment, in addition to the hot flame, for example, wide corrections are made for, for example, water surface fluctuations, tree vibrations, atmospheric haze, image distortion during underwater photography, etc. It can be widely used for the purpose of improving visibility by stopping an object that is irregularly shaking.

Further, each configuration, function, processing unit, processing means, etc. may be realized by hardware by designing a part or all of them, for example, by an integrated circuit. Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor. Information such as programs, tables, and files for realizing each function can be stored in a recording device such as a memory, a hard disk, an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

1: Image grouping unit, 2: Image memory, 3: Image group selection unit, 4: Fluctuation correction unit, 4i: Input unit, 4o: Output unit, 5: Control unit, 5io: Operation input unit, 10: Input image 11: Image for correction, 12: Image to be corrected, 13: Pixel information, 14: Group information, 15: Pixel information, 20: Output image, 70: Image processing device, 101-103, 102′-104 ′, 111 121 to 123, 140 to 146, input image, 104, 124, 147, 304: smoothed image, 201 to 204, 211, 221 to 224, 241 to 246, 251 to 257, 301 to 304, 401 to 404 402 ′ to 404 ′: Subject image, 501 to 506: Grouped image, 551 to 556: Pixel selection image, 560: Smooth Non-processed pixels, 640: correction result image, 511 to 516, 800, 801: area, 1500: imaging device, 1502: image display unit, 1600: monitoring system, 1601: imaging device, 1602: server, 1603: display device, 1700: Code decoding system, 1710: Encoding device, 1711: Imaging device, 1712: Encoder, 1721: Decoder, 1722: Display device, a1-a3, B1-B5, b1-b5, G1-G3, g1- g3: region.

Claims

An image memory for storing an input image input along a time series, and outputting a correction target image and a plurality of correction images stored along the time series from the stored image;
Pixels at the same position are extracted from the input image or the correction target image and the plurality of correction images, grouped according to luminance, and pixel information of the correction processing corresponding position and pixel frequency of the grouped group are calculated. A pixel grouping unit that outputs pixel information indicating the calculated pixel frequency and the position of the group;
A time group is selected by selecting a maximum frequency group using the input pixel frequency, comparing the pixel frequency with a predetermined first threshold value, and selecting whether to perform time smoothing for all pixels. A pixel group selection unit that selects a pixel group to be used for the smoothing process and outputs pixel information of the selected group;
A fluctuation correction unit that corrects fluctuation by temporal smoothing using the pixel information of the selected group;
A control unit for controlling each element constituting the device;
An image processing apparatus comprising: correcting the input image or the correction target image.
The image processing apparatus according to claim 1, wherein the pixel grouping unit classifies into a plurality of groups according to the luminance and the first threshold, and further classifies into a plurality of groups according to a color difference and a second threshold. A featured image processing apparatus.
3. The image processing apparatus according to claim 1, wherein the pixel grouping unit outputs the luminance value information or luminance value information and color difference information of the grouped group as group information together with the pixel frequency. An image processing apparatus.
An image input step of storing an input image input along a time series, and outputting a correction target image and a plurality of correction images stored along the time series from the stored image;
Pixels at the same position are extracted from the input image or the correction target image and the plurality of correction images, grouped according to luminance, and pixel information of the correction processing corresponding position and pixel frequency of the grouped group are calculated. An image grouping step for outputting pixel information indicating the calculated pixel frequency and the position of the group;
A time smoothing process is performed by selecting a maximum frequency group using the input pixel frequency, comparing the pixel frequency with a predetermined threshold value, and selecting whether to perform time smoothing for all pixels. A group selection step of selecting a pixel group to be used for outputting and outputting pixel information of the selected group;
A fluctuation correction step for correcting fluctuation by temporal smoothing using the pixel information of the selected group;
An image processing method comprising: generating a correction image from the input image or the correction target image.