JP5150862B2 - Image processing apparatus, method, program, and imaging system - Google Patents

Description

  The present invention relates to an image processing apparatus, method, program, and imaging system, and in particular, an image processing apparatus and method for generating a processed image after image processing by processing pixel data constituting a captured image captured by an imaging unit. , A program, and an imaging system.

Conventionally, imaging of an imaging object by a camera has been performed.
However, since an image cannot be captured when there is an obstacle between the camera and the object to be imaged, it has been studied to remove information on the obstacle as noise by image processing.

FIG. 22 is a system configuration example for imaging an imaging object using a conventional camera.
As illustrated in FIG. 22, an image captured by the camera 200 is transmitted using a transmission / transmission device 241 and a transmission / reception device 242. Thereby, for example, the camera 200 is installed in a remote place, and an image from a place where the camera 200 is installed can be viewed from a remote place. On the other hand, if there is an obstacle between the camera 200 and the object to be imaged (for example, a volcano and its surrounding area in the case of a volcano monitoring camera) at the place where the camera 200 is installed, the obstacle Things are reflected together.

FIG. 23 is a diagram illustrating a captured image when there is an obstacle between the camera and the imaging target.
FIG. 23A shows a state in which only the imaging object is copied, and FIG. 23B shows a state in which an obstacle is also copied in addition to the imaging object.

  As shown in FIG. 23, when there is an obstacle between the camera and the object to be imaged, such as during snowfall, the object to be imaged may be difficult to see due to the obstacle between them. In the worst case, the imaging object cannot be imaged.

  Therefore, an image processing apparatus that removes noise by performing a smoothing process on a captured image has been proposed (see, for example, Patent Document 1).

JP-A-5-136998

  However, according to the smoothing process, when the noise is the majority of the image, the smoothing process is performed so that it approaches the noise data side. There is a problem of disappearing.

FIG. 24 is a diagram showing a state when an image showing an obstacle is smoothed.
FIG. 24A is an example of an image before the smoothing process, FIG. 24B is an example of an image in a state where the smoothing process is started, and FIG. It is an example of an image which shows a state.

  As shown in FIG. 24, even if an image showing an obstacle is smoothed, an image showing only the imaging object excluding the obstacle cannot be obtained.

FIG. 25 is a diagram illustrating a state when the smoothing process is performed when the disturbance is small, and FIG. 26 is a diagram illustrating a state when the smoothing process is performed when the disturbance is large.
FIG. 25A is an image example before the smoothing process when the disturbance is small, and FIG. 25B is an image example after the smoothing process when the disturbance is small. FIG. 26A shows an example of an image before smoothing when there are many disturbances. FIG. 26B shows an example of an image after smoothing when there are many disturbances.

  As shown in FIG. 25 and FIG. 26, the smoothing process can be an effective noise removing means when removing small noises in the image. However, as shown in FIG. 25 and FIG. When the relative size of one obstacle is large relative to the size of the entire image, the area around the area where the snow is reflected is smoothed to approach the value of the image data of the obstacle that should be judged as noise. Therefore, the obstacle cannot be determined as noise and removed by the smoothing process regardless of whether there are many disturbances.

  Therefore, instead of smoothing processing, the input image that is input one by one from the imaging means such as a camera is weighted by a constant coefficient, or is thinned out by a fixed frame and added to the accumulated image data. It is possible to let them.

FIG. 27 is a diagram illustrating a state of the follow-up process in which the input image is made to follow the accumulated image.
FIG. 27A is an image example showing an image before the tracking process, FIG. 27B is an image example when the tracking process is started, and FIG. It is an image example at that time.

  When there are many disturbances, as shown in FIG. 27, the object to be imaged is blurred. Specifically, if the disturbance is snow, and if it is snowing in large quantities, the pixel data showing the disturbance snow is also uniformly weighted with a constant coefficient, or the pixel data is taken every other frame. Since the pixel data is added, the pixel data of the pixel in which the disturbance is reflected is continuously added at a constant rate, and is always affected by the value of the pixel data due to the disturbance at the constant rate.

FIG. 28 is a diagram illustrating a state in which the follow-up process is performed when there is little disturbance.
FIG. 28A is an image example showing an input image before the tracking process, FIG. 28B is an image example showing an initial image before the tracking process, and FIG. FIG. 28D shows an image example when the follow-up process has progressed.

  FIG. 28 shows a case where the disturbance is snow. When the disturbance is snow, since the snow appears white, the initial image is set to white (the brightness is maximum). Since the snow, which is a disturbance, appears white due to the reflection of light, the brightness is often higher than the background. Therefore, it is possible to determine that a portion having a higher luminance than the stored processed image is highly likely to be a portion due to disturbance. Therefore, by setting the initial value to the maximum luminance, the picture is gradually made under the influence of the pixel data of the input image.

FIG. 29 is a diagram illustrating a state in which the follow-up process is performed when there are many disturbances.
FIG. 29A is an image example showing the input image before the follow-up process, FIG. 29B is an image example showing the initial image before the follow-up process, and FIG. FIG. 29D shows an image example when the tracking process has progressed.

  As in FIG. 28, the disturbance in FIG. 29 is also snow. When there is a lot of disturbance, that is, when a lot of snow is falling, focusing on a certain point in the image data (for example, pixel data extracted from one pixel), a large amount of pixel data due to the appearance of snow is extracted.

  Therefore, if the pixel data is added at a constant rate, a lot of pixel data due to snow is added. In other words, when the disturbance is snow, the snow appears white, so if you make a picture by combining the pixel data for all pixels, the entire image will be affected by the pixel data due to snow, and the entire image will become whitish. is there.

  In FIG. 29, a case where a large amount of snow is falling is described as an example of an ideal environment where the imaging environment such as brightness is not changed, assuming that the background of the imaging target is a large disturbance. However, when the imaging background is actually captured. The imaging background itself and the imaging environment itself change greatly.

  Specifically, when applied to a surveillance camera that captures an outdoor surveillance object that is expected to generate a lot of disturbance, the background itself, such as the sun going in and out of the slope of the sun and clouds, and the imaging environment Changes significantly. In such a case, it is the same as when there are many disturbances.

  The present invention has been made in view of the above points, and an object of the present invention is to provide an image processing apparatus that can image an imaging object without being affected by disturbance as much as possible.

  In the present invention, in order to solve the above-described problem, in the image processing apparatus that generates the processed image after the image processing by processing the pixel data constituting the captured image captured by the imaging unit, the comparison target in the processed image There is a possibility that the pixel data is a result of imaging a disturbance rather than an imaging target by comparing the value of the pixel data of the range and the value of the pixel data of the comparison target range in the input image input from the imaging means. Judgment of the disturbance judgment processing with respect to the disturbance judgment means for judging whether the value is high, the value of the pixel data of the comparison target range in the input image, and the value of the pixel data of the comparison target range in the processed image Provided with an image processing apparatus comprising: an image processing unit that performs processing for generating a new post-processing image by weighting and adding based on content It is.

  Thereby, the disturbance determination unit compares the value of the pixel data in the comparison target range in the processed image with the value of the pixel data in the comparison target range in the input image input from the imaging unit, and the imaging target Instead, it is determined whether or not the pixel data is a result of imaging the disturbance, and the image processing means determines the value of the pixel data in the comparison target range in the input image and the comparison target range in the processed image. A process for generating a new post-process image is performed by weighting and adding to the pixel data value based on the determination contents of the disturbance determination process.

  Further, according to the present invention, in the imaging system that transmits the captured image captured by the imaging unit using a transmission device that transmits the captured image and displays the captured image at a location separated from the installation location of the imaging unit, the captured image is configured. Comparing the value of the pixel data in the comparison target range in the processed image after the image processing by processing the pixel data to be compared with the value of the pixel data in the comparison target range in the input image input from the imaging means, Disturbance determining means for determining whether or not there is a high possibility that the pixel data is a result of imaging a disturbance rather than an imaging target, a value of pixel data in a comparison target range in the input image, and a comparison in the processed image A new post-processing image is generated by weighting and adding to the pixel data value in the target range based on the determination contents of the disturbance determination processing. Imaging system comprising the image processing apparatus and an image processing means for performing management is provided.

  Thereby, the disturbance determination unit compares the value of the pixel data in the comparison target range in the processed image with the value of the pixel data in the comparison target range in the input image input from the imaging unit, and the imaging target Instead, it is determined whether or not the pixel data is a result of imaging the disturbance, and the image processing means determines the value of the pixel data in the comparison target range in the input image and the comparison target range in the processed image. A process for generating a new post-process image is performed by weighting and adding to the pixel data value based on the determination contents of the disturbance determination process.

  According to the present invention, in the image processing method for generating a processed image after image processing by processing pixel data constituting a captured image captured by the imaging unit, the disturbance determination unit includes a comparison target in the processed image. There is a possibility that the pixel data is a result of imaging a disturbance rather than an imaging target by comparing the value of the pixel data of the range and the value of the pixel data of the comparison target range in the input image input from the imaging means. A step of determining whether or not the image data is high, and the image processing means determines the disturbance for the value of the pixel data in the comparison target range in the input image and the value of the pixel data in the comparison target range in the processed image. And a step of performing a process of generating a new post-process image by weighting and adding based on the determination contents of the process. It is provided.

  Thereby, the disturbance determination unit compares the value of the pixel data in the comparison target range in the processed image with the value of the pixel data in the comparison target range in the input image input from the imaging unit, and the imaging target Instead, it is determined whether or not the pixel data is a result of imaging the disturbance, and the image processing means determines the value of the pixel data in the comparison target range in the input image and the comparison target range in the processed image. A process for generating a new post-process image is performed by weighting and adding to the pixel data value based on the determination contents of the disturbance determination process.

  According to the present invention, in an image processing program for generating a processed image after image processing by processing pixel data constituting a captured image captured by an imaging unit, a computer is used for the comparison target range in the processed image. Is it highly likely that the pixel data is the result of imaging a disturbance rather than an imaging target by comparing the value of the pixel data with the value of the pixel data in the comparison target range in the input image input from the imaging means? Disturbance determination means for determining whether or not the pixel data value of the comparison target range in the input image and the pixel data value of the comparison target range in the processed image are based on the determination content of the disturbance determination processing Image processing means for functioning as image processing means for performing processing for generating a new post-processing image by weighting and adding Program is provided.

  Thereby, the disturbance determination unit compares the value of the pixel data in the comparison target range in the processed image with the value of the pixel data in the comparison target range in the input image input from the imaging unit, and the imaging target Instead, it is determined whether or not the pixel data is a result of imaging the disturbance, and the image processing means determines the value of the pixel data in the comparison target range in the input image and the comparison target range in the processed image. A process for generating a new post-process image is performed by weighting and adding to the pixel data value based on the determination contents of the disturbance determination process.

  According to the image processing apparatus, method, program, and imaging system of the present invention, the disturbance determination unit determines whether the pixel data in the comparison target range in the input image is pixel data obtained as a result of imaging the disturbance, and performs image processing. Since the means sets the weighting according to the determination made by the disturbance determination means to the pixel data of the input image and the processed image and adds the pixel data to generate a new processed image, the data determined to be a disturbance is almost It is possible to generate a new processed image by reflecting a large amount of data that is not reflected and is not determined to be a disturbance, that is, that is determined to have captured the imaging target. Therefore, it is possible to image the imaging object without being affected by disturbance as much as possible.

It is a block diagram of the image processing apparatus which concerns on this Embodiment. 1 is a first system configuration example to which an image processing apparatus according to the present embodiment is applied. It is the 2nd system configuration example to which the image processing device concerning this embodiment is applied. It is the 3rd system configuration example to which the image processing device concerning this embodiment is applied. It is the 4th example of system configuration to which the image processing device concerning this embodiment is applied. It is a block diagram which shows an example of the processing function of a disturbance judgment process part. It is a block diagram which shows an example of the processing function of a disturbance judgment process part. It is a block diagram which shows an example of the processing function of a disturbance judgment process part. It is a flowchart which shows the procedure of the image processing of the random noise by an image processing apparatus. It is a flowchart which shows the procedure of the image processing of the random noise by an image processing apparatus. It is a flowchart which shows the procedure of an image process when the snow by an image processing apparatus is disturbance. It is a flowchart which shows the procedure of an image process when the snow by an image processing apparatus is disturbance. It is a figure which shows the table | surface about a tracking coefficient when the brightness | luminance of disturbance is high. It is a figure which shows a mode when image processing is performed by the image processing apparatus of this Embodiment. It is a figure which shows a mode when image processing is performed with the image processing apparatus of this Embodiment when there are few disturbances. It is a figure which shows a mode when image processing is performed in the image processing apparatus of this Embodiment when there are many disturbances. It is a block diagram of the image processing apparatus which concerns on this Embodiment. It is a block diagram concerning the error measurement process of an error measurement part. It is a flowchart which shows the procedure of the image process including the error measurement by the image processing apparatus, and a tracking count update process. It is a flowchart which shows the procedure of the image process including the error measurement by the image processing apparatus, and a tracking count update process. It is a flowchart which shows the procedure of the image process including the error measurement by the image processing apparatus, and a tracking count update process. It is an example of a system configuration for imaging an imaging object using a conventional camera. It is a figure which shows a captured image when there exists an obstruction between a camera and an imaging target object. It is a figure which shows a mode when the image in which the obstacle is reflected is smoothed. It is a figure which shows a mode when a smoothing process is performed when there are few disturbances. It is a figure which shows a mode when a smoothing process is performed when there are many disturbances. It is a figure which shows the mode of the tracking process which makes an input image track the accumulated image. It is a figure which shows a mode when a tracking process is performed when there are few disturbances. It is a figure which shows a mode when a tracking process is performed when there are many disturbances.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[First Embodiment]
FIG. 1 is a block diagram of an image processing apparatus according to this embodiment.
As shown in FIG. 1, a camera 210, an operation unit 220, and a display unit 230 are attached to the image processing apparatus 100. The image processing apparatus 100 includes an input image memory 110, a disturbance determination processing unit 120, an image. A processing unit 130, a post-processing image memory 140, and a tracking coefficient memory 150 are provided.

  When the captured image captured by the camera 200 is input, the input image memory 110 stores the input image.

  The disturbance determination processing unit 120 reads the input image from the input image memory 110 and reads the processed image from the processed image memory 140. Then, the comparison target range of the input image is compared with the value of each pixel data of the portion corresponding to the comparison target range in the processed image. For example, when the comparison target range is set for each pixel, the pixel data values of the same coordinates of the images are compared with each other. When the compared value has a certain deviation or more, it is determined that the pixel data is pixel data obtained by copying a disturbance. This disturbance determination is performed for all pixel data constituting the image. The processed image will be described later.

  The image processing unit 130 reads the tracking coefficient from the tracking coefficient memory 150 according to the determination by the disturbance determination processing unit 120, and converts it into the pixel data of the comparison target range of the input image and the pixel data of the comparison target range of the processed image. On the other hand, weighting using a tracking coefficient is performed, and image processing for adding the weighted pixel data is performed. This image processing is performed for all pixel data constituting the image. That is, a new post-processed image can be obtained by adding (picture making) the images that have been subjected to image processing for all pixel data. The new processed image is stored in the processed image memory 140. Note that the tracking coefficient stored in the tracking coefficient memory 150 is input and stored from the operation unit 220 such as a keyboard.

  The display unit 230 reads and displays the processed image stored in the processed image memory 140. As a result, it is possible to see an image of the imaging target that has been processed, that is, the disturbance has been removed.

FIG. 2 is a first system configuration example to which the image processing apparatus according to the present embodiment is applied.
As illustrated in FIG. 2, when an image captured by the camera 200 is transmitted using the transmission / transmission device 241 and the transmission / reception device 242, the image processing device 100 is installed between the transmission / reception device 242 and the display unit 230. be able to. In other words, the image processing apparatus 100 performs image processing on the transmitted image (input image) before processing so that an image obtained by imaging the imaging object from which disturbance has been removed can be viewed. According to such a system configuration, an image from which disturbance has been removed can be viewed with an inexpensive camera, and application to an existing system is easy.

FIG. 3 is a second system configuration example to which the image processing apparatus according to the present embodiment is applied.
As shown in FIG. 3, when an image captured by the camera 200 is transmitted using the transmission / transmission device 241 and the transmission / reception device 242, the image processing device 100 is installed between the camera 200 and the transmission / transmission device 241. You can also. That is, the transmission transmission device 241 and the transmission reception device 242 capture an image obtained by capturing an imaging target from which a disturbance has been removed by performing image processing on the unprocessed image (input image) captured by the camera 200. Therefore, it is possible to view an image obtained by imaging the imaging object from which the disturbance is removed. According to such a system configuration, an image from which disturbance is removed can be transmitted. That is, the transmission amount can be reduced.

FIG. 4 is a third system configuration example to which the image processing apparatus according to the present embodiment is applied.
As shown in FIG. 4, there are a plurality of systems that transmit images captured by the camera 200 using the transmission / transmission device 241 and the transmission / reception device 242. The image processing apparatus 100 can be installed between the display unit 230 and the display unit 230. In other words, the image processing apparatus 100 performs image processing on an unprocessed image (input image) transmitted from the system selected by the selector 250 to view an image obtained by capturing an imaging target from which disturbance has been removed. Will be able to. According to such a system configuration, it is possible to view an image from which disturbance is removed by using one image processing apparatus for a plurality of transmission systems.

FIG. 5 shows a fourth system configuration example to which the image processing apparatus according to this embodiment is applied.
As shown in FIG. 5, when an image captured by the camera 200 transmitted using the transmission transmission device 241 and the transmission reception device 242 is recorded in the recorder 260 and the recorded image is displayed, the recorder 260 and the display unit are displayed. 230, the image processing apparatus 100 can be installed. In other words, the image processing apparatus 100 can perform image processing on the pre-processed image (input image) recorded in the recorder 260 so that an image obtained by imaging the imaging object from which disturbance has been removed can be viewed. It goes without saying that the processed image may be recorded in the recorder 260 by installing the image processing apparatus 100 upstream of the recorder 260. In this way, it is natural that not only the real-time video from the camera but also an image obtained by removing disturbance from the video recorded on the recorder can be viewed.

FIG. 6 is a block diagram illustrating an example of processing functions of the disturbance determination processing unit.
As shown in FIG. 6, the disturbance determination processing unit 120 includes a difference calculation unit 121 and a magnitude comparison unit 122. The difference calculation unit 121 calculates the absolute value of the difference between the pixel data value in the comparison target range of the input image and the pixel data value in the comparison target range of the processed image.

  The magnitude comparison unit 122 determines whether or not the difference value calculated by the difference calculation unit 121 is greater than a value that is a predetermined criterion. If the difference value is large, the pixel data in the comparison target range of the input image is disturbance. It is determined that there is a high suspicion of pixel data, and the result is output to the image processing unit 130.

FIG. 7 is a block diagram illustrating an example of processing functions of the disturbance determination processing unit.
If the disturbance is white, such as rain or snow, compared to the object to be imaged or the imaging background, it is sufficient to determine that the white is more likely to be a disturbance.
That is, as shown in FIG. 7, the disturbance determination processing unit 120 only needs to include a luminance extraction unit 123 and a magnitude comparison unit 124.

  Specifically, the luminance extraction unit 123 extracts luminance information from the pixel data in the comparison target range of the input image and the pixel data in the comparison target range of the processed image. Then, the size comparison unit 124 compares the luminance information extracted by the luminance extraction unit 123, and determines that the luminance extracted from the pixel data in the comparison target range of the input image is large, the pixel in the comparison target range of the input image The data is judged to be highly likely to be pixel data due to disturbance, and the result is output to the image processing unit 130.

FIG. 8 is a block diagram illustrating an example of processing functions of the disturbance determination processing unit.
Not only luminance but also saturation may be used as a criterion.

  At this time, as shown in FIG. 8, the disturbance determination processing unit 120 includes a saturation extraction unit 125 and a magnitude comparison unit 126 in addition to the luminance extraction unit 123 and the magnitude comparison unit 124. An adder 127 is provided.

  The saturation extraction unit 125 extracts each saturation information from the pixel data of the comparison target range of the input image and the pixel data of the comparison target range of the processed image. The magnitude comparison unit 126 compares the saturation information extracted by the saturation extraction unit 125 and determines that the saturation extracted from the pixel data in the comparison target range of the processed image is larger. It is determined that the pixel data in the comparison target range is highly likely to be pixel data due to disturbance, and the result is output to the adder 127. When the adder 127 determines that both the magnitude comparison units 124 and 126 are highly likely to be pixel data due to disturbance, the adder 127 determines that the pixel data in the comparison target range of the input image is highly likely to be pixel data due to disturbance. Then, the result is output to the image processing unit 130.

  9 and 10 are flowcharts showing the procedure of random noise image processing by the image processing apparatus. Hereinafter, the processing illustrated in FIGS. 9 and 10 will be described in order of step number.

  [Step S11] When the power is turned on, the image processing apparatus 100 initializes the processed image in the processed image memory 140. Specifically, a gray (intermediate gradation) image is stored in the processed image memory 140. In the case of random noise, since it is not known whether the disturbance appears black or white, a gray image is used as the initial image.

  Further, the follow-up coefficient α1 is set to 1.0 by reading from a previously stored memory or the like. Further, the tracking coefficient α2 is set by an input signal from the operation unit 220. The follow-up coefficient α2 is appropriately set in the range of 1.0 ≧ α2> 0.0 depending on the amount of disturbance. Then, the threshold value TH for the difference between the absolute values of the luminance differences, which is a criterion for determining whether or not there is a disturbance, is set to 16.

  [Step S12] The disturbance determination processing unit 120 reads an input image input from the camera 210 and stored in the input image memory 110.

  [Step S13] The disturbance determination processing unit 120 acquires the width and height of the read input image, and sets the height to H and the width to W.

  [Step S14] The disturbance determination processing unit 120 initializes a loop variable in the height direction. That is, 0 is set in the loop variable Y.

  [Step S15] The disturbance determination processing unit 120 determines whether or not the loop variable Y is larger than the height H of the image. If Y is greater than H, the process proceeds to step S16. If Y is equal to or less than H, the process proceeds to step S25.

  [Step S16] The disturbance determination processing unit 120 initializes a loop variable in the width direction. That is, 0 is set in the loop variable X.

  [Step S17] The disturbance determination processing unit 120 determines whether or not the loop variable X is larger than the width W of the image. If X is greater than W, the process proceeds to step S18. If X is equal to or less than W, the process proceeds to step S23.

[Step S18] The disturbance determination processing unit 120 reads out values of pixels (hereinafter, referred to as (X, Y) pixels) corresponding to the loop variable X and the loop variable Y of the input image and the processed image, and the luminance difference. The absolute value of is calculated. Then, it is determined whether or not the calculated absolute value is greater than a threshold value TH. If the difference absolute value is larger than the threshold value TH, it is determined that the compared pixel is highly likely to be due to disturbance, and a message to that effect is output to the image processing unit 130, and the process proceeds to step S19. If the threshold value TH is less than or equal to the absolute difference value, the process proceeds to step S20. In the case of a color image, the luminance is calculated by the following formula 1.
Luminance = 0.257R + 0.504G + 0.098B + 16 (Equation 1)

[Step S19] The image processing unit 130 weights and adds the (X, Y) pixels of the processed image using the tracking coefficient α2, thereby obtaining (X, Y) pixels of a new processed image. . This weighting is performed as shown in Equation 2 below. In the case of a color image, the process is performed for each of RGB.
α2 × (X, Y) pixel value of input image + (1−α2) × (X, Y) pixel value of processed image ………………………………………… …………………………………………… (Formula 2)

[Step S20] The image processing unit 130 weights and adds the (X, Y) pixels of the processed image using the tracking coefficient α1, thereby obtaining (X, Y) pixels of a new processed image. . This weighting is performed according to Equation 3 below. In the case of a color image, the process is performed for each of RGB.
α1 × (X, Y) pixel value of the input image + (1−α1) × (X, Y) pixel value of the processed image ………………………………………… …………………………………………… (Formula 3)

  [Step S21] Add 1 to the loop variable X.

  [Step S22] The process returns to step S17.

  [Step S23] One is added to the loop variable Y.

  [Step S24] The process returns to step S15.

  [Step S25] The process returns to step S12.

  FIGS. 11 and 12 are flowcharts showing the procedure of image processing when snow by the image processing apparatus is a disturbance. In the following, the process shown in FIGS. 11 and 12 will be described in order of step number.

  [Step S31] When the power is turned on, the image processing apparatus 100 initializes the processed image in the processed image memory 140. Specifically, a white image is stored in the processed image memory 140. Further, the follow-up coefficient α1 is set to 1.0 by reading from a previously stored memory or the like. Further, the tracking coefficient α2 is set by an input signal from the operation unit 220. The follow-up coefficient α2 is appropriately set in the range of 1.0 ≧ α2> 0.0 depending on the amount of disturbance.

  [Step S32] The disturbance determination processing unit 120 reads an input image input from the camera 210 and stored in the input image memory 110.

  [Step S33] The disturbance determination processing unit 120 acquires the width and height of the read input image, and sets the height to H and the width to W.

  [Step S34] The disturbance determination processing unit 120 initializes a loop variable in the height direction. That is, 0 is set in the loop variable Y.

  [Step S35] The disturbance determination processing unit 120 determines whether or not the loop variable Y is larger than the height H of the image. If Y is greater than H, the process proceeds to step S36. If Y is equal to or less than H, the process proceeds to step S45.

  [Step S36] The disturbance determination processing unit 120 initializes a loop variable in the width direction. That is, 0 is set in the loop variable X.

  [Step S37] The disturbance determination processing unit 120 determines whether or not the loop variable X is larger than the width W of the image. If X is greater than W, the process proceeds to step S38. If X is equal to or less than W, the process proceeds to step S43.

[Step S38] The disturbance determination processing unit 120 reads the values of (X, Y) pixels of the input image and the processed image, and compares the luminance and saturation values of each pixel. The luminance of the (X, Y) pixel of the input image is larger than the (X, Y) pixel of the processed image, and the color of the (X, Y) pixel of the processed image is larger than the (X, Y) pixel of the input image. If the degree is high, it is determined that the compared pixel is highly likely to be due to disturbance, and a message to that effect is output to the image processing unit 130, and the process proceeds to step S39. Otherwise, the process proceeds to step S40. Proceed. The saturation is calculated by the following equation 4.
M = MAX (R, G, B)
m = MIN (R, G, B)
Saturation = M−m / M (M = m ≠ 0), 0 (M = m = 0) ………………………… (Formula 4)

  [Step S39] The image processing unit 130 weights and adds the (X, Y) pixels of the processed image using the tracking coefficient α2, thereby obtaining (X, Y) pixels of a new processed image. . This weighting is performed as described in Equation 2 above. In the case of a color image, it is performed for each of RGB.

  [Step S40] The image processing unit 130 weights and adds the (X, Y) pixels of the processed image using the tracking coefficient α1, thereby obtaining (X, Y) pixels of a new processed image. . This weighting is performed as described in Equation 3 above. In the case of a color image, it is performed for each of RGB.

  [Step S41] 1 is added to the loop variable X.

  [Step S42] The process returns to step S37.

  [Step S43] One is added to the loop variable Y.

  [Step S44] The process returns to step S35.

  [Step S45] The process returns to step S32.

FIG. 13 is a diagram illustrating a table of the tracking coefficient when the luminance of the disturbance is high.
When the disturbance is snow, that is, when the luminance is relatively high, it is often sufficient to determine that the luminance is a disturbance by comparing the luminance values of the input image and the processed image as described above. . However, when the brightness of the background is higher than that of the snow, which is a disturbance, the brightness of the input image and that of the processed image are compared. Many are reflected (that is, it appears darker than the original background brightness).

  Therefore, for example, when the brightness of snow is 200 at 256 gradations and the brightness of the background is 230, a brightness value 160 is set, and when the input image is 160 or more which is the set value, it is uniform. The tracking coefficient α2 may be set as follows. In other words, if a luminance of a certain value or more is input, the luminance of snow is reflected more in the processed image by reflecting it less in the processed image as in the case where it is determined that the suspicion of disturbance is high. Can be prevented.

  Note that when there is a lot of disturbance (when the amount of snowfall is large in this case), it is more effective to add a contrast enhancement process after the follow-up process.

FIG. 14 is a diagram illustrating a state when image processing is performed by the image processing apparatus according to the present embodiment.
FIG. 14A is an image example before image processing, FIG. 14B is an image example in a state where image processing has started, and FIG. 14C shows a state in which image processing has progressed. It is an example of an image shown.

  As shown in FIG. 14, when an image showing an obstacle is image-processed by the image processing apparatus 100, a range (pixel) that is highly likely to be a disturbance is multiplied by a small tracking coefficient, and a range that is less likely to be a disturbance ( Pixels) are multiplied by a large follow-up coefficient and added to the processed image so far, so that a new processed image is obtained. Therefore, a pixel having a high suspicion of disturbance is added to the new processed image. Since reflection is reduced, an image having no influence of disturbance can be obtained as a new processed image obtained as a result.

FIG. 15 is a diagram illustrating a state when image processing is performed when there is little disturbance in the image processing apparatus according to the present embodiment.
FIG. 15A is an image example showing an input image before image processing, FIG. 15B is an image example showing an initial image before image processing, and FIG. FIG. 15D shows an image example when image processing has advanced.

  FIG. 15 shows a case where the disturbance is snow. When the disturbance is snow, since the snow appears white, the initial image is made white. Then, by performing image processing according to the present embodiment, an image from which disturbance has been removed can be obtained.

FIG. 16 is a diagram illustrating a state in which image processing is performed when there are many disturbances in the image processing apparatus according to the present embodiment.
16A is an image example showing an input image before image processing, FIG. 16B is an image example showing an initial image before image processing, and FIG. 16C shows image processing. FIG. 16D shows an image example when image processing has advanced.

  As in FIG. 15, the disturbance in FIG. 16 is also snow. When there is a lot of disturbance, that is, when a lot of snow is falling, focusing on a certain point in the image data (for example, pixel data extracted from one pixel), a large amount of pixel data due to the appearance of snow is extracted.

  However, since the disturbance determination processing unit 120 in the image processing apparatus 100 determines whether or not the pixel data has a high suspicion of the pixel data due to the disturbance, unlike the case of following the entire image, the disturbance determination An image having no influence can be obtained.

  By performing the above processing, it is determined whether the pixel data in the comparison target range in the input image is pixel data obtained as a result of imaging the disturbance, and weighting according to the determination in the disturbance determination processing is performed on the input image and the processed image. Since pixel data is added and the pixel data is added to generate a new processed image, the data determined to be a disturbance is hardly reflected and it is determined that the disturbance is not determined, that is, the imaging object is imaged. A new post-processing image can be generated by reflecting a large amount of data. Therefore, it is possible to image the imaging object without being affected by disturbance as much as possible.

  In the present embodiment, when pixels are compared with each other and there is a difference in luminance or saturation exceeding a certain level, or when the luminance of the pixel of the input image is higher, the pixel is a disturbance. As a matter of judgment, when the camera is panned or tilted, naturally, a difference in luminance and saturation exceeding a certain range is produced in many pixels.

  In this case, for example, when it is determined that a difference of 16 gradations or more has occurred in 256 gradations in 80% of the total number of pixels, it is determined that the shooting direction of the camera has changed, and the processed image It is also possible to discard the processed image stored in the memory and use the input image as a new processed image as it is.

  According to this, it is possible to follow the image in real time even when the camera is moved, as well as when the background is moving, and the disturbance removal process is performed again from the replaced image. It is possible to image the imaging object without being affected as much as possible.

  In the present embodiment, it has been described that the tracking coefficient α2 is appropriately set from the operation unit 220 when it is determined whether or not the pixel data is highly likely to be pixel data due to disturbance. You may add the process part which judges whether a grade is suitable.

  Hereinafter, as a second embodiment, an example in which a processing unit that determines how appropriate the tracking coefficient α2 is added will be described.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. The image processing apparatus of the present embodiment is the same as the configuration shown in the first embodiment, except that it includes a tracking coefficient update processing unit that updates the tracking coefficient. For this reason, about the component similar to the said 1st Embodiment, the same code | symbol is attached | subjected and the description is abbreviate | omitted suitably.

FIG. 17 is a block diagram of the image processing apparatus according to the present embodiment.
As shown in FIG. 17, a camera 210, an operation unit 220, and a display unit 230 are attached to the image processing device 2100. The image processing device 2100 includes an input image memory 110, a disturbance determination processing unit 120, and an image. A processing unit 130, a post-processing image memory 140, an image quality reference setting memory 2150, an error measurement unit 2160, and a tracking coefficient update processing unit 2170 are provided.

  When the captured image captured by the camera 200 is input, the input image memory 110 stores the input image.

  The disturbance determination processing unit 120 reads the input image from the input image memory 110 and reads the processed image from the processed image memory 140. Then, the comparison target range of the input image is compared with the value of each pixel data of the portion corresponding to the comparison target range in the processed image. When the compared value has a certain deviation or more, it is determined that the pixel data is pixel data obtained by copying a disturbance. This disturbance determination is performed for all pixel data constituting the image.

  The image processing unit 130 reads out the tracking coefficient from the tracking coefficient update processing unit 2170 according to the determination by the disturbance determination processing unit 120, and assigns weighting according to the tracking coefficient to the pixel data in the comparison target range of the input image, and The image processing is performed on the pixel data in the comparison target range of the processed image, and the weighted pixel data is added. This image processing is performed on all the pixel data constituting the image. That is, a new post-processed image can be obtained by adding (picture making) the images that have been subjected to image processing for all pixel data. The new processed image is stored in the processed image memory 140.

  The display unit 230 reads and displays the processed image stored in the processed image memory 140. As a result, it is possible to see an image of the imaging target that has been processed, that is, the disturbance has been removed.

  The error measurement unit 2160 reads the processed image processed by the image processing unit 130 and the processed image one frame before the processed image processed by the image processing unit 130 and stored in the processed image memory 140. Error measurement.

  Specifically, the determination is made based on whether or not the difference between the pixel data values in a predetermined range of each image exceeds a certain threshold value. As this predetermined range, a portion with a small background motion is adopted in the captured image. In other words, when the change in luminance is large within that range, it means that there are many disturbances.

  The error measurement unit 2160 counts the number of pixels in which the difference between the pixel data values exceeds a certain threshold within a predetermined range, and outputs the number of pixels to the tracking coefficient update processing unit 2170. The threshold value for determining the difference between the pixel data values within a predetermined range is appropriately set from the operation unit 220 and stored in the image quality reference setting memory 2150.

  The tracking coefficient update processing unit 2170 determines that there is a lot of disturbances and sets the tracking coefficient when it exceeds the reference value of the error ratio that is set as appropriate from the operation unit 220 and stored in the image quality standard setting memory 2150. Decrease the level, and if the reference value of the error rate is not exceeded, determine that the disturbance is small and increase the tracking coefficient by one level. This makes it possible to perform image processing so as not to be affected by disturbances when there are many disturbances, and to perform image processing so as not to be affected by disturbances when there are few disturbances. It becomes possible.

FIG. 18 is a block diagram relating to error measurement processing of the error measurement unit.
As shown in FIG. 18, the error measurement unit 2160 includes a frame memory 2161, an interframe difference calculation unit 2162, a magnitude comparison unit 2163, a counting unit 2164, region determination units 2165 and 2166, and an adder 2167.

  The frame memory 2161 stores the processed image output from the image processing unit 130 for one frame.

  The inter-frame difference calculation unit 2162 calculates the inter-frame difference between the post-processing image output from the image processing unit 130 and the post-processing image one frame before and stored in the frame memory 2161. .

  The magnitude comparison unit 2163 outputs the output signals from the region determination units 2165 and 2166 to the adder 2167, and when it is determined that both the loop variables X and Y are within the region, the difference value of the (X, Y) pixel and the allowable value Compare the values. When the difference value exceeds the allowable value, the fact is output to the counting unit 2164, and the counting unit 2164 counts the number of pixels whose difference value exceeds the allowable value, and the difference value within the determination area is allowable. The number of pixels exceeding the value is output to the tracking coefficient update processing unit 2170.

  19, 20, and 21 are flowcharts illustrating image processing procedures including error measurement and tracking count update processing by the image processing apparatus. In the following, the processes shown in FIGS. 19, 20, and 21 will be described in order of step number.

  [Step S51] When the power is turned on, the image processing apparatus 100 initializes the processed image in the processed image memory 140. Specifically, when the disturbance is snow, a white image is stored in the post-processing image memory 140. Further, the tracking coefficient α1 is set to 1.0, and the tracking coefficient α2 is set to 0.5.

  [Step S52] The disturbance determination processing unit 120 reads an input image input from the camera 210 and stored in the input image memory 110.

  [Step S53] The disturbance determination processing unit 120 and the image processing unit 130 perform disturbance determination processing and image processing. The disturbance determination process and the image process are the same as those described with reference to FIGS. 9 and 10 of the first embodiment.

  [Step S54] The error measurement unit 2160 sets end point coordinates (X_END, Y_END) of the error measurement range, which is a range in the post-processing image for performing error measurement, and resets the counter (counting unit 2164).

  [Step S55] A loop variable Y in the height direction of the image is initialized. Specifically, “Y start point” which is the height coordinate of the start point coordinates (X start point, Y start point) of the error measurement range is set in the loop variable Y.

  [Step S56] The loop variable X in the width direction of the image is initialized. Specifically, “X start point” which is the width coordinate of the start point coordinates (X start point, Y start point) of the error measurement range is set in the loop variable X.

  [Step S57] It is determined whether Y_END is larger than the loop variable Y. If Y_END is larger than loop variable Y, the process proceeds to step S58. If Y_END is equal to or smaller than loop variable Y, the process proceeds to step S66.

  [Step S58] It is determined whether X_END is greater than the loop variable X. If X_END is greater than loop variable X, the process proceeds to step S59. If X_END is equal to or less than loop variable X, the process proceeds to step S64.

  [Step S59] The absolute value of the difference in luminance of (X, Y) pixels between the processed image and the processed image one frame before stored in the frame memory 2161 is calculated, and the difference absolute value is set in advance. It is determined whether or not it is larger than the allowable value. If the absolute difference value is greater than the allowable value, the process proceeds to step S60. If the absolute difference value is equal to or smaller than the allowable value, the process proceeds to step S61.

  [Step S60] The counter is incremented by one.

  [Step S61] The pixel data of the processed image for which it is determined whether or not the absolute difference value is greater than the allowable value is stored in the frame memory 2161.

  [Step S62] The loop variable X is incremented by one.

  [Step S63] The process returns to step S58.

  [Step S64] The loop variable Y is incremented by one.

  [Step S65] The process returns to step S56.

  [Step S66] It is determined whether or not the preset reference value of the error ratio is larger than the counter value of the counter 2164. If the reference value is greater than the counter value, the process proceeds to step S67. If the reference value is equal to or less than the counter value, the process proceeds to step S68.

  [Step S67] The follow-up coefficient update processing unit 2170 raises the follow-up coefficient α2 by one step.

  [Step S68] The follow-up coefficient update processing unit 2170 lowers the follow-up coefficient α2 by one step.

  [Step S69] The process returns to step S52.

The above processing functions can be realized by a computer. In that case, an image processing program describing the processing contents of the functions that the image processing apparatus should have is provided. By executing the image processing program on a computer, the above processing functions are realized on the computer. The image processing program describing the processing contents can be recorded on a computer-readable recording medium. Examples of the computer-readable recording medium include a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory. Magnetic recording devices include HDDs, FDs, magnetic tapes, and the like. Optical discs include DVD (Digital Versatile Disc), DVD-RAM, CD-ROM, CD-R (Recordable) / RW (ReWritable), and the like. Magneto-optical recording devices include MO (Magneto
Optical disk).

  When distributing the image processing program, for example, portable recording media such as a DVD and a CD-ROM in which the image processing program is recorded are sold. It is also possible to store the program in a storage device of a server computer and transfer the image processing program from the server computer to another computer via a network.

  The computer that executes the image processing program stores, for example, the image processing program recorded on the portable recording medium or the image processing program transferred from the server computer in its own storage device. Then, the computer reads the image processing program from its own storage device and executes processing according to the image processing program. The computer can also read the image processing program directly from the portable recording medium and execute processing according to the image processing program. The computer can also sequentially execute processing according to the received image processing program every time the image processing program is transferred from the server computer.

  The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

  The above merely illustrates the principle of the present invention. In addition, many modifications and changes can be made by those skilled in the art, and the present invention is not limited to the precise configuration and application shown and described above, and all corresponding modifications and equivalents may be And the equivalents thereof are considered to be within the scope of the invention.

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 110 Input image memory 120 Disturbance judgment processing part 130 Image processing part 140 Post-processing image memory 150 Tracking coefficient memory 210 Camera 220 Operation part 230 Display part

Claims (14)

In an image processing apparatus that generates a processed image after image processing by processing pixel data constituting a captured image captured by an imaging unit,
A disturbance, not an imaging object, was imaged only by comparing the value of the pixel data in the comparison target range in the processed image with the value of the pixel data in the comparison target range in the input image input from the imaging means. Disturbance judging means for judging whether or not the pixel data of the result is high,
By weighting and adding the pixel data value of the comparison target range in the input image and the pixel data value of the comparison target range in the processed image based on the determination content of the disturbance determination process Image processing means for performing processing for generating a new post-processing image;
With
The image processing means includes
When the disturbance determination means determines that the pixel data value in the comparison target range in the processed image is likely to be pixel data resulting from copying a disturbance, tracking of a numerical value that is larger than 0 but lower Weighting with coefficients,
When it is determined that the pixel data value in the comparison target range in the processed image is unlikely to be pixel data resulting from copying a disturbance, weighting is performed using a high numerical tracking coefficient,
The disturbance determination means is
As an initial value image, a uniform image set to brightness or saturation similar to the disturbance is provided as initial value data in advance, and in the initial state where the processed image is not present, it is replaced with the processed image. Using the initial value image,
When the pixel data value in the comparison target range in the processed image is compared with the pixel data value in the comparison target range in the input image input from the imaging unit, there is a discrepancy greater than the threshold value. When this is the case, it is determined that the pixel data in the comparison target range in the input image is likely to be pixel data obtained as a result of imaging the imaging target .   The initial value image is
  The image processing apparatus according to claim 1, wherein the image processing apparatus is a single white image. Compares the value of the pixel data in the comparison target range in the input image with the value of the pixel data in the comparison target range in the processed image in a range where the movement of the imaging target set in advance is small in the captured image. And a tracking coefficient determining means for lowering the tracking coefficient when there is a divergence of a value equal to or greater than the threshold value, and for increasing the tracking coefficient when the number of pixels exceeds the threshold. The image processing apparatus according to claim 1 .   When the value of pixel data in the comparison target range in the processed image and the value of pixel data in the comparison target range in the input image input from the imaging unit are compared, a certain number of pixels or more in the entire image is constant. The image processing apparatus according to claim 1, wherein when the above values deviate, the input image is set as a new post-processing image.   The comparison between the pixel data value in the comparison target range in the processed image and the pixel data value in the comparison target range in the input image input from the imaging unit is performed by comparing the luminance of the pixel data. The image processing apparatus according to claim 1.   When the luminance of the pixel data in the comparison target range in the input image input from the imaging unit is larger than the luminance of the pixel data in the comparison target range in the processed image, the pixel data in the comparison target range in the input image The image processing apparatus according to claim 5, wherein the image processing apparatus determines that there is a high possibility that the pixel data is a result of imaging a disturbance rather than an imaging target. When the brightness of the object being captured is higher than the brightness of the disturbance,
6. The image processing apparatus according to claim 5, wherein weighting is performed using a low tracking coefficient when pixel data having a luminance equal to or higher than a set value less than the luminance of the disturbance is input.   The comparison between the pixel data value in the comparison target range in the processed image and the pixel data value in the comparison target range in the input image input from the imaging unit is performed by further comparing the saturation of the pixel data. The image processing apparatus according to claim 5, wherein the image processing apparatus is performed.   When the saturation of the pixel data in the comparison target range in the input image input from the imaging means is smaller than the saturation of the pixel data in the comparison target range in the processed image, the comparison target range in the input image The image processing apparatus according to claim 8, wherein the pixel data is determined to be highly likely to be pixel data obtained as a result of imaging a disturbance rather than an imaging target. In an imaging system that transmits using a transmission device that transmits a captured image captured by the imaging unit, and displays the captured image at a location separated from an installation point of the imaging unit,
The value of the pixel data in the comparison target range in the processed image after the image processing by processing the pixel data constituting the captured image, and the value of the pixel data in the comparison target range in the input image input from the imaging means A disturbance determination means for determining whether or not there is a high possibility that the pixel data is a result of imaging a disturbance rather than an imaging object;
By weighting and adding the pixel data value of the comparison target range in the input image and the pixel data value of the comparison target range in the processed image based on the determination content of the disturbance determination process Image processing means for performing processing for generating a new post-processing image;
With
The image processing means includes
When the disturbance determination means determines that the pixel data value in the comparison target range in the processed image is likely to be pixel data resulting from copying a disturbance, tracking of a numerical value that is larger than 0 but lower Weighting with coefficients,
When it is determined that the pixel data value in the comparison target range in the processed image is unlikely to be pixel data resulting from copying a disturbance, weighting is performed using a high numerical tracking coefficient,
The disturbance determination means is
As an initial value image, a uniform image set to brightness or saturation similar to the disturbance is provided as initial value data in advance, and in the initial state where the processed image is not present, it is replaced with the processed image. Using the initial value image,
When the pixel data value in the comparison target range in the processed image is compared with the pixel data value in the comparison target range in the input image input from the imaging unit, there is a discrepancy greater than the threshold value. In some cases, the imaging system includes an image processing apparatus that determines that the pixel data in the comparison target range in the input image is likely to be pixel data as a result of imaging the imaging target .   The imaging system according to claim 10, wherein the image processing apparatus is connected between the imaging unit and the transmission apparatus.   The imaging system according to claim 10, wherein the image processing device is connected between the transmission device and a display device that displays an image transmitted by the transmission unit. In an image processing method for generating a processed image after image processing by processing pixel data constituting a captured image captured by an imaging unit,
The disturbance determination unit compares the pixel data value in the comparison target range in the processed image with the pixel data value in the comparison target range in the input image input from the imaging unit, and the disturbance is not an imaging target object. Determining whether there is a high possibility that the pixel data is a result of imaging
Image processing means weights the pixel data value of the comparison target range in the input image and the pixel data value of the comparison target range in the processed image based on the determination content of the disturbance determination process. Performing a process of generating a new post-processing image by adding
Wherein the image processing means, the disturbance decision means, the value of the pixel data of the comparison target range in the processed image is, when it is determined that there is a high possibility that the pixel data due to the copy of the disturbance, 0 than Perform weighting using large but low tracking factors,
When it is determined that the pixel data value of the comparison target range in the processed image is unlikely to be pixel data resulting from copying a disturbance, weighting is performed using a high numerical tracking coefficient;
The disturbance determination means, as an initial value image, is provided with a uniform image set in luminance or saturation similar to the disturbance as initial value data in advance, and in the initial state where the processed image is not present, Using the initial value image instead of the processed image,
When the pixel data value in the comparison target range in the processed image is compared with the pixel data value in the comparison target range in the input image input from the imaging unit, there is a discrepancy greater than the threshold value. Determining that the pixel data in the comparison target range in the input image is likely to be pixel data obtained by imaging the imaging target ; and
An image processing method comprising: In an image processing program for generating a processed image after image processing by processing pixel data constituting a captured image captured by an imaging unit,
Computer
Compare the pixel data value of the comparison target range in the processed image with the pixel data value of the comparison target range in the input image input from the imaging means, Disturbance judgment means for judging whether or not there is a high possibility of pixel data,
By weighting and adding the pixel data value of the comparison target range in the input image and the pixel data value of the comparison target range in the processed image based on the determination content of the disturbance determination process Function as an image processing means for performing processing to generate a new post-processing image,
The image processing means includes
When the disturbance determination means determines that the pixel data value in the comparison target range in the processed image is likely to be pixel data resulting from copying a disturbance, tracking of a numerical value that is larger than 0 but lower Weighting with coefficients,
When it is determined that the pixel data value in the comparison target range in the processed image is unlikely to be pixel data resulting from copying a disturbance, weighting is performed using a high numerical tracking coefficient,
The disturbance determination means is
As an initial value image, a uniform image set to brightness or saturation similar to the disturbance is provided as initial value data in advance, and in the initial state where the processed image is not present, it is replaced with the processed image. Using the initial value image,
When the pixel data value in the comparison target range in the processed image is compared with the pixel data value in the comparison target range in the input image input from the imaging unit, there is a discrepancy greater than the threshold value. When this is the case, it is determined that there is a high possibility that the pixel data in the comparison target range in the input image is pixel data as a result of imaging the imaging target .