JP2019062528A - Image processing method, image processing system, imaging apparatus, and program - Google Patents

Abstract

To restrain creation of such image data looking as if a region where blurring occurs and a region where blurring does not occur appears, when cyclic NR processing is applied.SOLUTION: A motion amount detection circuit detects a motion amount from captured image data. A blur amount calculation circuit calculates the blur amount of an imaging apparatus which captured the image data. A strength determination circuit sets the strength of cyclic NR processing. A cyclic NR processing circuit performs cyclic NR processing for the image data with a set strength. A blur correction circuit performs blur correction processing for suppressing blur of the image data subjected to cyclic NR processing, on the basis of the blur amount. When blur correction processing is not operating, the strength determination circuit controls strength of cyclic NR processing in units of region including one or more pixels, on the basis of the motion amount, and when blur correction processing is operating, performs strength control of uniform cyclic noise reduction processing for all region of the image data.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for reducing noise and blurring of image data.

  A cyclic noise reduction process is known as an image processing technique for the purpose of noise reduction of image data. For example, Patent Document 1 discloses a technique for detecting a motion amount in units of pixels and performing cyclic noise reduction processing at an intensity set in units of pixels in accordance with the detected motion amount. As an application example of these cyclic noise reduction processes, an imaging device can be mentioned.

  Further, Patent Document 2 discloses a technique of estimating a motion amount between two image data and generating motion compensated image data from the estimated motion amount. By using this motion compensated image data, blur correction processing can be performed on the image data.

JP, 2012-129617, A JP, 2014-39169, A

  By the way, when the above-described cyclic noise reduction processing is applied to the image data and then the shake correction processing is applied, image data is generated that looks like an area in which blurring occurs and an area in which blurring does not occur. It turns out that there is a case that

  Therefore, an object of the present invention is to suppress generation of image data that appears to cause an area in which blurring occurs and an area in which blurring does not occur when cyclic NR processing is applied. .

  According to the image processing apparatus of the present invention, a motion amount detection unit that detects a motion amount from a captured image, a blur detection unit that detects a blur amount of an imaging device that captures the image, and cyclic noise A control unit for controlling a reduction process, a processing unit for performing a cyclic noise reduction process on the image under the control of the cyclic noise reduction process by the control unit, and a blur of the image based on the blur amount And the control unit performs, on the basis of the movement amount, the area unit including one or more pixels when the shake correction process is not in operation. The intensity of the cyclic noise reduction process is controlled, and when the shake correction process is operating, the intensity control of the cyclic noise reduction process is uniform on all regions of the image. And performing.

  According to the present invention, when cyclic NR processing is applied, it is possible to suppress the generation of image data that looks like an area in which blurring occurs and an area in which blurring does not occur. A processing device can be provided.

It is a block diagram showing a schematic structure of an imaging device of a 1st embodiment. It is a flowchart which shows the flow of the process in 1st Embodiment. It is a flow chart of uniform intensity calculation processing in a 1st embodiment. It is a figure showing the relationship of the motion amount for every area | region and the intensity | strength of the area unit of cyclic NR process in 1st Embodiment. It is a figure showing the relationship between the motion amount for every area | region in 1st Embodiment, and uniform intensity | strength of cyclic NR processing. FIG. 8 is a diagram for describing an aspect of image data when an intensity is set in area units and shake correction processing is performed. It is a figure for demonstrating the relationship between the intensity | strength of the cyclic | annular NR process in 2nd Embodiment, and the intensity | strength of space type NR process. It is a figure for demonstrating the relationship between the intensity | strength of the area | region unit in 3rd Embodiment, and uniform intensity | strength.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.
In the present embodiment, an imaging apparatus will be described as an example application of the present invention.
The imaging device of the present embodiment has a function of executing cyclic noise reduction processing as a function of reducing noise of image data. In the following description, cyclic noise reduction processing is referred to as cyclic NR processing. The image pickup apparatus according to the present embodiment also has a shake correction function that obtains shake (shake) of the image pickup apparatus at the time of shooting and performs shake correction processing on image data based on the shake information. . Furthermore, the imaging apparatus operates the shake correction function when the shake amount (magnitude of shake) obtained at the time of shooting is larger than the predetermined shake amount, while the shake amount is less than the predetermined shake amount (the imaging device It also has a function to stop the shake correction function when it hardly shakes. Here, it is assumed that the predetermined shake amount when deciding whether to operate the shake correction function is a predetermined value.

  Although details will be described later, the imaging apparatus according to the present embodiment has a function of detecting the amount of movement in units of regions and controlling the strength of cyclic NR processing in units of regions according to the detected amount of movement. . In the following description, the cyclic NR process performed with the intensity in the area unit is referred to as a cyclic NR process in the area unit. Furthermore, the imaging apparatus according to the present embodiment also has a function of performing cyclic NR processing by calculating the strength of cyclic NR processing applied uniformly to all regions based on information of motion amount in units of regions. doing. In the following description, cyclic NR processing applied to all areas with uniform strength will be referred to as uniform cyclic NR processing. Note that instead of the configuration in which the cyclic NR processing is performed with the intensity set in area units, the cyclic NR processing may be performed in the intensity set in pixel units. Here, in order to simplify the description, a configuration in which cyclic NR processing is performed at an intensity set in an area unit including one or more pixels will be described as an example.

First Embodiment
The configuration and process for controlling the strength of cyclic NR processing based on the blur amount and the motion amount in the imaging device of the first embodiment will be described below with reference to FIGS. 1 to 5.
FIG. 1 is a block diagram showing an example of a schematic configuration of an imaging apparatus according to the present embodiment. Note that FIG. 1 shows only the main components that control the intensity based on the amount of shake and the amount of movement, and illustration of other general components included in the imaging apparatus is omitted.

The imaging device includes an image generation device 100, an image processing device 110, a display device 121, and a recording circuit 122.
The image generation apparatus 100 includes an imaging optical system 101, an imaging element 102 for converting an optical image formed by the imaging optical system 101 into an electrical signal, and image data generation for generating image data from the electrical signal generated by the imaging element 102. A circuit 103 is included. The image data generated by the image data generation circuit 103 is output to the image processing apparatus 110.

The image processing apparatus 110 includes a motion amount detection circuit 111, a blur amount calculation circuit 112, a uniform intensity calculation circuit 113, an intensity determination circuit 114, a cyclic NR processing circuit 115, an image processing circuit 116, and a blur correction circuit 117.
The motion amount detection circuit 111 receives image data of continuous frames from the image data generation circuit 103, and divides each image data into a plurality of regions. Then, the motion amount (motion vector) of each area is detected from the reference image data which is any frame and the image data one frame before the reference image data. Then, information on the detected motion amount is output to the shake amount calculation circuit 112, the uniform intensity calculation circuit 113, and the intensity determination circuit 114.

  The shake amount calculation circuit 112 calculates the magnitude (shake amount) of the shake and the direction of the shake when the imaging device shakes at the time of shooting. For example, the shake amount calculation circuit 112 acquires information on the movement amount detected by the movement amount detection circuit 111, and estimates the shake amount of the imaging device based on the movement amount. There are various methods for estimating the blur amount based on the motion amount, and any of these methods may be used in this embodiment. As another example, the shake amount calculation circuit 112 may detect the shake amount of the imaging apparatus based on data obtained from a device such as an acceleration sensor or an angular velocity sensor (not shown). Various methods exist for calculating the shake amount based on data from an acceleration sensor, an angular velocity sensor or the like, and any of these methods may be used in this embodiment. Then, the shake amount calculation circuit 112 outputs the information of the shake amount estimated or calculated as described above to the intensity determination circuit 114 and the shake correction circuit 117.

The uniform intensity calculation circuit 113 calculates the intensity (uniform intensity) of cyclic NR processing applied uniformly to all regions based on the information of the motion amount acquired from the motion amount detection circuit 111.
Based on the shake amount acquired from the shake amount calculation circuit 112, the intensity determination circuit 114 determines whether or not the shake correction process is to be performed in the shake correction circuit 117 in the subsequent stage. Then, when the shake correction process is performed in the shake correction circuit 117, the intensity determination circuit 114 performs cyclic operation with the intensity calculated by the uniform intensity calculation circuit 113 prior to the start of the shake correction process by the cyclic NR processing circuit 115. Perform type NR processing. Note that it can be determined whether, for example, the calculated shake amount is larger than a predetermined shake amount, for example, whether or not the shake correction processing is performed by the shake correction circuit 117 in the subsequent stage. For example, when the calculated shake amount is larger than a predetermined shake amount, the shake correction process is performed. On the other hand, when the calculated shake amount is smaller than the predetermined shake amount, it is determined that the shake correction process is not performed.

  The uniform intensity calculation circuit 113 acquires statistical data of the motion amount detected by the motion amount detection circuit 111. The statistical data is, for example, the average value, median value, maximum value, minimum value, and variance of the motion amount. Next, the uniform intensity calculation circuit 113 obtains the degree of variation of the motion amount in the reference image data from the variance of the statistical data. Furthermore, the uniform intensity calculation circuit 113 selects the reference value of the motion amount when calculating the uniform intensity from the statistical data based on the degree of variation of the motion amount. Next, the uniform intensity calculation circuit 113 uses the reference value of the selected motion amount to calculate a uniform intensity uniformly applied to all the regions. The method of selecting the reference value of the motion amount when calculating the uniform intensity will be described later. Then, the uniform intensity calculation circuit 113 outputs the calculated uniform intensity information to the intensity determination circuit 114.

  The strength determination circuit 114 calculates the strength (the strength in area units) of the cyclic NR processing in area units based on the movement amount in each area detected by the movement amount detection circuit 111. Further, the intensity determination circuit 114 obtains, from the uniform intensity calculation circuit 113, uniform intensity applied to the entire area, and based on the shake amount calculated by the shake amount calculation circuit 112, uniform intensity and intensity in area units. Decide which of the two to select and output. In the case of the present embodiment, the intensity determination circuit 114 selects the intensity in units of regions when the calculated shake amount is equal to or less than a predetermined shake amount and the shake correction processing in the subsequent stage is not performed. On the other hand, in the present embodiment, when the detected shake amount is larger than the predetermined shake amount and the shake correction processing in the latter stage is performed, the intensity determination circuit 114 selects a uniform intensity. Then, the strength determination circuit 114 outputs, to the cyclic NR processing circuit 115, information of the selected one of the uniform strength and the strength in the area unit as described above.

  The cyclic NR processing circuit 115 executes cyclic NR processing on the image data received from the image generation apparatus 100 based on the uniform intensity or the intensity in units of regions supplied from the intensity determination circuit 114. Then, the cyclic NR processing circuit 115 outputs the image data after the cyclic NR processing to the image processing circuit 116.

  The image processing circuit 116 executes various types of image processing other than the cyclic NR processing on the image data after the cyclic NR processing by the cyclic NR processing circuit 115. For example, the image processing circuit 116 performs a spatial noise reduction process (spatial NR process) for performing a spatial filter process on one image data, a sharpness process for enhancing the sharpness, and a floor for improving the expression of the intermediate gradation. Execute various image processing such as tone correction processing. The image processing circuit 116 outputs the image data after the image processing to the blur correction circuit 117.

  The shake correction circuit 117 switches whether to perform the shake correction processing based on the shake amount from the shake amount calculation circuit 112. When it is determined that the shake correction process is to be performed, the shake amount calculation circuit 112 executes the shake correction process according to the shake amount from the shake amount calculation circuit 112. Specifically, the shake correction circuit 117 stops the shake correction process if the detected shake amount is less than a predetermined shake amount, and if the shake amount is larger than the predetermined shake amount, the shake correction process is performed. Shake correction processing is performed according to the direction and the amount of shake. The shake correction process is, for example, a process of cutting out part of image data at a position shifted by an amount corresponding to the shake amount in the direction opposite to the direction of the shake. Image data that has been subjected to shake correction processing by the shake correction circuit 117 or image data for which shake correction processing has not been performed because there is no shake is output to the display device 121 or the recording circuit 122 or the like.

  The image data output from the shake correction circuit 117 may be recorded on a recording medium by the recording circuit 122, for example, or may be used to display an image by the display device 121. Alternatively, the image data output from the shake correction circuit 117 may be transmitted to an external device by a communication device (not shown).

FIG. 2 shows motion amount detection, blur amount calculation, determination of strength of cyclic NR processing, cyclic NR processing, blur correction processing, and image data, which are performed by the imaging device of the present embodiment shown in FIG. It is the flowchart which showed the flow of a series of processes to an output.
In addition, in FIG. 2, step S10 to S80 of each process is abbreviated as S10 to S80, respectively. Further, the processing of the flowchart in FIG. 2 may be executed by a hardware configuration, or may be realized by the CPU or the like executing a program. The same applies to the other flowcharts described later. The process of the flowchart of FIG. 2 is started by the image generating apparatus 100 generating image data.

First, in step 10, the image processing device 110 acquires image data generated by the image generation device 100, and the motion amount detection circuit 111 of the image processing device 110 detects a motion amount (motion vector) for each area.
In step 20, the shake amount calculation circuit 112 calculates the shake amount of the imaging device. As described above, the shake amount may be detected based on the movement amount detected by the movement amount detection circuit 111, or may be detected based on an output from a device such as an acceleration sensor. The information on the shake amount detected by the shake amount calculation circuit 112 is sent to the intensity determination circuit 114 and the shake correction circuit 117 as described above.

  Next, in step 30, the uniform intensity calculation circuit 113 calculates the intensity of the uniform cyclic NR process uniformly applied to the entire area of the image data as described above based on the motion amount detected in step 10. Do. The uniform intensity calculation process will be described later with reference to the flowchart of FIG.

In step 40, the strength determination circuit 114 calculates the strength of cyclic NR processing for each area based on the movement quantity for each area detected by the movement quantity detection circuit 111 as described above.
Furthermore, in step 50, the intensity determination circuit 114 selects one of the uniform intensity and the intensity for each area based on the shake amount calculated by the shake amount calculation circuit 112 as described above, and performs cyclic NR processing. It outputs to the circuit 115.
Next, in step 60, the cyclic NR processing circuit 115 performs cyclic NR processing on the image data using the intensity selected in step 50.

Here, the signal level of the image data at the coordinates (x, y) of the Nth frame before applying the cyclic NR processing is V _xyN , and the coordinates of the Nth frame after applying the cyclic NR processing The signal level of the image data at (x, y) is W _xyN , and the signal level of the image data at the coordinates (x, y) of the (N−1) th frame after applying cyclic NR processing is W _{xy ( N−1)} , let k be a coefficient that is the strength of cyclic NR processing,
_{W xyN = k × (V xyN} -W xy (N-1)) + W xy (N-1) ··· (1)
It can be expressed by

Alternatively, Equation 1 can be transformed and expressed as Equation 2 below.
_{W xyN = k × V xyN +} (1-k) × W xy (N-1) ··· (2)

The coefficient k is a value of 0 or more and 1 or less, and as the value of the coefficient k is closer to 0, the value W _xyN after applying the cyclic NR processing is a value after applying the cyclic NR processing of the previous frame It becomes close to W _{xy (N-1)} . On the other hand, as the value of the coefficient k is closer to 1, the value W _xyN after applying the cyclic NR processing is _closer to the value V _xyN before applying the cyclic NR processing. That is, as the value of the coefficient k approaches 0, the amount of change in the signal level of image data at coordinates (x, y) becomes smaller between the preceding and succeeding frames, and fluctuations in the signal level due to noise can be suppressed. Conversely, the closer the value of the coefficient k is to 1, the smaller the influence of the signal level of the previous frame on the signal level of the subsequent frame can be made. Generally, as the amount of movement of the subject is smaller, the value of the coefficient k is set to a value closer to 0 in order to suppress the fluctuation of the signal level due to noise, and as the amount of movement of the subject is larger In order to make it smaller, the value of the coefficient k is made close to one. The uniform intensity indicates the coefficient k which is set to be the same for all the regions included in one frame, and the intensity in the region unit indicates the coefficient k which is individually set for each region of one frame. In this embodiment, the closer the value of the coefficient k to 0, the higher the strength of the cyclic NR processing (the greater the influence of the signal level of the previous frame), and the closer to 1 the value of the coefficient k, the cyclic NR processing It is assumed that the intensity is low (the influence of the signal level of the previous frame is small).

Further, at step 70, the shake correction circuit 117 executes shake correction processing based on the shake amount calculated at step 20 on the image data after cyclic NR processing by the cyclic NR processing circuit 115.
Thereafter, in step 80, the shake correction circuit 117 outputs the image data subjected to cyclic NR processing, shake correction processing, and the like to the display device 121 and the recording circuit 122.

FIG. 3 is a flowchart showing a flow of detailed processing of uniform intensity calculation performed by the uniform intensity calculation circuit 113 in step 30 of FIG.
In step 31 of FIG. 3, the uniform intensity calculation circuit 113 acquires the motion amount for each area detected by the motion amount detection circuit 111, and further, statistical data of the motion amount in the reference image data (average value of motion amount , Median, maximum, minimum, variance, etc.).
Next, in step 32, the uniform intensity calculation circuit 113 calculates the degree of variation of the amount of motion in the reference image data from the variance of the statistical data of the amount of motion calculated in step 31.

  Next, in step 33, the uniform intensity calculation circuit 113 selects the reference value of the movement amount used when calculating the uniform intensity based on the degree of variation of the movement amount calculated in step 32. In this case, first, the uniform intensity calculation circuit 113 selects reference data (reference value) from the statistical data of the motion amount of the reference image data described above. For example, when the variation degree of the motion amount of the reference image data is smaller than a predetermined value, the uniform intensity calculation circuit 113 selects the average value among the above-described statistical data of the motion amount as the motion amount reference value. . That is, when the variation degree of the motion amount is smaller than a predetermined value, the difference from the average value of the motion amount to the maximum value of the motion amount, or the difference from the average value to the minimum value is small. The average value of statistical data of may be used as a reference value. The uniform intensity calculation circuit 113 may select, as the reference value, the median value of the statistical data of the movement amount when the variation degree of the movement amount of the reference image data is smaller than a predetermined value.

  On the other hand, if the variation of the motion amount is equal to or more than a predetermined value, using the average value or the median value of the statistical data of the motion amount as a reference value as described above, the strength of cyclic NR processing may be more than necessary depending on the area. It may be high and an afterimage may occur due to the influence of the previous frame. Therefore, when the variation degree of the movement amount is equal to or more than the predetermined value, the uniform intensity calculation circuit 113 determines the maximum value of the movement amount at which the intensity is lowest among the statistical data of the movement amount in order to suppress the occurrence of an afterimage. , As the reference value of the movement amount. This uniform intensity also makes the value of the coefficient k closer to 0 (increases the intensity) as the movement amount of the subject is smaller, and the larger the movement amount of the subject, as in the general cyclic NR processing. Make the value of coefficient k close to 1 (lower the intensity).

Then, in the next step 34, the uniform intensity calculation circuit 113 calculates a uniform intensity uniformly applied to all the regions based on the reference value acquired in step 33.
In addition, the above-mentioned predetermined value compared with the variation degree of the acquired motion amount can be calculated | required experimentally. When the predetermined value is determined experimentally, for example, a plurality of test moving images with different degrees of variation of the movement amount are prepared. Then, the degree of the afterimage when the strength based on the average value of the motion amount is set is determined in order from the moving image having the smaller degree of variation of the motion amount, and the degree of variation of the motion amount of the moving image initially determined May be set as a predetermined value.

FIG. 4 is a diagram showing the relationship between the motion amount for each area and the strength of cyclic NR processing in strength control. FIG. 4 is a diagram showing the intensity in units of regions determined by the intensity determination circuit 114 when the blur correction process is not performed (when the blur correction process is invalid).
When the image pickup apparatus hardly shakes (shakes) and shake correction processing is not performed, the intensity determination circuit 114 selects not the uniform intensity but the intensity in area units.

  Then, as shown in FIG. 4, the intensity determination circuit 114 lowers the intensity in order to give priority to preventing the occurrence of an afterimage in the area of a subject with motion as the area having a large amount of movement. On the other hand, since the risk that the afterimage occurs in the area of the subject decreases as the area where the amount of movement is smaller, the strength determination circuit 114 makes the strength stronger in order to prioritize reduction of noise. Specifically, in the example of FIG. 4, the value of coefficient k is set to 0 if the amount of motion is less than threshold Th1, and the value of coefficient k is set to 1 if the amount of motion is equal to or greater than threshold Th2. If the amount is greater than or equal to the threshold Th1 and less than Th2, the coefficient k approaches 1 as the amount of movement increases. Note that Th1 may be 0. In this case, if there is a plurality of areas having different magnitudes of movement amount in one image data, as shown in FIG. It will exist.

Similar to FIG. 4, FIG. 5 is a diagram showing the relationship between the amount of movement and the intensity for each area. FIG. 5 is a view showing the uniform intensity determined by the intensity determination circuit 114 when the blur correction process is performed (when the blur correction process is effective).
As in the example of FIG. 5, when shake correction processing for correcting shake (shake) of the imaging apparatus is performed, the intensity determination circuit 114 selects not the intensity in units of regions but a uniform intensity.

  That is, when shake correction processing is performed, as shown in FIG. 5, the strength determination circuit 114 does not depend on the amount of movement for each area, but uses the strength based on the reference value of one movement amount determined in the frame. By setting, no difference occurs in the blurring amount depending on the area in the image data. That is, the cyclic NR processing circuit 115 performs cyclic NR processing with uniform intensity on all regions.

Here, when blurring correction processing is performed, the reason why it is desirable to set a uniform intensity instead of setting the intensity in units of regions will be described.
FIG. 6 is a diagram for explaining the state of image data when the intensity is set in area units and the blur correction process is performed. In FIG. 6, it is assumed that a window of a building is included as a subject in the image data of each of the (N−1) th frame and the Nth frame.

  FIG. 6A shows the window 601 in the Nth frame and the window 602 in the (N-1) th frame. The window 601 and the window 602 are the same window, and it is assumed that the position in the image data of each frame is shifted by the amount indicated by the arrow due to blurring. In the window 601 and the window 602, the sky is reflected inside the window frame indicated by a solid line, and although the signal level inside the window frame is substantially uniform, there is a fluctuation of the signal level due to noise. On the contrary, it is assumed that the signal levels of the window 601 (window 602) and the surrounding area are largely different.

Here, the processing in the case of applying the cyclic NR processing to the (N−1) th frame by setting the strength in area units will be described.
As shown in FIG. 6B, of the window 601 and the window 602, it is determined that the difference in signal level is small and the amount of movement is small in the shaded portion. That is, in window 602, the value of coefficient k is set to 0 (or a value close to 0) for the shaded portion and corrected to the same value (or substantially the same value) as the signal level of the shaded portion of window 601. Ru. In the other areas, it is determined that the difference in signal level between frames is large, and the value of the coefficient k is set to 1 (or a value close to 1). That is, the area of the window 602 other than the shaded portion is hardly affected by the signal level of the window 601.

  Thereafter, as shown in FIG. 6C, the position in the image data of the window 602 is corrected by the shake correction process. At this time, while the signal level of the shaded portion of the window 602 matches the signal level of the shaded portion of the window 601, the signal level of the area other than the shaded portion does not match the window 601. Therefore, to the user, the image data of the shaded portion in the window 601 appears to move in the direction of the arrow in the window 602. That is, for the user, it appears that the inside of the window 602 includes a blurred area and a non-shake area.

  Therefore, in the present embodiment, when the blur correction process is performed after the cyclic NR process, the intensity is not calculated in the area unit but is calculated uniformly in the frame. With such a configuration, as shown in FIG. 6C, it can be suppressed that only a part of the area looks blurred.

  As described above, in the imaging apparatus according to the first embodiment, the cyclic NR processing based on the movement amount for each area is performed by the intensity control in the area unit when the blur correction process is not performed. On the other hand, when blur correction processing is performed, cyclic NR processing with uniform intensity is performed on all regions of image data. That is, according to the image pickup apparatus of the first embodiment, the intensity is appropriately controlled based on the amount of shake and the amount of movement so that an area in which the shake occurs and an area in which the shake does not occur are generated. The generation of visible image data can be suppressed.

Second Embodiment
Hereinafter, the second embodiment will be described with reference to FIG.
FIG. 7 is a diagram for explaining the relationship between the strength of cyclic NR processing and the strength of spatial NR processing. Note that the configuration of the imaging device of the second embodiment is substantially the same as that of FIG. 1 described above, and thus the illustration thereof is omitted.

  In FIG. 7A, the alternate long and short dash line indicates the intensity in units of regions with respect to the amount of movement for each region, and the solid line indicates uniform intensity based on the amount of movement reference. Also in the second embodiment, when the shake correction process is performed (when the shake correction process is effective), the intensity determination circuit 114 selects a uniform intensity as in the first embodiment.

  Here, when the variation degree of the movement amount of the area is equal to or more than a predetermined value, in the second embodiment as well as in the first embodiment described above, the maximum value of the movement amount in the statistical data of the movement amount is A uniform intensity is calculated based on the reference value, which is used as a reference value. That is, also in the second embodiment, when the variation degree of the movement amount of the area is equal to or more than a predetermined value, the afterimage is generated by performing cyclic NR processing in consideration of the area having a large movement amount. You can prevent.

  On the other hand, in this case, if the intensity calculated using the maximum value of the motion amount as the reference value as described above is used, an area in which cyclic NR processing is weaker than the intensity to be originally applied occurs. . That is, the uniform intensity calculated using the maximum value of the motion amount as the reference value is weaker than the intensity to be originally applied to the region where the motion amount is smaller than the reference value. That is, in the region where the amount of motion is smaller than the reference value, the effect of noise reduction by cyclic NR processing is reduced.

  Therefore, in the imaging apparatus according to the second embodiment, as shown in FIG. 7B, in a region where the motion amount is smaller than the reference value, the intensity to be originally applied and the uniform intensity calculated from the reference value. As the difference increases, the strength of the spatial NR process is increased to reduce noise. In other words, assuming that the reference value of the amount of motion is Th3, in the region where the amount of motion is smaller than the reference value Th3, the intensity of the spatial NR process is increased (stronger) as the region including the region where the amount of motion is smaller Try to reduce it.

  Here, the space type NR processing in the second embodiment is performed by the image processing circuit 116 of FIG. 1 described above. Therefore, in the second embodiment, the intensity determination circuit 114 obtains the difference between the intensity to be originally applied and the uniform intensity, and the image processing circuit 116 controls the intensity of the spatial NR processing based on the difference. Do. That is, in the case of the second embodiment, the intensity determination circuit 114 controls the image processing circuit 116 so as to increase the intensity of the spatial NR processing as the difference between the intensity in the area unit and the uniform intensity increases. . As a result, in the case of the second embodiment, the effect of the spatial NR process is intensified in the region where the difference between the intensity to be originally applied and the uniform intensity is large.

  As spatial NR processing, for example, there are the following methods. First, an area of a predetermined width is set centering on the pixel to be processed, and within the set area, a pixel having a signal level whose difference from the signal level of the pixel to be processed is equal to or less than a threshold is extracted Do. Then, a value obtained by averaging the signal level of the pixel to be processed and the signal level of the extracted pixel or a weighted average is set as the signal level of the pixel to be processed. As a method of increasing the strength of the spatial NR, it is conceivable to enlarge a predetermined area or to increase the threshold.

  Alternatively, the direction of the edge in the region including the processing target is determined, and a low pass filter is applied along the edge direction. By changing the filter coefficient of the low pass filter, the strength of the spatial NR can be increased. The strength determination circuit 114 may control the strength of the spatial NR processing when the area where the difference between the strength to be originally applied and the uniform strength is large is larger than a predetermined number.

  As described above, in the second embodiment, when the variation degree of the movement amount of the area is equal to or more than the predetermined value, the maximum movement amount is used as the reference value when calculating the uniform intensity, and the movement amount Calculate uniform intensity based on the reference value of. Furthermore, in the second embodiment, as shown in FIG. 7, the intensity of the spatial NR process is increased in the region where the uniform intensity value is lower than the intensity in the area unit and the difference is large. Thus, according to the second embodiment, the effect of noise reduction is also improved while suppressing generation of image data that appears to cause an area in which blurring occurs and an area in which blurring does not occur. be able to.

Third Embodiment
Next, with reference to FIG. 8, intensity control in the imaging device of the third embodiment will be described.
FIG. 8 is a diagram for explaining the relationship between the intensity in units of regions and the uniform intensity. The configuration of the imaging device of the third embodiment is substantially the same as that of FIG.
In the case of the imaging apparatus of the third embodiment, the intensity determination circuit 114 in FIG. 1 sets the intensity in area units according to the movement amount for each area based on the blurring amount calculated by the blurring amount calculation circuit 112. Determine the possible range.

  If the shake amount calculated by the shake amount calculation circuit 112 is larger than the first threshold value, the influence of the shake correction process executed after the cyclic NR process becomes large. The uniform intensity is calculated as in the second embodiment. That is, when the detected shake amount is larger than the first threshold value, the strength determination circuit 114 of the third embodiment does not perform the cyclic NR processing in which the strength in the area unit is set. If the detected blur amount is larger than the first threshold, the intensity control of the spatial NR processing in the image processing circuit 116 is performed as needed in the third embodiment as well as the second embodiment described above. You may

FIG. 8 is a diagram for explaining the strength of cyclic NR processing when the calculated shake amount is equal to or less than a first threshold and larger than a second threshold that is smaller than the first threshold.
The shake correction circuit 117 performs the shake correction process when the calculated shake amount is equal to or less than the first threshold and equal to or more than the second threshold.
When the calculated shake amount is equal to or less than the first threshold and equal to or more than the second threshold, the intensity determination circuit 114 performs area units within a range in which the influence of the shake correction processing in the subsequent stage does not appear as shown in FIG. Set the strength of

  When the calculated shake amount is equal to or less than the first threshold and equal to or more than the small second threshold, it is considered that the difference in the amount of shake due to the area of the image data is smaller than in the case where the shake amount is larger than the first threshold Be However, even when the calculated shake amount is equal to or less than the first threshold and equal to or more than the small second threshold, the difference in the amount of shake may increase to some extent depending on the area of the image data. For this reason, as shown in FIG. 8, the intensity determination circuit 114 narrows the settable range of the intensity in units of regions so that the difference in blur amount due to the region in the image data becomes smaller (the difference does not increase). Restrict to

  Specifically, when the calculated shake amount is equal to or less than the first threshold value and equal to or more than the small second threshold value, the strength determination circuit 114 first acquires statistical data of the movement amount, and moves from the statistical data. Calculate the reference value of the quantity. As the reference value of the motion amount, for example, an average value or a median value in statistical data of the motion amount can be used. Next, the strength determination circuit 114 calculates the settable range of strength in area units based on the reference value of the movement amount.

  In FIG. 8, a thick solid line indicates the intensity in the area unit, a thick dotted line indicates the uniform intensity based on the reference value of the movement amount, and a dashed dotted line indicates the intensity in the area unit when the blur amount is larger than the first threshold. . For example, when the blur amount is equal to or less than the first threshold and equal to or more than the small second threshold, the intensity determination circuit 114 makes a difference with respect to the uniform intensity so that the difference in the blur amount depending on the area in the image data is reduced. Limits the settable range of intensity in area units narrowly so that it falls within a predetermined range.

  As a method of narrowly limiting the settable range of the intensity, for example, a method of referring to a table in which the calculated shake amount is associated with the value of the settable range of the intensity in the area unit can be used. That is, in this table, the value of the settable range of the intensity in the unit of area is set for the value of the blur amount which is equal to or less than the first threshold and equal to or more than the second threshold. Note that the first threshold, the second threshold, and the values stored in the table may be determined experimentally while determining the difference in the amount of blurring included in the image data after processing while actually changing the values. . Further, as another method of narrowly limiting the settable range of the intensity, a method of obtaining the settable range of the intensity by calculation using the detected blur amount as a parameter may be used.

  Then, the strength determination circuit 114 sets the strength in the area unit according to the movement amount for each area in the settable range of the strength in the area unit set as described above, and executes the cyclic NR processing. That is, in the case of the third embodiment, it is possible to set the cyclic NR intensity of the area unit according to the movement amount for each area within the settable range of the intensity of the area unit. As a result, the cyclic NR processing circuit 115 can realize noise reduction suitable for the movement of the subject. Furthermore, in the third embodiment as well as in each of the above-described embodiments, the blur correction process is performed after the cyclic NR process, so it appears that an area in which blurring occurs and an area in which blurring does not occur may occur. It is possible to suppress the generation of image data.

  Furthermore, the shake correction circuit 117 performs the shake correction process also when the calculated shake amount is equal to or less than the second threshold and larger than a third threshold smaller than the second threshold. When the blurring amount is equal to or less than the second threshold and larger than the third threshold, the influence of the blurring correction processing performed thereafter is small even if the cyclic NR processing in which the intensity in the area unit is set is performed. Therefore, even if there is a difference in the amount of blurring due to the area of the image data, it is not noticeable to the eye. Therefore, when the calculated shake amount is equal to or less than the second threshold and larger than the third threshold, the strength determination circuit 114 sets the settable range of the strength in area units wide. In other words, when the detected shake amount is smaller than the second threshold value, the intensity determination circuit 114 sets the settable range of the coefficient k narrower than in FIG. 4 and in FIG. 8 (a) (or FIG. Set wider than the settable range shown in). That is, when the calculated shake amount is equal to or less than the second threshold and larger than the third threshold, the strength determination circuit 114 performs a suitable cyclic NR process according to the movement amount in units of regions. Note that the shake correction circuit 117 does not perform the shake correction process if the calculated shake amount is equal to or less than the third threshold value, and the cyclic NR processing circuit 115 performs cyclic NR processing with the intensity in units of regions shown in FIG. I do.

  As described above, in the image pickup apparatus according to the third embodiment, even when shake correction processing is performed, an area in which a shake occurs is generated by setting the intensity in an area unit in a range corresponding to the shake amount. The effect of noise reduction can also be improved while suppressing the generation of image data that appears to produce an area where blurring does not occur.

As mentioned above, although the preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.
Note that the imaging device according to the present embodiment described above can be applied to, for example, various portable terminals such as digital cameras and digital video cameras, smartphones and tablet terminals having a camera function, industrial cameras, on-vehicle cameras, medical cameras and the like. It is. Further, the present invention can be implemented only by the image processing apparatus 110 of FIG. 1, and the image generation apparatus 100, the display device 121, and the recording circuit 122 can be separate devices. For example, the present invention is also applicable to a personal computer, a server computer, etc. that receives image data from the above-mentioned various cameras by communication and performs cyclic NR processing and blur correction processing.

  The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

  The above-described embodiments are merely examples of implementation for practicing the present invention, and the technical scope of the present invention should not be interpreted limitedly by these. That is, the present invention can be implemented in various forms without departing from the technical concept or the main features thereof.

100 image generation apparatus 101 imaging optical system 102 image pickup element 103 image data generation circuit 110 image processing apparatus 111 motion amount detection circuit 112 shake amount calculation circuit 113 uniform intensity calculation circuit 114 intensity determination circuit 115 cyclic NR processing circuit 116 image processing circuit 117 Shake correction circuit 121 Display device 122 Recording circuit

Claims (17)

An image processing method performed by an image processing apparatus, comprising:
Detecting a motion amount from image data;
Calculating a blur amount of an imaging device that has captured the image data;
Setting the strength of the cyclic noise reduction process;
Performing the cyclic noise reduction process on the image data;
Performing blurring correction processing for suppressing blurring of the image data subjected to the cyclic noise reduction processing based on the blurring amount;
When the shake correction process is not in operation, the strength of the cyclic noise reduction process is set in an area unit including one or more pixels based on the movement amount.
An image processing method, wherein the strength of the cyclic noise reduction process is uniformly set to all the areas of the image data when the blur correction process is operating.   The image processing method according to claim 1, wherein the strength of the cyclic noise reduction processing uniformly set in all the regions is calculated based on statistical data of the motion amount. The variation degree of the movement amount is determined from the statistical data of the movement amount,
The image processing method according to claim 2, wherein the strength of the cyclic noise reduction processing uniformly set in all the regions is calculated based on the degree of variation of the motion amount. A reference value is selected based on the degree of variation of the movement amount,
The image processing method according to claim 3, wherein the strength of the cyclic noise reduction processing applied uniformly to all the regions is calculated based on a reference value.   5. The image processing method according to claim 4, wherein when the variation degree of the motion amount is smaller than a predetermined value, an average value or a median value of the motion amounts is selected as the reference value.   5. The image processing method according to claim 4, wherein the maximum value of the motion amount is selected as a reference value when the variation degree of the motion amount is equal to or more than a predetermined value. Performing a spatial noise reduction process on the image data;
The strength of the spatial noise reduction process is set based on the strength of the cyclic noise reduction process that is uniformly applied to all the regions. Image processing method.   The intensity of the spatial noise reduction process is higher in a region where the difference between the intensity of the cyclic noise reduction process uniformly applied to all the regions and the intensity of the cyclic noise reduction process based on the motion amount is larger. The image processing method according to claim 7, wherein the image processing method is set. An image processing method performed by an image processing apparatus, comprising:
Detecting a motion amount from image data;
Calculating a blur amount of an imaging device that has captured the image data;
Setting the strength of the cyclic noise reduction process;
Performing the cyclic noise reduction process on the image data;
Performing blurring correction processing for suppressing blurring of the image data subjected to the cyclic noise reduction processing based on the blurring amount;
If the shake amount is equal to or less than a first threshold, the strength of the cyclic noise reduction process is set in an area unit including one or more pixels based on the movement amount.
The image processing method, wherein the strength of the cyclic noise reduction process is uniformly set to all the areas of the image data when the blur amount is larger than the first threshold.   The shake correction process is not performed when the shake amount is smaller than a second threshold smaller than the first threshold, and is performed when the shake amount is equal to or larger than the second threshold. The image processing method according to claim 9. The settable range of the strength of the cyclic noise reduction process set in the area unit when the blur amount is equal to or less than the first threshold and equal to or more than the second threshold is:
The image processing according to claim 10, wherein the image processing is characterized in that it is narrower than the settable range of the strength of the cyclic noise reduction processing set in the area unit when the blur amount is equal to or less than the second threshold. Method.   12. The image processing method according to claim 11, wherein the settable range is set based on a table in which the blur amount is associated with the settable range. Motion amount detection means for detecting a motion amount from image data;
Shake calculating means for calculating the amount of shake of an imaging device that has captured the image data;
Processing means for performing cyclic noise reduction processing on the image data;
Control means for controlling the cyclic noise reduction process;
And correcting means for performing blurring correction processing for suppressing blurring of the image data subjected to the cyclic noise reduction processing based on the blurring amount.
The control means
When the shake correction process is not in operation, the strength of the cyclic noise reduction process is set in an area unit including one or more pixels based on the movement amount.
An image processing apparatus, wherein the strength of the cyclic noise reduction process is uniformly set in all the areas of the image data when the shake correction process is operating. Imaging means,
Motion amount detection means for detecting a motion amount from the image data generated by the imaging means;
Shake calculating means for calculating the amount of shake of an imaging device that has captured the image data;
Processing means for performing cyclic noise reduction processing on the image data;
Control means for controlling the cyclic noise reduction process;
And correcting means for performing blurring correction processing for suppressing blurring of the image data subjected to the cyclic noise reduction processing based on the blurring amount.
The control means
When the shake correction process is not in operation, the strength of the cyclic noise reduction process is set in an area unit including one or more pixels based on the movement amount.
When the shake correction process is operating, an intensity of the cyclic noise reduction process is set in units of regions based on the movement amount. Motion amount detection means for detecting a motion amount from image data;
Shake calculating means for calculating the amount of shake of an imaging device that has captured the image data;
Processing means for performing cyclic noise reduction processing on the image data;
Control means for controlling the cyclic noise reduction process;
And correcting means for performing blurring correction processing for suppressing blurring of the image data subjected to the cyclic noise reduction processing based on the blurring amount.
The control means
If the shake amount is equal to or less than a first threshold, the strength of the cyclic noise reduction process is set in an area unit including one or more pixels based on the movement amount.
An image processing apparatus, wherein the strength of the cyclic noise reduction process is uniformly set to all the areas of the image data when the shake amount is larger than the first threshold. Imaging means,
Motion amount detection means for detecting a motion amount from the image data generated by the imaging means;
Shake calculating means for calculating the amount of shake of an imaging device that has captured the image data;
Processing means for performing cyclic noise reduction processing on the image data;
Control means for controlling the cyclic noise reduction process;
And correcting means for performing blurring correction processing for suppressing blurring of the image data subjected to the cyclic noise reduction processing based on the blurring amount.
The control means
If the shake amount is equal to or less than a first threshold, the strength of the cyclic noise reduction process is set in an area unit including one or more pixels based on the movement amount.
An image pickup apparatus, wherein the strength of the cyclic noise reduction process is set in an area unit based on the movement amount when the shake amount is larger than the first threshold.   A program for causing a computer to execute the image processing method according to any one of claims 1 to 12.

Images