JP2016208307A - Image processing apparatus, control method therefor, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable noise reduction processing using time direction correlativity in combination with electronic zoom processing.SOLUTION: In the resizing unit 4, a digital image signal input via an image input terminal 401 is subjected to cutting out processing in accordance with the cut-out region of a current frame, zoom interpolation processing is performed, the resultant signal is output from the image output terminal 410, the digital image signal input via the image input terminal 401 is subjected to cutting out processing in accordance with a cut-out region of a next frame, zoom interpolation processing is performed, and the resultant signal is output from an image output terminal 411. A NR (noise reduction) processing unit 5 mixes the digital image signal input from the image output terminal 410 of the resizing unit 4 with a digital image signal obtained by delaying the digital image signal input from the image output terminal 411 of the resizing unit 4 by one frame time, on the basis of a difference absolute value.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus that performs noise reduction processing using temporal correlation on a moving image, a control method thereof, and a program.

  An image signal obtained from an imaging system such as a digital camera or a digital video camera generally includes a noise component. Among them, random noise is noise that is randomly generated mainly in the time direction and the spatial direction during high-sensitivity shooting, and in particular, the noise in the time direction is one of the factors that degrade the image quality of moving images. As a random noise reduction process in a moving image, a cyclic noise reduction process using temporal correlation is known.

  Also, when realizing an electronic zoom process for enlarging a video, a part of the image signal is cut out, and the image signal is electrically expanded by performing a pixel extension process and an interpolation process between adjacent pixels. . An effect like zooming can be obtained electrically by continuously changing the size of the cutout region.

Here, when the noise reduction process using the correlation in the time direction is realized in combination with the electronic zoom process, there is a problem that the correlation of data is lost between the previous and subsequent frames.
For example, in Patent Document 1, when the electronic zoom and the camera shake correction function are operated, the correlation between the data is lost between the input of the cyclic noise reduction circuit and the field memory output, so that the operation of the noise reducer is stopped. Is disclosed. This method can prevent malfunction of the cyclic noise reduction process such as afterimage and tailing, but the noise reduction effect cannot be obtained.

JP-A-6-46317

  The present invention has been made in view of the above points, and an object of the present invention is to realize a noise reduction process using correlation in the time direction in combination with an electronic zoom process.

  An image processing apparatus according to the present invention is an image processing apparatus that performs noise reduction processing using temporal correlation on a moving image, and extracts a previous frame of a target frame constituting the moving image from the target frame. Using a generating unit that generates an image for noise reduction processing by resizing in accordance with a region, and an image for noise reduction processing generated by the generating unit, noise is generated with respect to the image that has been resized the target frame. It is characterized by comprising noise reduction processing means for performing reduction processing.

  According to the present invention, the cut-out area of the previous frame is matched with the current frame, so that it is possible to realize a noise reduction process using correlation in the time direction in combination with the electronic zoom process.

It is a block diagram which shows the structure of the digital camera which concerns on 1st Embodiment. It is a block diagram which shows the structural example of the resizing part in 1st Embodiment. It is a block diagram which shows the structural example of the NR process part in 1st Embodiment. It is a figure for demonstrating the noise reduction process using the (epsilon) -filter in a spatial NR part. It is a figure for demonstrating the filter process in LPF. It is a diagram showing the relationship between the temporal NR coefficient K and the difference absolute value d _n. It is a block diagram which shows the structure of the digital camera which concerns on 2nd Embodiment. It is a block diagram which shows the structural example of the NR process part in 2nd Embodiment. It is a flowchart which shows the determination process of the method of the resizing process and noise reduction process by NR determination part in 2nd Embodiment.

Preferred embodiments of the present invention will be described below with reference to the accompanying drawings.
<First Embodiment>
In the first embodiment, in a digital camera that performs moving image shooting, an electronic zoom process that continuously changes the size of a cutout region at the center of an image and a combined noise reduction process that uses correlation in the time direction are performed. Propose.

(Configuration of digital camera)
FIG. 1 is a block diagram schematically showing the configuration of the digital camera according to the first embodiment. In this embodiment, the digital camera functions as an image processing apparatus to which the present invention is applied.
In a digital camera, light passing through an optical system 1 such as a diaphragm and a lens is photoelectrically converted by an image sensor 2 composed of a COMS, a CCD, or the like, and supplied to the first signal processing unit 3 as an analog image signal in pixel units. The
The first signal processing unit 3 performs A / D conversion, gain control, and the like on the analog image signal supplied from the imaging device 2 and supplies the analog image signal to the resizing unit 4 as a digital image signal.

The resizing unit 4 performs resizing processing on the digital image signal supplied from the first signal processing unit 3 by clipping and zoom interpolation processing, and supplies it to the noise reduction processing unit (NR processing unit) 5.
The NR processing unit 5 performs noise reduction processing using correlation in the time direction on the digital image signal subjected to the resizing process supplied from the resizing unit 4, and supplies the reduced signal to the second signal processing unit 6.

The second signal processing unit 6 performs color adjustment or the like on the digital image signal subjected to noise reduction processing supplied from the NR processing unit 5 and supplies the digital image signal to the output unit 7.
The output unit 7 outputs the digital image signal supplied from the second signal processing unit 6 to an output interface such as HDMI (registered trademark), records it on a recording medium such as a semiconductor memory card, and the display device ( At least one of the outputs to (not shown).

  The user interface unit (UI unit) 9 includes one or more input devices such as switches, buttons, and a touch panel provided on the display device, and is used by the user to give various settings and instructions to the digital camera. An instruction for switching between the moving image capturing mode and the still image capturing mode, an instruction for switching between the image capturing mode and the playback mode, an instruction for starting, pausing, and ending moving image capturing, GUI operation displayed on the display device, etc. To the digital camera.

The control unit 10 controls the operation of the entire digital camera. The control unit 10 includes, for example, a CPU, a RAM having a non-volatile area, and a ROM. The control program stored in the ROM is expanded in the RAM and executed by the CPU to control each component. Implement various functions. Setting values such as imaging parameters such as imaging frame rate and resolution set through the UI unit 9 are stored in, for example, a nonvolatile RAM included in the control unit 10.
Each component of the digital camera configured as described above is connected to be communicable with each other via the bus 8.

(Configuration of resizing unit 4)
FIG. 2 is a block diagram illustrating a configuration example of the resizing unit 4. Hereinafter, the configuration and operation of the resizing unit 4 will be described with reference to FIG.
The clipping unit 402 performs a clipping process on the digital image signal input from the first signal processing unit 3 via the image input terminal 401 and supplies the digital image signal to the zoom interpolation unit 403. The cutout process is performed according to the cutout region of the current frame input via the input terminal 406.
The zoom interpolation unit 403 performs zoom interpolation processing on the digital image signal cut out by the cut-out unit 402 and supplies the digital image signal to the image output terminal 410. The zoom interpolation process is performed according to the zoom magnification of the current frame input via the input terminal 406.
The clipping unit 404 performs clipping processing on the digital image signal input from the first signal processing unit 3 via the image input terminal 401 and supplies the digital image signal to the zoom interpolation unit 405. The cutout process is performed according to the cutout area of the next frame input via the input terminal 406.
The zoom interpolation unit 405 performs zoom interpolation processing on the digital image signal cut out by the cut-out unit 404 and supplies it to the image output terminal 411. The zoom interpolation process is performed in accordance with the zoom magnification of the next frame input via the input terminal 406. In this manner, the cutout unit 404 and the zoom interpolation unit 405 generate an image that is used when noise reduction processing using the correlation in the time direction is performed on the next frame.
The image output terminal 410 supplies the digital image signal supplied from the zoom interpolation unit 403 to the NR processing unit 5 as an output of the resizing unit 4. The image output terminal 411 supplies the digital image signal supplied from the zoom interpolation unit 405 to the NR processing unit 5 as an output of the resizing unit 4.

(Configuration of NR processing unit 5)
FIG. 3 is a block diagram illustrating a configuration example of the NR processing unit 5. Hereinafter, the configuration and operation of the NR processing unit 5 will be described with reference to FIG.
The spatial NR unit 502 performs a two-dimensional (spatial) noise reduction process on the digital image signal input from the image output terminal 410 of the resizing unit 4 via the image input terminal 501, and outputs an LPF (low pass filter). ) 503 and the mixing unit 510.
The spatial NR unit 505 performs two-dimensional (spatial) noise reduction processing on the digital image signal input from the image output terminal 411 of the resizing unit 4 via the image input terminal 504, and stores it in the frame memory 508. Supply.
The frame memory 508 delays the input digital image signal by one frame time and supplies it to the LPF 509 and the mixing unit 510.

The LPF 503 performs low-pass filter processing on the input digital image signal and supplies it to the absolute difference calculation unit 506.
The LPF 509 performs low-pass filter processing on the input delayed digital image signal and supplies it to the difference absolute value calculation unit 506.
The difference absolute value calculation unit 506 calculates a difference absolute value between the digital image signal subjected to the low-pass filter processing by the LPF 503 and the delayed digital image signal subjected to the low-pass filter processing by the LPF 509 to calculate a temporal NR coefficient. Part 507.
The temporal NR coefficient calculation unit 507 sets the temporal NR coefficient based on the difference absolute value calculated by the difference absolute value calculation unit 506 and supplies the temporal NR coefficient to the mixing unit 510.
The mixing unit 510 uses the digital image signal that has been subjected to noise reduction processing by the spatial NR unit 502 and the digital image signal that has been subjected to noise reduction processing by the spatial NR unit 505 and has been delayed, based on the temporal NR coefficient. And mix. And it supplies to the image output terminal 511 obtained by mixing.
The image output terminal 511 supplies the digital image signal supplied from the mixing unit 510 to the second signal processing unit 6 as an output of the NR processing unit 5.

(Resize processing by the resizing unit 4 and noise reduction processing by the NR processing unit 5)
Hereinafter, specific examples of the resizing process by the resizing unit 4 and the noise reduction process by the NR processing unit 5 will be described.
A moving image is sequentially input to the resizing unit 4 in units of frames via an image input terminal 401. For example, it is assumed that the input resolution is 4096 pixels × 2160 pixels and the output resolution is 1920 pixels × 1080 pixels. The imaging sensitivity (ISO sensitivity) is assumed to be 3200.

First, a case where a target frame (current frame) to which noise reduction processing is applied is input to the resizing unit 4 via the image input terminal 401.
The cutout unit 402 performs cutout processing on the current frame according to the cutout area of the current frame. For example, it is assumed that the cutout area of the current frame is 2880 pixels × 1620 pixels in the center of the image.
The zoom interpolation unit 403 performs zoom interpolation processing using a bicubic filter. In this example, since the input resolution is 2880 pixels × 1620 pixels and the output resolution is 1920 pixels × 1080 pixels, the zoom magnification applied to the previous frame is 0.66 times in both horizontal and vertical directions.
In addition, the cutout unit 404 performs cutout processing on the current frame according to the cutout area of the next frame. For example, the cutout area of the next frame is 3200 pixels × 1800 pixels in the center of the image.
The zoom interpolation unit 403 performs zoom interpolation processing using a bicubic filter. In this example, since the input resolution is 3200 pixels × 1800 pixels and the output resolution is 1920 pixels × 1080 pixels, the zoom magnification applied to the current frame is 0.60 times both horizontally and vertically.

Subsequently, the current frame that has been subjected to the resizing process in the resizing unit 4 is input to the NR processing unit 5 via the image input terminals 501 and 504.
Spatial NR units 502 and 505 perform noise reduction processing using an ε-filter. As shown in FIG. 4, when the pixel of interest and a x _n, reduce noise by performing smoothing using the pixel of interest and the pixel value of the pixel x _n-5 ~x _n + ₅ in its periphery. At this time, smoothing is performed only with a pixel value within a range of ± ε with respect to the pixel value of the target pixel x _n . In the example of FIG. 4, the pixel value of the pixel _{x n-5, x n-} 2, x n + 4 is not within the range of ± epsilon to the pixel value of the target pixel x _n. Therefore, the pixel _{values of} the pixels x _n-5 , x _n-2 , x _{n + 4} are replaced with the pixel values of the pixel of interest x _n , respectively, and a weighted average of the pixel values of the pixels x _{n-5 to} x _{n + 5} is obtained. Do. The calculation formulas (1) and (2) are shown below. a _k indicates a weight in each pixel, and is set to be smaller as it is geometrically farther from the target pixel x _n and larger as it is closer. Moreover, although the tap number is 11 here, it is not restricted to this.

  The current frame resized according to the position and size of the cutout area for the next frame and input via the image input terminal 504 is stored in the frame memory 508 after being subjected to noise reduction processing by the spatial NR unit 505. Is done.

The LPF 503 uses a median filter in a 3 × 3 region. It is to sort the 9 pixels in the 3 × 3 area, and outputs a central value as l _n. An image of the median filter is shown in FIG.
Similarly, the LPF 509 applies a median filter in the 3 × 3 region to a frame delayed by one frame read from the frame memory.
Difference absolute value calculator 506 calculates each pixel of the current frame supplied from the LPF 503, the absolute difference d _n between each pixel of a frame which is delayed is supplied from LPF509. Since the frame supplied from the LPF 509 is a frame delayed by one frame, it is the previous frame that has been cut and resized according to the size of the current frame. The calculation formula (3) is shown below. _Let the pixel value of the current frame be l _n and the pixel value of the previous frame be _pn .
d _n = | p _n −l _n | (3)

Temporal NR coefficient calculating unit 507, in accordance with the calculated difference absolute value calculator 506 absolute difference d _n, sets the temporal NR coefficient K. Here, when the difference absolute value d _n is 8 bits and is 0, 10, 20, 30, 40, 50, 60, 70, 120, 150, 180, the values of the respective temporal NR coefficients K are represented by K ₀ , K ₁₀ , K ₂₀ , K ₃₀ , K ₄₀ , K ₅₀ , K ₆₀ , K ₇₀ , K ₁₂₀ , K ₁₅₀ , K ₁₈₀ are set as follows.
K ₀ = 1
K ₁₀ = 1
K ₂₀ = 1
K ₃₀ = 1
K ₄₀ = 0.9
K ₅₀ = 0.45
K ₆₀ = 0.4
K ₇₀ = 0.3
K ₁₂₀ = 0.25
K ₁₅₀ = 0.2
K ₁₈₀ = 0

Further, values other than these values are calculated from neighboring values by linear interpolation. It shows the relationship between the temporal NR coefficient K and the difference absolute value d _n at this time is shown in FIG. With reference to FIG. 6, it calculates the value of the temporal NR coefficient K corresponding to the difference absolute value d _n.

Mixing unit 510, weighted by using the pixel value y _n of the current frame supplied from the spatial NR unit 502, a temporal NR coefficient K and the corresponding pixel value p _n in the previous frame supplied from the frame memory 508 Average. The calculation formula (4) is shown below.
o _n = K * p _n + (1−K) * y _n (4)

As described above, the position and size of the cut-out area of the previous frame mixed with the current frame are matched with the current frame. Thereby, only by preparing frame memories for the number of electronic zoom output pixels, combined noise reduction processing can be realized in combination with electronic zoom processing.
In the present embodiment, the electronic zoom process of changing the size of the cutout area at the center of the image at a constant rate has been described as an example, but the present invention is not limited to this. For example, an effect like panning can be obtained electrically by continuously changing the position of the cutout region. Even in this case, the present invention can be applied if it is known how the position and size of the cutout region change between the previous and next frames.

<Second Embodiment>
In the second embodiment, when performing digital zoom processing and noise reduction processing using temporal correlation (hereinafter also referred to as temporal NR) in a digital camera that performs moving image shooting, noise reduction is performed depending on shooting conditions. We propose to control the effect and the amount of frame memory. In the following, description of the contents common to the first embodiment will be omitted, and description will be made centering on differences from the first embodiment.
(Configuration of digital camera)
FIG. 7 is a block diagram schematically showing the configuration of the digital camera according to the second embodiment. In addition to the configuration of FIG. 1 described in the first embodiment, an NR determination unit 11 and a resize unit 12 subsequent to the NR processing unit 5 are added.
The NR determination unit 11 determines the resizing processing and noise reduction processing methods based on the moving image shooting conditions, and supplies control signals to the resizing unit 4, the NR processing unit 5, and the resizing unit 12. In FIG. 7, the NR determination unit 11 is shown as an independent block. However, for example, the control unit 10 may function as an NR determination unit.
The resizing unit 12 performs resizing processing on the digital image signal subjected to noise reduction processing supplied from the NR processing unit 5 by clipping and zoom interpolation processing, and supplies the resized processing to the signal processing unit 6. The configuration of the resizing unit 12 is the same as that of the resizing unit 4.

(Configuration of NR processing unit 5)
FIG. 8 is a block diagram illustrating a configuration example of the NR processing unit 5. An input terminal 512 and a selection unit 513 are added to the configuration of FIG. 3 described in the first embodiment.
Temporal NR coefficient calculation unit 507 sets a temporal NR coefficient based on the control signal input from input terminal 512 and the absolute difference value calculated by differential absolute value calculation unit 506, and mixing unit 510. To supply.
The selection unit 513 selects the output of the spatial NR unit 505 or the output of the mixing unit 510 and supplies it to the frame memory 508. Selection of the supply source to the frame memory 508 is performed according to a control signal input via the input terminal 512.

(Determination process by the NR determination unit 11)
FIG. 9 is a flowchart showing the determination process of the resizing process and the noise reduction process by the NR determination unit 11.
In the present embodiment, the input resolution and the output resolution are input and set from, for example, the UI unit 9. For example, it is assumed that the input resolution is 4096 pixels × 2160 pixels and the output resolution is 1920 pixels × 1080 pixels.
Further, the imaging frame rate is set by being input from the UI unit 9, for example. For example, it is assumed that selection is possible from 60 fps and 1 fps.
Further, the imaging sensitivity (ISO sensitivity) is input and set, for example, from the UI unit 9. For example, it is assumed that the user can select from 400, 3200, and 25600.
In addition, the current frame and the previous frame cut-out area in the electronic zoom process are input and set from the control unit 10, for example.

  In step S <b> 1, the NR determination unit 11 compares the imaging frame rate with the threshold 1 to determine whether temporal NR is applicable. The threshold 1 is a preset value, and is input from the control unit 10 via the bus 8, for example. For example, the threshold value 1 is set to 5 fps as a standard for reducing the correlation between frames. If the imaging frame rate is lower than 5 fps, the temporal NR is not applicable and the process proceeds to step S8. On the other hand, if the imaging frame rate exceeds 5 fps, the temporal NR is applicable and the process proceeds to step S2. That is, if the imaging frame rate is 1 fps, the temporal NR is not applicable and the process proceeds to step S8. On the other hand, if the imaging frame rate is 60 fps, the temporal NR is applicable and the process proceeds to step S2.

  In step S2, the NR determination unit 11 compares the ISO sensitivity with the threshold 2 to determine whether temporal NR is necessary. The threshold 2 is a preset value, and is input from the control unit 10 via the bus 8, for example. For example, the threshold value 2 is set to 800 as a standard ISO sensitivity that does not require noise reduction because the S / N of the moving image is sufficiently high. If the ISO sensitivity is less than 800, the time NR is unnecessary and the process proceeds to step S8. On the other hand, if the ISO sensitivity is 800 or more, a temporal NR is required and the process proceeds to step S3. That is, if the ISO sensitivity is 400, the temporal NR is unnecessary and the process proceeds to step S8. On the other hand, if the ISO sensitivity is 3200 or 25600, the temporal NR is necessary and the process proceeds to step S3.

  In step S <b> 3, the NR determination unit 11 compares the cut-out areas of the previous frame and the current frame, and determines the temporal NR method. When both the previous frame and current frame cut-out areas are 1920 pixels × 1080 pixels in the center of the image, the same resize processing is applied to the previous frame and the current frame. Therefore, it is possible to perform a cyclic noise reduction process using the resized image, and, similarly to the composite noise reduction process described in the first embodiment, frames for the number of electronic zoom output pixels. It can be realized with memory. Select the cyclic noise reduction processing because the cyclic noise reduction processing using the correlation of the past multiple frames has a higher noise reduction effect than the synthetic noise reduction processing using the correlation of the last two frames. The process proceeds to step S7. If the cutout area of the current frame is 2880 pixels × 1820 pixels in the center of the image and is different from the cutout area of 1920 pixels × 1080 pixels of the previous frame, the process proceeds to step S4.

  In step S <b> 4, the NR determination unit 11 compares the ISO sensitivity with the threshold 3 to determine the effect required for the temporal NR. The threshold 3 is a preset value, and is input from the control unit 10 via the bus 8, for example. In order to realize cyclic noise reduction processing, it is necessary to use an image signal before resizing, and a frame memory corresponding to the input resolution is always required. Therefore, the processing load is larger than that of the synthesis type noise reduction processing. . For example, the threshold value 3 is set to 12800 as a standard ISO sensitivity for which a low noise S / N of a moving image and a highly effective noise reduction process is required. If the ISO sensitivity exceeds 12800, the noise reduction effect is more important than the reduction in processing load, and it is determined that cyclic noise reduction processing needs to be applied, and the process proceeds to step S6. On the other hand, if the ISO sensitivity is 12800 or less, the reduction in processing load is more important than the noise reduction effect, and the process proceeds to step S5 because the combined noise reduction processing is suitable. That is, if the ISO sensitivity is 25600, the noise reduction effect is more important than the reduction in processing load, and the cyclic noise reduction process needs to be applied, and the process proceeds to step S6. On the other hand, if the ISO sensitivity is 6400, the reduction in processing load is more important than the noise reduction effect, and the process proceeds to step S5 because the combined noise reduction processing is suitable.

Hereinafter, the control of the resizing unit 4, the NR processing unit 5, and the resizing unit 12 in steps S5 to S8 will be described in detail.
(Control in step S5)
In step S5, the combined noise reduction processing described in the first embodiment is performed.
Specifically, a control signal is input to the selection unit 513 of the NR processing unit 5 via the input terminal 512, and the output of the spatial NR unit 505 is controlled to be supplied to the frame memory 508.
In addition, a control signal is input to the resizing unit 12 so that the cutout area is 4096 pixels × 2160 pixels at the center of the image and the resizing rate is 1 times, so that the resizing process is not performed.
Thereby, it becomes the structure equivalent to 1st Embodiment, and as demonstrated in 1st Embodiment, the position and size of the cutting-out area | region of the previous frame mixed with the present frame can be match | combined with the present frame. This makes it possible to achieve combined noise reduction processing in combination with electronic zoom processing while avoiding increased component costs and computational complexity simply by preparing frame memories for the number of electronic zoom output pixels. Can do.

(Control in step S6)
In step S6, cyclic noise reduction processing is performed before resizing.
Specifically, a control signal is input to the clipping units 402 and 404 and the zoom interpolation units 403 and 405 of the resizing unit 4 so that the clipping region is 4096 pixels × 2160 pixels centered on the image and the resizing rate is 1 ×. Then, control is performed so that the resizing process is not performed.
In addition, a control signal is input to the selection unit 513 of the NR processing unit 5 via the input terminal 512, and the output of the mixing unit 510 is controlled to be supplied to the frame memory 508. That is, the frame memory 508 holds a frame that has been subjected to cyclic noise reduction processing.

Processing in the resizing unit 4, the NR processing unit 5, and the resizing unit 12 controlled as described above will be described in more detail.
A moving image is sequentially input to the resizing unit 4 in units of frames via the image input terminal 401. However, since the resizing unit 4 does not substantially perform the resizing operation, the first signal processing is performed from the image output terminal 410. An image signal equivalent to the output of the unit 3 is output.
An image signal supplied from the image output terminal 410 is input to the image input terminal 501.
The spatial NR unit 502 performs the operation described in the first embodiment.
The selection unit 513 supplies the output of the mixing unit 510 to the frame memory 508.
The LPF 503, the difference absolute value calculation unit 506, and the temporal NR coefficient calculation unit 507 perform the operations described in the first embodiment.
The mixing unit 510 mixes the current frame output from the spatial NR unit 502 and the previous frame output from the frame memory 508 in the same manner as described in the first embodiment. As a result, the image signal subjected to the cyclic noise reduction processing is supplied to the image output terminal 511 and the selection unit 513.
The resizing unit 12 resizes the image signal that has been subjected to the cyclic noise reduction process supplied from the image output terminal 511.

  Accordingly, it is possible to realize a cyclic noise reduction process in combination with an electronic zoom process that continuously changes the size of the cutout region at the center of the image. In this case, a frame memory for the input resolution is required, but a high noise reduction effect can be expected.

(Control in step S7)
In step S7, cyclic noise reduction processing is performed after resizing.
A control signal is input to the selection unit 513 of the NR processing unit 5 via the input terminal 512, and the output of the mixing unit 510 is controlled to be supplied to the frame memory 508. That is, the frame memory 508 holds a frame that has been subjected to cyclic noise reduction processing.
Further, the resizing unit 12 is controlled not to perform the resizing process.

Processing in the resizing unit 4, the NR processing unit 5, and the resizing unit 12 controlled as described above will be described in more detail.
The cutout unit 402 performs cutout processing for all frames input from the image input terminal 401, assuming that the cutout region is 1920 pixels × 1080 pixels at the center of the image.
The zoom interpolation unit 403 performs zoom interpolation processing using a bicubic filter. However, since both input resolution and output resolution are equal to 1920 pixels × 1080 pixels, zoom processing is not substantially performed.
An image signal supplied from the image output terminal 410 is input to the image input terminal 501.
The spatial NR unit 502 performs the operation described in the first embodiment.
The selection unit 513 supplies the output of the mixing unit 510 to the frame memory 508.
The LPF 503, the difference absolute value calculation unit 506, and the temporal NR coefficient calculation unit 507 perform the operations described in the first embodiment.
The mixing unit 510 mixes the current frame output from the spatial NR unit 502 and the previous frame output from the frame memory 508 in the same manner as described in the first embodiment. As a result, the image signal subjected to the cyclic noise reduction processing is supplied to the image output terminal 511 and the selection unit 513.
Although the moving image is sequentially input to the resizing unit 12 in units of frames, the resizing unit 12 does not substantially perform the resizing operation, and therefore outputs an image signal equivalent to the output of the NR processing unit 5.

  As a result, when the cut-out area is constant between the previous frame and the current frame, it is possible to realize a cyclic noise reduction process by using the electronic zoom process together with the frame memory for the electronic zoom output resolution. Although not applicable when the cutout region changes between frames, a high noise reduction effect can be expected at low cost when the cutout region is constant.

(Control in step S8)
In step S8, the noise reduction process using the correlation in the time direction is invalidated and only the resizing process is performed.
Specifically, a control signal is input to the resizing unit 4 so that the cutout area is 4096 pixels × 2160 pixels at the center of the image and the resizing rate is 1 time, and control is performed so that resizing processing is not performed.
Further, a control signal is input to the temporal NR coefficient calculation unit 507 of the NR processing unit 5 via the input terminal 512, and control is performed so as to output the coefficient K = 0.

Processing in the resizing unit 4, the NR processing unit 5, and the resizing unit 12 controlled as described above will be described in more detail.
A moving image is sequentially input to the resizing unit 4 in units of frames via the image input terminal 401. However, since the resizing unit 4 does not substantially perform the resizing operation, the first signal processing is performed from the image output terminal 410. An image signal equivalent to the output of the unit 3 is output.
An image signal supplied from the image output terminal 410 is input to the image input terminal 501.
The spatial NR unit 502 performs the operation described in the first embodiment.
The absolute difference calculation unit 506 supplies the temporal NR coefficient K = 0 to the mixing unit 510.
The mixing unit 510 performs a weighted average using the temporal NR coefficient K based on the equation (4). However, since K = 0 is supplied, the output of the spatial NR unit 502 is output to the image output terminal 511 as it is. .
The resizing unit 12 resizes the image signal supplied from the image output terminal 511 by performing the same operation as the resizing unit 4 described in the first embodiment.

  Thereby, it is possible to invalidate the noise reduction process using the correlation in the time direction and perform only the resizing process. Although a noise reduction effect cannot be obtained, a cost reduction can be expected because a frame memory becomes unnecessary.

  As described above, by controlling the resizing process and the noise reduction process according to the shooting conditions, the noise reduction effect and the frame memory amount can be automatically controlled according to the shooting conditions when performing the electronic zoom process and the temporal NR. it can.

Although the present invention has been described together with the embodiments, the above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention is interpreted in a limited manner by these. It must not be. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.
(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

2: Image sensor 4, 12: Resizing unit 5: NR processing unit 10: Control unit 11: NR determining unit 402, 404: Clipping unit 403, 405: Zoom interpolating unit 507: Temporal NR coefficient calculating unit 508: Frame memory 510 : Mixing unit 513: Selection unit

Claims (9)

An image processing apparatus that performs noise reduction processing using correlation in a time direction on a moving image,
Generating means for resizing a previous frame of a target frame constituting a moving image in accordance with a cutout region for the target frame to generate an image for noise reduction processing;
An image processing apparatus comprising: noise reduction processing means for performing noise reduction processing on an image obtained by resizing the target frame using an image for noise reduction processing generated by the generation means.   The image processing apparatus according to claim 1, wherein the noise reduction processing unit mixes pixels of an image obtained by resizing the target frame and pixels of an image for the noise reduction processing. An image processing apparatus that performs noise reduction processing using correlation in a time direction on a moving image,
First resizing means for resizing an image of a frame constituting a moving image;
Noise reduction processing means for performing noise reduction processing using correlation in the time direction on the image of the frame supplied from the first resizing means;
Second resizing means for resizing the image of the frame supplied from the noise reduction processing means;
An image processing apparatus comprising: a control unit that controls the first resizing unit, the noise reduction processing unit, and the second resizing unit according to a shooting condition of the moving image. The noise reduction processing means includes
Holding means for holding an image of a frame used for noise reduction processing of the target frame;
Mixing means for performing noise reduction processing by mixing pixels of the image of the target frame and pixels of the image of the frame held by the holding means;
4. The image processing apparatus according to claim 3, further comprising: a selection unit that selects an image supplied from the first resizing unit and an image supplied from the mixing unit as an image held by the holding unit. Image processing apparatus. The control means includes
The first resizing means resizes the previous frame of the target frame in accordance with a cutout area for the target frame,
The selection unit of the noise reduction processing unit holds the resized image of the previous frame supplied from the first resizing unit by the holding unit,
The first resizing means resizes the target frame with a cutout region for the target frame,
The pixel of the image obtained by resizing the target frame supplied from the first resizing unit and the pixel of the image obtained by resizing the previous frame held by the holding unit are mixed by the mixing unit of the noise reduction processing unit. To reduce noise,
The image processing apparatus according to claim 4, wherein the second resizing unit is controlled so that the resizing process is not performed. The control means includes
The first resizing means prevents the resizing process from being performed,
The selection unit of the noise reduction processing unit holds the image supplied from the mixing unit by the holding unit,
The mixing unit of the noise reduction processing unit mixes the pixel of the image of the target frame supplied from the first resizing unit and the pixel of the image obtained by performing noise reduction processing on the previous frame held by the holding unit. Noise reduction processing,
6. The image processing apparatus according to claim 4, wherein the second resizing unit resizes the target frame supplied from the mixing unit in a cutout region for the target frame. The control means includes
The first resizing means resizes the target frame with a cutout region for the target frame,
The selection unit of the noise reduction processing unit holds the image supplied from the mixing unit by the holding unit,
By means of the mixing means of the noise reduction processing means, the pixels of the image resized the target frame supplied from the first resizing means, and the image of the image subjected to noise reduction processing by resizing the previous frame held by the holding means Perform noise reduction processing by mixing with pixels,
The image processing apparatus according to claim 4, wherein the second resizing unit is controlled so that the resizing process is not performed. A control method of an image processing apparatus that performs noise reduction processing using correlation in a time direction for a moving image,
Resizing a previous frame of a target frame constituting a moving image in accordance with a cutout region for the target frame to generate an image for noise reduction processing;
And a step of performing noise reduction processing on the resized image of the target frame using the generated image for noise reduction processing. A program for controlling an image processing apparatus that performs noise reduction processing using correlation in a time direction for a moving image,
A process of generating an image for noise reduction processing by resizing a previous frame of a target frame constituting a moving image in accordance with a cutout region for the target frame;
A program for causing a computer to execute processing for performing noise reduction processing on an image obtained by resizing the target frame using the generated image for noise reduction processing.

Images