JP5653674B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus.

  When performing frame correlation NR (Noise Reduction, noise reduction) processing in an FNR (Frame Noise Reduction) block, image data of the current frame to be processed and image data (cyclic image) taken before that Data).

  Further, a technique for performing noise reduction processing using Y / Cb / Cr of image data of the current frame and image data of the previous frame is known (see, for example, Patent Document 1). In this technique, for each Y / Cb / Cr, the difference between the target pixel A of the image data of the current frame and the target pixel B of the cyclic image data corresponding to the same position as the target pixel A, and the periphery of the target pixel A The difference between the pixel and the peripheral pixel of the target pixel B is calculated, a weighting coefficient for the target pixel A is obtained from the difference data, and noise reduction processing is performed on the image data of the current frame based on the weighting coefficient.

JP 2001-8228 A

  However, in the above-described method, in order to perform noise reduction processing of the image data of the current frame, a memory area for storing YCbCr image data having the same image size as that of the image data of the current frame is secured. There is a need to.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus capable of reducing a necessary memory area.

The present invention includes an imaging device for converting an object image into an electrical signal, and an image processing unit for generating image data from said electric signal, and a display unit for displaying a through image, one frame prior to the image data of the noise removal target A cyclic image generating unit that generates cyclic image data by reducing the image size of the image data, a cyclic image recording unit that records the cyclic image data, and an enlargement of the cyclic image data recorded by the cyclic image recording unit A cyclic enlarged image generating unit that generates cyclic enlarged image data, and a noise removing unit that performs noise removal of the image data to be noise-removed using the cyclic enlarged image data, and the display unit includes: The image processing apparatus displays the cyclic image data as the through image .

  In the image processing device of the present invention, the cyclic enlarged image generation unit generates the cyclic enlarged image data by expanding the image size of the cyclic image data to the same size as the image size of the image data. It is characterized by.

  In the image processing apparatus of the present invention, the noise removing unit changes the noise removal strength of the image data based on a correlation between the cyclic enlarged image data and the image data.

  In the image processing apparatus of the present invention, the noise removal unit increases the noise removal strength of the image data as the correlation between the cyclic enlarged image data and the image data is higher, and the cyclic enlarged image data. The lower the correlation between the image data and the image data, the weaker the noise removal strength of the image data.

  In the image processing apparatus of the present invention, the noise removal unit changes the noise removal strength of the image data based on a ratio between the image size of the cyclic image data and the image size of the cyclic enlarged image data. It is characterized by doing.

  In the image processing apparatus of the present invention, the noise removal unit increases the noise removal strength of the image data as the ratio between the image size of the cyclic image data and the image size of the cyclic enlarged image data increases. The smaller the ratio between the image size of the cyclic image data and the image size of the cyclic enlarged image data, the weaker the noise removal strength of the image data.

  According to the present invention, the cyclic image generation unit generates cyclic image data by reducing the image size of the image data one frame before the image data to be denoised. The cyclic image recording unit records cyclic image data. The cyclic enlarged image generation unit generates cyclic enlarged image data by expanding the cyclic image data recorded by the cyclic image recording unit. Further, the noise removing unit removes noise from the image data to be removed using the cyclically enlarged image data. As a result, the size of data recorded by the cyclic image recording unit is reduced, so that a necessary memory area can be reduced.

1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention. It is the flowchart which showed the noise removal processing procedure of the image processing apparatus in the 1st Embodiment of this invention. It is the schematic which showed the example of arrangement | positioning of the attention pixel in the 1st Embodiment of this invention. It is the block diagram which showed the structure of the image processing apparatus in the 2nd Embodiment of this invention. It is the flowchart which showed the noise removal processing procedure of the image processing apparatus in the 2nd Embodiment of this invention.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of an image processing apparatus 1 in the present embodiment. In the illustrated example, the image processing apparatus 1 includes an imaging optical system 101, an imaging element 102, an image processing unit 103, an NR processing unit 104 (noise removal unit), and a resizing unit 105 (cyclic image generation unit, cyclic enlargement). An image generation unit), a DMA (Direct Memory Access) unit 106, a cyclic image recording unit 107, a recording unit 108, and a display unit 109.

  The imaging optical system 101 focuses light from the subject on the imaging surface of the imaging element 102. The image sensor 102 converts the formed image into an electric signal according to timing control from a control unit (not shown) and outputs the electric signal to the image processing unit 103. The image processing unit 103 performs image processing for improving image quality such as γ correction, color separation, and color matrix, and output format conversion processing on the electrical signal output from the image sensor 102, and converts the image data into Generate. The NR processing unit 104 performs NR processing on the image data. The resizing unit 105 reduces the image data, and generates through image data and thumbnail image data. In the present embodiment, live view image data and thumbnail image data are used as cyclic image data. In addition, the resizing unit 105 enlarges the image size of the cyclic image data (the size of the image and the number of pixels) to the same size as the image size of the image data, and generates cyclic enlarged image data. It is assumed that the image size of the cyclic image data (through image data or thumbnail image data) is smaller than the image size of the image data.

  The DMA unit 106 reads and writes (R / W) data from and to the cyclic image recording unit 107 and the recording unit 108. The cyclic image recording unit 107 records cyclic image data. The cyclic image data in this embodiment is data having an image size smaller than that of the image data. The recording unit 108 is a temporary storage area, an external recording medium, or the like, and temporarily records data necessary for processing, image data, and the like. The display unit 109 is a liquid crystal display or the like, and displays an image based on the image data.

  Next, the noise removal processing (FNR) procedure of the image processing apparatus 1 will be described. FIG. 2 is a flowchart illustrating a noise removal processing procedure of the image processing apparatus 1 according to the present embodiment. This flowchart shows a processing procedure for image data for each frame. Note that, as processing before the noise removal processing, image data is generated by the imaging optical system 101, the imaging device 102, and the image processing unit 103, and is temporarily recorded in the recording unit 108. Specifically, an optical image obtained through the imaging optical system 101 is picked up by the image pickup element 102 and converted into an electrical signal, the image processing unit performs processing such as high image quality and output format conversion, and the DMA unit 106. Is temporarily recorded in the recording unit 108.

(Step S <b> 101) The NR processing unit 104 acquires image data to be processed from the recording unit 108 via the DMA unit 106. Thereafter, the process proceeds to step S102.
(Step S102) The resizing unit 105 determines whether or not the cyclic image data is recorded in the cyclic image recording unit 107. If the resizing unit 105 determines that the cyclic image data is recorded in the cyclic image recording unit 107, the process proceeds to step S103. Otherwise, the process proceeds to step S112.

(Step S <b> 103) The resizing unit 105 acquires the cyclic image data from the cyclic image recording unit 107 via the DMA unit 106. Thereafter, the process proceeds to step S104.
(Step S104) The resizing unit 105 enlarges the cyclic image data read in step S103 so that the image size of the cyclic image data is the same as the image size of the image data, and generates cyclic enlarged image data. Thereafter, the process proceeds to step S105.

  (Step S105) The NR processing unit 104 calculates difference data from the target pixel of the image data acquired in step S101 and its peripheral pixels, and the target pixel of the cyclically enlarged image data generated in step S104 and its peripheral pixels. Thereafter, the process proceeds to step S106. The difference data will be described later.

(Step S106) The NR processing unit 104 compares the difference data acquired in Step S105 with a predetermined threshold value. If the difference data is less than the threshold, the process proceeds to step S107. If the difference data is equal to or greater than the threshold, the process proceeds to step S108.
(Step S107) The NR processing unit 104 sets the weighting coefficient (LPF coefficient) indicating the strength of NR processing (LPF (Low-pass filter) processing) to “large”. Thereafter, the process proceeds to step S109.
(Step S108) The NR processing unit 104 sets the weighting coefficient (LPF coefficient) indicating the strength of the NR processing (LPF processing) to “small”. Thereafter, the process proceeds to step S109.

  (Step S109) The NR processing unit 104 performs NR processing on the image data acquired in step S101 using the weighting factor determined in step S107 or step S108, and generates post-NR processing image data. Thereafter, the process proceeds to step S110.

(Step S110) The resizing unit 105 reduces the NR-processed image data generated by the NR processing unit 104 in step S109, and generates cyclic image data. Thereafter, the process proceeds to step S111.
(Step S111) The NR processing unit 104 causes the recording unit 108 to record the post-NR processing image data generated in step S109 via the DMA unit 106. Thereafter, the process proceeds to step S113.
(Step S112) The resizing unit 105 reduces the image data acquired by the NR processing unit 104 in step S101, and generates cyclic image data. Thereafter, the process proceeds to step S113.

  (Step S113) The resize unit 105 causes the cyclic image recording unit 107 to record the cyclic image data generated in step S110 or step S112 via the DMA unit. In the case where the cyclic image recording unit 107 has already recorded the cyclic image data, the cyclic image data newly generated by the resizing unit 105 is recorded after erasing the already recorded cyclic image data. Thereafter, the process ends.

  After performing the above-described noise removal processing, the display unit 109 reads out and displays the NR-processed image data recorded in the recording unit 108 via the DMA unit 106.

  Next, a method for calculating difference data will be described. The difference data is a value indicating the correlation between the image data and the cyclic enlarged image data. The lower the value, the higher the correlation between the image data and the cyclic enlarged image data.

  FIG. 3A is a schematic diagram illustrating an arrangement example of the target pixel of the image data. FIG. 3B is a schematic diagram illustrating an arrangement example of the target pixel of the cyclic enlarged image data. In the example shown in FIG. 3A, the pixel a4 is shown as the target pixel of the image data, and the pixels a0 to a3 and a5 to a8 are shown as the peripheral pixels of the target pixel a4. In the example illustrated in FIG. 3B, the pixel b4 is illustrated as the target pixel of the cyclically enlarged image data, and the pixels b0 to b3 and b5 to b8 are illustrated as the peripheral pixels of the target pixel b4.

When the arrangement of the pixel of interest and the surrounding pixels is the arrangement shown in the figure, the difference data is calculated by the following equation.
Difference data = | (a0−b0) | + | (a1−b1) | + | (a2−b2) | + | (a3−b3) | + | (a4−b4) | + | (a5−b5) | + | (A6-b6) | + | (a7-b7) | + | (a8-b8) |

  When the calculated difference data value is large, since the correlation between the image data and the cyclically enlarged image data at the target pixel is low, the weighting coefficient (LPF coefficient) is reduced and the effect of NR processing (LPF processing) on the image data is increased. To weaken. Further, when the calculated difference data value is small, the correlation between the image data at the target pixel and the cyclically enlarged image data is high. Therefore, the weight coefficient (LPF coefficient coefficient) is increased, and NR processing (LPF processing) for the image data is performed. ).

  As described above, according to the present embodiment, the image size of the cyclic image data recorded in the cyclic image recording unit 107 is made smaller than the image size of the image data. Further, when performing the NR process, enlargement processing is performed so that the image size of the cyclic image data is the same as the image size of the image data, and cyclic enlarged image data is generated. Then, NR processing is performed using the generated cyclic enlarged image data. Thereby, since the image size of the cyclic image data to be recorded in the cyclic image recording unit 107 is reduced, the memory area of the cyclic image recording unit 107 can be reduced.

  Further, according to the present embodiment, the correlation between the image data and the cyclically enlarged image data is calculated, and when the correlation between the image data and the cyclically enlarged image data is high, the effect of the NR process on the image data is strongly increased. To do. Further, when the correlation between the image data and the cyclically enlarged image data is low, the effect of the NR process on the image data is weakened. Thereby, NR with higher accuracy can be performed.

(Second Embodiment)
Next, a second embodiment of the present invention will be described with reference to the drawings. FIG. 4 is a block diagram showing a configuration of the image processing apparatus 2 in the present embodiment. In the illustrated example, the image processing apparatus 2 includes an imaging optical system 101, an imaging element 102, an image processing unit 103, an NR processing unit 104, a resizing unit 105, a DMA unit 106, and a cyclic image recording unit 107. A recording unit 108, a display unit 109, and a threshold value calculation unit 110.

  An imaging optical system 101, an image sensor 102, an image processing unit 103, an NR processing unit 104, a resizing unit 105, a DMA unit 106, a cyclic image recording unit 107, a recording unit 108, and a display unit 109 The operation is the same as the operation of each unit in the first embodiment. The threshold calculation unit 110 calculates a threshold for determining the strength of the NR process. Note that the threshold calculation unit 110 calculates a threshold according to the resizing rate when the resizing unit 105 changes the image size.

  Next, a threshold value calculation method for determining the strength of NR processing will be described. In general, after resizing and reducing the image data, the image data that has been enlarged and resized to the original image size is the same as the original image data because the missing information is interpolated by interpolation processing by enlargement. It is not image data. When the resizing rate is high (the reduction rate and the enlargement rate are large), the amount of information missing from the original image data is large, and the amount of data to be interpolated increases. Therefore, the difference data between the cyclic enlarged image data having a high resizing rate and the image data tends to be large.

  Therefore, a difference occurs in the effect of the NR process due to the difference in the resizing rate of the cyclic enlarged image data. Specifically, when the resize rate of the cyclic enlarged image data is high, the amount of information missing due to resizing is large. For this reason, the difference data between the cyclic enlarged image data and the image data becomes large. That is, the correlation between the cyclic enlarged image data and the image data is low. Therefore, the effect of NR processing is reduced. Further, when the resizing rate of the cyclic enlarged image data is low, the amount of information missing due to resizing is small. Therefore, the difference data between the cyclic enlarged image data and the image data becomes small. That is, the correlation between the cyclically enlarged image data and the image data is high. Therefore, the effect of NR processing is enhanced.

  As described above, the difference in resizing rate causes a difference in the effectiveness of the NR process. Therefore, in order to suppress the difference in the NR process, the threshold value is changed, and the weighting coefficient (LPF coefficient) indicating the intensity of the NR process (LPF process) is changed. Specifically, when the resizing rate of the cyclic enlarged image data is high, the difference data between the cyclic enlarged image data and the image data is large, and the effect of the NR process is low. Therefore, by increasing the threshold value, the weighting factor is increased to increase the effectiveness of the NR processing. Further, when the resize ratio of the cyclic enlarged image data is low, the difference data between the cyclic enlarged image data and the image data is small, and the effect of the NR process is high. Therefore, by reducing the threshold value, the weighting coefficient is reduced to reduce the effectiveness of the NR processing.

  Next, the noise removal processing (FNR) procedure of the image processing apparatus 2 will be described. FIG. 5 is a flowchart showing a noise removal processing procedure of the image processing apparatus 2 in the present embodiment. This flowchart shows a processing procedure for image data for each frame. The process before the noise removal process is the same as the process in the first embodiment.

(Step S201) to (Step S203) The processing from step S201 to step S203 is the same as the processing from step S101 to step S103 in the first embodiment.
(Step S204) The resizing unit 105 enlarges the cyclic image data read out in step S203 so that the image size of the cyclic image data is the same as the image size of the image data, and generates cyclic enlarged image data. In addition, the resizing unit 105 inputs information indicating an enlargement rate during the enlargement process to the threshold value calculating unit 110. Thereafter, the process proceeds to step S205.

  (Step S205) The threshold value calculation unit 110 calculates a threshold value based on the information indicating the enlargement ratio input in step S204. For example, when the enlargement ratio is large, the threshold is increased, and when the enlargement ratio is low, the threshold is decreased. Thereafter, the process proceeds to step S206.

(Step S206) The process of step S206 is the same as the process of step S105 in the first embodiment.
(Step S207) The NR processing unit 104 compares the difference data acquired in Step S206 with the threshold value calculated in Step S205. If the difference data is less than the threshold value, the process proceeds to step S208. If the difference data is equal to or greater than the threshold value, the process proceeds to step S209.

  (Step S208) to (Step S214) The processing of step S208 to step S214 is the same as the processing of step S107 to step S113 in the first embodiment.

  After performing the above-described noise removal processing, the display unit 109 reads out and displays the NR-processed image data recorded in the recording unit 108 via the DMA unit 106.

  As described above, according to the present embodiment, the image size of the cyclic image data recorded in the cyclic image recording unit 107 is made smaller than the image size of the image data. Further, when performing the NR process, enlargement processing is performed so that the image size of the cyclic image data is the same as the image size of the image data, and cyclic enlarged image data is generated. Then, NR processing is performed using the generated cyclic enlarged image data. Thereby, since the image size of the cyclic image data to be recorded in the cyclic image recording unit 107 is reduced, the memory area of the cyclic image recording unit 107 can be reduced.

  Further, according to the present embodiment, the correlation between the image data and the cyclically enlarged image data is calculated, and when the correlation between the image data and the cyclically enlarged image data is high, the effect of the NR process on the image data is strongly increased. To do. Further, when the correlation between the image data and the cyclically enlarged image data is low, the effect of the NR process on the image data is weakened. Thereby, NR with higher accuracy can be performed.

  Further, according to the present embodiment, the threshold calculation unit 110 calculates a threshold for determining the strength of the NR process according to the resizing rate of the cyclically enlarged image data. Thereby, it is possible to adjust the effectiveness of the NR processing due to the difference in the resizing rate of the cyclic enlarged image data.

  The first embodiment and the second embodiment of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and does not depart from the gist of the present invention. Range design etc. are also included.

DESCRIPTION OF SYMBOLS 1, 2 ... Image processing apparatus, 101 ... Imaging optical system, 102 ... Imaging element, 103 ... Image processing part, 104 ... NR processing part, 105 ... Resize part, 106. .. DMA unit 107... Cyclic image recording unit 108... Recording unit 109... Display unit 110.

Claims (6)

An image sensor for converting a subject image into an electrical signal;
An image processing unit for generating image data from the electrical signal;
A display unit for displaying a through image;
A cyclic image generating unit that reduces the image size of the image data of one frame earlier than the image data of the noise removal target to generate the cyclic image data,
A cyclic image recording unit for recording the cyclic image data;
A cyclic enlarged image generating unit that generates the cyclic enlarged image data by enlarging the cyclic image data recorded by the cyclic image recording unit;
Using the cyclic enlarged image data, a noise removing unit that removes noise of the image data to be removed from noise,
Equipped with a,
The display unit displays the traveling image data as the through image.
An image processing apparatus. The said cyclic | annular expansion image production | generation part expands the image size of the said cyclic | annular image data to the same size as the image size of the said image data, and produces | generates the said cyclic | annular expansion image data. Image processing device. The image processing apparatus according to claim 1, wherein the noise removal unit changes a noise removal strength of the image data based on a correlation between the cyclic enlarged image data and the image data. The noise removal unit increases the noise removal strength of the image data as the correlation between the cyclic enlarged image data and the image data is high, and the correlation between the cyclic enlarged image data and the image data is low. The image processing apparatus according to claim 3, wherein the intensity of noise removal of the image data is weakened. The noise removal unit changes the noise removal strength of the image data based on a ratio between an image size of the cyclic image data and an image size of the cyclic enlarged image data. The image processing apparatus described. The noise removal unit increases the noise removal strength of the image data as the ratio between the image size of the cyclic image data and the image size of the cyclic enlarged image data increases, and the image size of the cyclic image data The image processing apparatus according to claim 5, wherein as the ratio of the cyclic enlarged image data to the image size is smaller, the noise removal strength of the image data is reduced.

Images