JP2013225779A - Image processing device, imaging device, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique capable of reliably removing low-frequency noise from an image.SOLUTION: An image processing device comprises: an image input unit that inputs each of a plurality of temporally consecutively captured images as a target image; an image reduction unit that reduces the target image to generate a reduced image; a frame memory that stores a reference image for noise removal processing; a reduced image noise removal unit that removes low-frequency noise from the reduced image while referring to the reference image; and a target image noise removal unit that removes low-frequency noise from the target image on the basis of the reduced image from which the low-frequency noise is removed.

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

  Conventionally, various techniques for removing noise from a moving image frame have been developed.

  For example, there is a technique for inputting a current image of a moving image and an image one frame before, obtaining a difference between frames, determining whether noise is present according to the magnitude of the difference, and removing noise from the current image ( (See Patent Document 1).

JP 2008-113352 A

  However, in the prior art, noise is removed from the current image based only on the difference between the current image and the image one frame before. Therefore, a removal effect is obtained for low frequency noise having a spatial wavelength larger than the size of the current image. There is a problem that it is not enough.

  In view of the problems of the above-described conventional technology, an object of the present invention is to provide a technology that can remove low-frequency noise from an image with high accuracy.

  In order to solve the above-described problem, an aspect of an image processing apparatus illustrating the present invention includes an image input unit that inputs each of a plurality of images captured continuously in time as a target image, and a reduction in the target image. An image reduction unit that generates a reduced image, a frame memory that stores a reference image for noise removal processing, a reduced image noise removal unit that removes low-frequency noise from the reduced image while referring to the reference image, and a low-frequency noise A target image noise removing unit that removes low frequency noise from the target image based on the removed reduced image.

  The frame memory may store a reduced image from which low frequency noise has been removed by the reduced image noise removing unit as a reference image.

  The reduced image noise removing unit may remove low-frequency noise from the pixels of the reduced image while referring to an image area around the pixels of the reference image corresponding to the pixels of the reduced image.

  One aspect of an imaging apparatus illustrating the present invention includes an imaging unit that captures an image of a subject and generates an image, and the image processing apparatus of the present invention.

  The reduced image noise removing unit includes an operation member, and when the still image mode is set as the shooting mode by the user, the reduced image noise removing unit refers to the reduced image generated by the image reducing unit as a reference image, from the reduced image. Low frequency noise may be removed.

  In addition, a reference image noise removing unit that removes noise from the reduced image generated by the image reducing unit and records it in the frame memory as a reference image may be provided.

  One aspect of an image processing program that exemplifies the present invention is an image input procedure for inputting each of a plurality of images captured continuously in time as a target image, and an image reduction procedure for generating a reduced image by reducing the target image. , Reduced image noise removal procedure for removing low frequency noise from the reduced image while referring to the reference image of the noise removal processing stored in the frame memory, low frequency noise from the target image based on the reduced image from which the low frequency noise has been removed To cause the computer to execute a target image noise removal procedure for removing the target image noise.

  According to the present invention, low frequency noise can be accurately removed from an image.

The figure which shows an example of a structure of the digital camera which concerns on one Embodiment of this invention. The figure which shows an example of the flow of the image data in the noise removal process in the case of the moving image mode by the digital camera concerning one embodiment The flowchart which shows an example of the noise removal process of FIG. The flowchart which shows an example of the noise removal process in the case of still image mode by the digital camera which concerns on other embodiment of this invention. The figure which shows an example of the flow of the image data in the noise removal process shown in FIG. The figure which shows an example of a structure of the digital camera which concerns on the modification of other embodiment of this invention. The figure which shows an example of the flow of the image data of the noise removal process in the modification of other embodiment

  FIG. 1 shows an example of the configuration of a digital camera 100 according to an embodiment of the present invention.

  The digital camera 100 according to this embodiment includes an imaging optical system 11, an imaging device 12, an AFE 13, a frame memory 14, a monitor 15, a storage unit 16, a recording I / F 17, a CPU 19, an operation member 23, and a bus 24. The frame memory 14, the monitor 15, the storage unit 16, the recording I / F 17, and the CPU 19 are connected to each other via a bus 24 so that information can be transmitted. The operation member 23 is connected to the CPU 19.

  The imaging optical system 11 includes a plurality of lens groups including a zoom lens and a focusing lens. The lens position of the imaging optical system 11 is adjusted in the optical axis direction by a lens driving unit (not shown). For simplicity, the imaging optical system 11 is illustrated as a single lens in FIG.

  The image sensor 12 is a device that captures a subject image formed by a light beam that has passed through the imaging optical system 11. The output of the image sensor 12 is connected to the AFE 13. Note that the image sensor 12 of the present embodiment may be a progressive scanning solid-state image sensor (CCD or the like) or an XY address type solid-state image sensor (CMOS or the like).

  A plurality of light receiving elements are arranged in a matrix on the light receiving surface of the imaging element 12. In each light receiving element of the image sensor 12, red (R), green (G), and blue (B) color filters are arranged according to a known Bayer array. Therefore, each light receiving element of the imaging element 12 outputs an image signal corresponding to each color by color separation in the color filter. Thereby, the image sensor 12 can acquire a color image at the time of imaging.

  The AFE 13 is an analog front end circuit that performs analog signal processing on the output of the image sensor 12. The AFE 13 performs correlated double sampling, image signal gain adjustment, and image signal A / D conversion. The output of the AFE 13 is temporarily recorded in the frame memory 14. In the present embodiment, the imaging device 12 and the AFE 13 constitute an imaging unit. Further, as will be described later, the frame memory 14 of the present embodiment is a low-frequency noise image that is one frame before the image together with the image data of the frame imaged by the imaging unit when the imaging mode is the moving image mode. The reduced image from which is removed is also temporarily stored. As the frame memory 14, a general volatile semiconductor memory or the like can be appropriately selected and used.

  The monitor 15 is a display unit such as a liquid crystal monitor, and displays various images in accordance with instructions from the CPU 19. That is, based on an instruction from the CPU 19, a captured image is displayed, and buttons in a GUI (Graphical User Interface) format that allows input of various information necessary for imaging and various setting items are superimposed and displayed.

  The storage unit 16 stores a file of an image captured by the digital camera 100 together with a control program and an image processing program executed by the CPU 19. As the storage unit 16, a nonvolatile semiconductor memory or the like can be used.

  A connector for connecting the storage medium 18 is formed in the recording I / F 17. The recording I / F 17 executes data writing / reading with respect to the storage medium 18 connected to the connector. The storage medium 18 is composed of a hard disk, a memory card incorporating a semiconductor memory, or the like. In FIG. 1, a memory card is shown as an example of the storage medium 18.

  The CPU 19 is a processor that comprehensively controls each unit of the digital camera 1. For example, the CPU 19 executes a known autofocus (AF) control based on contrast detection or a known automatic exposure (AE) calculation based on the output of the image sensor 12. In addition, the CPU 19 performs digital processing on the image data output from the AFE 13. As an example, the digital processing of the present embodiment includes interpolation processing, white balance processing, gradation conversion processing, contour enhancement processing, color conversion processing, and the like. Furthermore, the CPU 19 of this embodiment operates as an image reduction unit 20, a reduced image noise removal unit 21, and a target image noise removal unit 22 by executing the image processing program. Therefore, the operations of the image reducing unit 20, the reduced image noise removing unit 21, and the target image noise removing unit 22 will be described with reference to the flow of image data shown in FIG.

  The image reduction unit 20 uses the following equation (1) to reduce the image obtained by the image pickup device 12 (hereinafter referred to as the target image A) by averaging the pixel values for each image area of 4 pixels × 4 pixels. A reduced image B having a rate of 1/16 is generated.

  Note that the target image A has pixel values of all color components of RGB or YCrCb at each pixel position by digital processing of the CPU 19, and the image reduction unit 20 generates a reduced image B for each color component. In the present embodiment, the reduction ratio is 1/16 (that is, the image area to be averaged is 4 pixels × 4 pixels), but is appropriately determined according to the image size of the target image A, the accuracy of noise removal processing, and the like. It is preferred that

  The reduced image noise removing unit 21 generates a reduced image D by removing low-frequency noise from the reduced image B using a known method. Specifically, the reduced image noise removing unit 21 reads the reference image C of the noise removal process in which the low frequency noise has already been removed from the frame memory 14 as a reduced image of the image one frame before the target image A, Based on the following equation (2), low-frequency noise is removed from the reduced image B while referring to the reference image C to generate a reduced image D.

Here, Equation (2) shows a case where an epsilon filter EPS is used as a known technique. The constant ε is a value based on the standard deviation σ of the pixel value in the flat area of the image, and is set to about ε = 3σ to 4σ in this embodiment. Based on Expression (2), the reduced image noise removing unit 21 removes the low-frequency noise of the reduced image B by smoothing with reference to the reference image C because it is a flat region when | B−C | ≦ ε. If | B−C |> ε, the reduced image B is referenced to leave the image structure.

  The target image noise removing unit 22 generates the target image E by removing the low-frequency noise superimposed on the target image A using the reduced images B and D based on the following equation (3).

Here, the coordinates (x / 2, y / 2) indicate the pixel positions of the reduced images B and D corresponding to the pixel (x, y) of the target image A, and are below the decimal points of x / 2 and y / 2. The fraction is rounded down.

  The operation member 23 includes, for example, a release button, a power button, a command dial, a cross-shaped cursor key, a determination button, and the like. The operation member 23 receives various inputs of the digital camera 100 from the user. For example, the operation member 23 may be an instruction input for starting AE operation before imaging by half-pressing the release button, an instruction input for starting imaging according to the imaging mode by full-pressing operation, or an operation for switching the operation mode of the digital camera 100. In addition, an input operation on the setting screen is received from the user. In addition, a touch panel configured by a transparent panel having the same shape as the monitor 15 and stacked on the entire surface of the monitor 15 may be used as the operation member 23. That is, the touch panel may detect the position of a stylus (or fingertip or the like) in contact with the panel surface and output the detected position information to the CPU 19 to accept an instruction input from the user.

  Next, noise removal processing by the digital camera 100 according to the present embodiment will be described with reference to the flowchart of image data in FIG. 2 and the flowchart in FIG. 3. In the digital camera 100, it is assumed that the shooting mode is set to the moving image mode in advance based on an operation instruction of the operation member 23 by the user.

  When the CPU 19 receives an instruction for capturing a moving image (for example, full pressing operation of the release button of the operation member 23) from the user, the CPU 19 causes the image sensor 12 to start capturing a moving image of the subject. CPU19 starts the process from step S101 with respect to the frame of the imaged moving image.

  Step S101: The CPU 19 reads a frame output from the image sensor 12 via the AFE 13 and the frame memory 14, performs digital processing, and records it as the target image A in the frame memory 14.

  Step S102: The image reducing unit 20 reads the target image A, and generates a reduced image B having a reduction ratio of 1/16 using Expression (1).

  Step S103: The reduced image noise removing unit 21 reads the reference image C stored in the frame memory 14, and removes low-frequency noise from the reduced image B while referring to the reference image C based on Expression (2). Then, a reduced image D is generated. The CPU 19 overwrites and records the reduced image D in the frame memory 14 as the reference image C of the reduced image of the next frame.

  In the present embodiment, when the target image A is the first frame, since the reference image C does not exist, the reduced image noise removing unit 21 does not use the equation (2), and the reduced image D = the reduced image B. To process.

  Step S104: The target image noise removing unit 22 generates the target image E by removing the low-frequency noise superimposed on the target image A using the reduced images B and D based on Expression (3).

  Step S105: The CPU 19 compresses the target image E from which noise has been removed in accordance with a moving image format such as H.264 or Motion JPEG to generate moving image data. Note that the CPU 19 preferably removes high-frequency noise superimposed on the target image E using a known noise removal filter when compressing the target image E in a moving image.

  Step S106: The CPU 19 determines whether or not a moving image capturing end instruction has been received, for example, when the user has fully released the release button of the operation member 23. When the CPU 19 receives an instruction to end moving image capturing (YES side), the CPU 19 generates a moving image file of moving image data that has been compressed, and records the moving image file in the storage unit 16 or the storage medium 18. The CPU 19 ends the series of processes. On the other hand, if the CPU 19 has not received a moving image capturing end instruction, the CPU 19 proceeds to step S101 (NO side), and performs the processes of steps S101 to S105 on the next frame until the end instruction is received.

  As described above, in the present embodiment, when the reduced image noise removing unit 21 removes the low frequency noise from the pixel (x, y) of the reduced image B, the pixel ( Reference is made to an image area of 5 pixels × 5 pixels called x + i, y + j). Similarly, when removing the low frequency noise from the reference image C, the reduced image noise removing unit 21 refers to an image area of 5 pixels × 5 pixels in the reference image two frames before. That is, the low-frequency noise removal of the reduced image B by the reduced image noise removing unit 21 is substantially performed by referring to an area larger than the image area of 5 pixels × 5 pixels in relation to the reference image two frames before. The low frequency noise can be removed from the target image A with high accuracy and high accuracy.

  Since the image data stored in the frame memory 14 and referred to by low-frequency noise removal is only the reference image C from which the low-frequency noise has been removed by the reduced image of the image one frame before, the frame memory 14 -CPU 19. The amount of data transferred between the CPU 19 and the CPU 19 can be reduced, and the cost can be greatly reduced.

Furthermore, since noise is removed using a reduced image, low-frequency noise that is easy to see when the entire image is displayed, not pixel-size display, can be removed.
<< Other embodiments >>
A digital camera according to another embodiment of the present invention is the same as the digital camera 100 according to one embodiment shown in FIG. 1, and a detailed description of components of the digital camera 100 according to this embodiment is omitted.

  Further, the noise removal processing by the digital camera 100 of the present embodiment is the same as that of the embodiment shown in FIGS. 2 and 3 when the shooting mode is the moving image mode, and detailed description thereof is omitted.

  Note that the difference between the digital camera 100 of the present embodiment and that of the first embodiment is that the noise removal processing when the shooting mode is set to the single-shot still image mode is the flowchart shown in FIG. 4 and FIG. The flow of the image data shown in FIG. A noise removal process in the still image mode by the digital camera 100 according to the present embodiment will be described with reference to the flowchart of FIG. 4 and the flowchart of image data of FIG.

  When the CPU 19 receives an instruction for capturing a still image (for example, full pressing operation of a release button included in the operation member 23) from the user, the CPU 19 causes the image sensor 12 to capture a still image of the subject. The CPU 19 starts processing from step S201.

  Step S201: The CPU 19 reads the still image output from the image sensor 12 via the AFE 13 and the frame memory 14, performs digital processing, and records it as the target image A in the frame memory 14.

  Step S202: The image reducing unit 20 reads the target image A, and generates a reduced image B having a reduction ratio of 1/16 using Expression (1).

  Step S203: The reduced image noise removing unit 21 removes the low frequency noise superimposed on the reduced image B based on the equation (2), and generates a reduced image D. For this purpose, the reduced image noise removing unit 21 of the present embodiment removes low-frequency noise from the reduced image B with reference to the reduced image B itself as the reference image C as shown in FIG.

  Step S204: The target image noise removing unit 22 generates the target image E by removing the low-frequency noise superimposed on the target image A using the reduced images B and D based on Expression (3).

  Step S205: The CPU 19 removes high-frequency noise superimposed on the target image E using a known noise removal filter, and generates and stores still image files such as JPEG format and YUV format for the image data of the target image E. The data is recorded in the unit 16 and the storage medium 18. The CPU 19 ends the series of processes.

  As described above, in the present embodiment, in the moving image mode, the reduced image noise removing unit 21 refers to the low frequency noise from the pixel (x, y) of the reduced image B based on the expression (2). An image area of 5 pixels × 5 pixels called pixel (x + i, y + j) of image C is referred to. Similarly, when removing the low frequency noise from the reference image C, the reduced image noise removing unit 21 refers to an image area of 5 pixels × 5 pixels in the reference image two frames before. That is, the low-frequency noise removal of the reduced image B by the reduced image noise removing unit 21 is substantially performed by referring to an area larger than the image area of 5 pixels × 5 pixels in relation to the reference image two frames before. The low frequency noise can be removed from the target image A with high accuracy and high accuracy.

  Even in the still image mode, the low-frequency noise of the still image can be removed using the same configuration as in the moving image mode.

  Further, the image data stored in the frame memory 14 and referred to by the low-frequency noise removal is only the reference image C from which the low-frequency noise has been removed by the reduced image of the image one frame before in the moving image mode. The amount of data transferred between the frame memory 14 and the CPU 19 and within the CPU 19 can be reduced, and the cost can be greatly reduced.

Furthermore, since noise is removed using a reduced image, low-frequency noise that is easy to see when the entire image is displayed, not pixel-size display, can be removed.
<< Modification of other embodiment >>
FIG. 6 shows an example of the configuration of a digital camera 200 according to a modification of another embodiment of the present invention. In FIG. 6, among the components of the digital camera 200 of the present embodiment, the same components as those of the digital camera 100 of the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted.

  Further, the noise removal processing by the digital camera 200 of the present embodiment is the same as that of the one embodiment shown in FIGS. 2 and 3 when the shooting mode is the moving image mode, and detailed description thereof is omitted.

  The difference between the digital camera 200 of the present embodiment and the digital camera 100 of the other embodiments is that the noise removal processing when the shooting mode is the still image mode is the same as that in FIG. The point is that the flow becomes as shown in FIG.

  That is, in the still image mode, the CPU 19 of this embodiment operates as a reference image noise removing unit 30 together with an image reducing unit 20, a reduced image noise removing unit 21, and a target image noise removing unit 22, as shown in FIG. To do. Then, as shown in FIG. 7, the reference image noise removing unit 30 removes noise from the reduced image B in advance using a known method such as multi-resolution analysis and records it as the reference image C in the frame memory 14. To do. The reduced image noise removing unit 21 removes the low frequency noise from the reduced image B while referring to the reference image C from which the low frequency noise has been removed by the reference image noise removing unit 30 based on Expression (2). D is generated.

  As described above, in the present embodiment, in the moving image mode, the reduced image noise removing unit 21 refers to the low frequency noise from the pixel (x, y) of the reduced image B based on the expression (2). An image area of 5 pixels × 5 pixels called pixel (x + i, y + j) of image C is referred to. Similarly, when removing the low-frequency noise from the reference image C, the reduced image noise removing unit 21 refers to an image area of 5 pixels × 5 pixels in the reference image two frames before. That is, the low-frequency noise removal of the reduced image B by the reduced image noise removing unit 21 is substantially performed by referring to an area larger than the image area of 5 pixels × 5 pixels in relation to the reference image two frames before. The low frequency noise can be removed from the target image A with high accuracy and high accuracy.

  In the still image mode, the reduced image noise removing unit 21 refers to the reduced image B from which noise has been removed by the reference image noise removing unit 30 as the reference image C. Can be removed. Furthermore, since the reduced image noise removing unit 21 can adjust the timing for reading the target image A and the reference image C, the cost required for synchronization can be suppressed.

  In addition, in the moving image mode, the image data stored in the frame memory 14 and referred to by the low frequency noise removal is only the reference image C from which the low frequency noise has been removed by the reduced image of the image one frame before. In the case of the mode, since only the reference image C of the reduced image B from which noise has been removed by the reference image noise removing unit 30, only the amount of data transferred between the frame memory 14 and the CPU 19 and within the CPU 19 can be reduced, and the cost can be reduced. Can be greatly reduced.

Furthermore, since noise is removed using a reduced image, low-frequency noise that is easy to see when the entire image is displayed, not pixel-size display, can be removed.
<< Additional items of embodiment >>
(1) In the above-described embodiment, the CPU 19 performs each process of the image reduction unit 20, the reduced image noise removal unit 21, the target image noise removal unit 22, and the reference image noise removal unit 30 of the digital cameras 100 and 200 in terms of software. Although an example of realization has been described, it goes without saying that each of these processes may be realized by hardware using an ASIC.

  (2) The image processing apparatus of the present invention is not limited to the examples of the digital cameras 100 and 200 of the above embodiment. For example, the captured image is read into the computer directly from the digital cameras 100 and 200 or via the Internet, and the image processing program installed in the computer is executed, so that the computer is used as the image processing apparatus of the present invention. May function. Then, when there is only one read image, the computer determines that the read image is a still image and can process it according to the flowchart shown in FIG. 4 and the flow of image data shown in FIG. 5 or FIG. preferable.

  (3) In the above embodiment, when the reduced image noise removing unit 21 removes the low frequency noise from the pixel (x, y) of the reduced image B based on the equation (2), the pixel (x + i) of the reference image C , Y + j) is referred to as an image area of 5 pixels × 5 pixels, but the present invention is not limited to this, and an image area having a size other than 5 pixels × 5 pixels may be referred to.

  In addition, although the low frequency noise is removed by using the epsilon filter EPS, the expression (2) may be performed by using another filter.

  If the subject to be photographed is a subject with little movement such as a landscape, the same reference image C stored in the frame memory 14 may be used without updating. Thereby, the data transfer amount between the frame memory 14 and the CPU 19 can be further reduced, and the cost can be further reduced.

  (4) In the modification of the other embodiment, the reference image noise removing unit 30 performs a known method such as multi-resolution analysis in order to generate the reference image C by removing noise from the reduced image B. However, the present invention is not limited to this, and the same processing as the reduced image noise removing unit 21 may be performed.

  From the above detailed description, features and advantages of the embodiments will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments as described above without departing from the spirit and scope of the right. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and modifications, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to use appropriate improvements and equivalents within the scope disclosed in.

DESCRIPTION OF SYMBOLS 11 ... Imaging optical system, 12 ... Imaging device, 13 ... AFE, 14 ... Frame memory, 15 ... Monitor, 16 ... Storage part, 17 ... Recording I / F, 18 ... Storage medium, 19 ... CPU, 20 ... Image reduction part 21 ... Reduced image noise removal unit, 22 ... Target image noise removal unit, 24 ... Bus, 30 ... Reference image noise removal unit, 100, 200 ... Digital camera,

Claims (7)

An image input unit for inputting each of a plurality of images captured continuously in time as target images;
An image reduction unit that reduces the target image to generate a reduced image;
A frame memory for storing a reference image for noise removal processing;
A reduced image noise removing unit that removes low-frequency noise from the reduced image while referring to the reference image;
A target image noise removing unit that removes the low frequency noise from the target image based on the reduced image from which the low frequency noise has been removed;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
The frame memory stores the reduced image from which the low-frequency noise has been removed by the reduced image noise removing unit as the reference image. The image processing apparatus according to claim 1 or 2,
The reduced image noise removing unit removes the low-frequency noise from the pixels of the reduced image while referring to an image area around the pixels of the reference image corresponding to the pixels of the reduced image. apparatus. An imaging unit that images a subject and generates an image;
An image processing apparatus according to claim 1;
An imaging apparatus comprising: The imaging apparatus according to claim 4,
With operating members,
The reduced image noise removing unit refers to the reduced image from the reduced image while referring to the reduced image itself generated by the image reducing unit as the reference image when the still image mode is set as the shooting mode by the user. An imaging apparatus characterized by removing noise. The imaging apparatus according to claim 5,
An imaging apparatus comprising: a reference image noise removing unit that removes noise from the reduced image generated by the image reducing unit and records the reference image as the reference image in the frame memory. An image input procedure for inputting each of a plurality of images captured continuously in time as target images,
An image reduction procedure for generating a reduced image by reducing the target image;
A reduced image noise removal procedure for removing low-frequency noise from the reduced image while referring to a reference image for noise removal processing stored in a frame memory;
A target image noise removal procedure for removing the low frequency noise from the target image based on the reduced image from which the low frequency noise has been removed;
An image processing program for causing a computer to execute.

Images