TWI602153B - Image denoising method and image denoising apparatus - Google Patents

Description

Image denoising method and image denoising device

The present invention relates to an image processing technique, and more particularly to an image denoising method and an image denoising device.

In the application of digital cameras, the use of high sensitivity (ISO) to capture multiple continuous shooting images, resulting in reduced jitter caused by jitter or object movement, has been widely used in handheld night shooting mode. However, because of the high ISO image capture, it is necessary to deal with the extra noise introduced by the high ISO.

A common method of de-noising is to perform correlation de-noising action on the same position of two images after alignment after capturing multiple images. 1 is a flow chart of a conventional image denoising method. Referring to FIG. 1, a conventional digital camera enters a live view image mode, for example, when a user activates the camera or presses a live view button on the camera (step S102). At this time, the digital camera will display the scene image captured by the current lens on the screen for the user to perform framing or image processing on the screen. When the user presses the shutter button of the digital camera, the digital camera focuses (step S104) And performing image capture on the current screen (step S106). In the handheld night shot mode, the digital camera captures a plurality of current images, and performs motion estimation (step S108) and image alignment (step S110) on the current images, thereby These current images perform denoising processing (step S112), and finally output denoising images (step S114). However, the conventional image denoising method starts after the shutter is triggered, and the complexity required for the denoising process using the plurality of current images is high and the calculation amount is large, so the user presses the shutter. When the button is pressed for shooting, the digital camera will generate additional image processing time. In addition, the denoising processing of multiple current images will also limit the speed at which the digital camera captures multiple consecutive photos and the time when the digital camera returns to the instant preview mode.

In view of the above, how to effectively improve the operational efficiency of digital camera output to noise-carrying images has become one of the problems that manufacturers are eager to solve.

The invention provides an image de-noising method and an image de-noising device, which can reduce the time for performing the de-noising processing on the currently captured image, thereby improving the performance of the image processing.

The image denoising method of the present invention is applicable to an electronic device having an image capturing unit. The image denoising method uses the image capturing unit to continuously capture multiple live view images. Then, a noise removal process is performed on the instant preview image to output a noise removal image. Then, receive the shutter trigger signal to facilitate Use the image capture unit to capture the current image. Finally, the access to the noise image is accessed, and the noise removal image is used to perform the denoising processing on the current image to output the current image of the noise.

In an embodiment of the present invention, the step of performing the denoising process on the instant preview image, and outputting the denoising image is to perform the current instant preview image by using the previous instant preview image in the instant preview image. The noise processing is performed, and the result image after performing the noise removal processing is performed on the next instant preview image to perform the noise processing, and then the above steps are repeated to perform the result of the noise processing on the last instant preview image. The image output is used as a noise image.

In an embodiment of the invention, the resolution of the instant preview image is smaller than the current image.

In an embodiment of the invention, the step of performing denoising processing on the current image by using the denoising image, and outputting the denoised current image includes first amplifying the resolution of the denoising image to be the same as the current image. The resolution is then used to perform a denoising process on the current image using the amplified denoising image to output the current image of the noise.

In an embodiment of the invention, before the step of performing the denoising processing on the current image by using the amplified denoising image to output the denoised current image, the method is more on the current image and the amplification. The subsequent de-noise image performs motion estimation and image alignment.

The image denoising device of the present invention comprises an image capturing unit, a storage unit and a processing unit. The image capturing unit is used to capture multiple images. Storage The storage unit is for storing images. The processing unit is configured to execute a plurality of image processing modules to process the image. The image processing module includes an instant preview module, a first de-noise module, an image capturing module, and a second de-noising module. The instant preview module continuously captures multiple instant preview images by using the image capturing unit. The first de-noising module is configured to perform a de-noising process on the instant preview image to output a de-noise image to the storage unit. The image capturing module is configured to receive a shutter trigger signal, and the image capturing unit is used to capture the current image. The second de-noising module is configured to access the de-noise image in the storage unit, and perform denoising processing on the current image by using the de-noise image to output the current image of the de-noising.

In an embodiment of the present invention, the first de-noising module performs denoising processing on the current instant preview image by using the previous instant preview image in the instant preview image, and performs de-noising by using the same. The processed result image performs denoising processing on the next instant preview image, and repeats the above steps, and outputs the result image after the last instant preview image is subjected to the noise processing as the noise removal image.

In an embodiment of the invention, the resolution of the instant preview image is smaller than the current image.

In an embodiment of the invention, the image processing module further includes an image enlargement module. The image amplifying module is configured to enlarge the resolution of the denoising image to the same resolution as the current image, so that the second denoising module performs the denoising on the current image by using the amplified denoising image. Processing, outputting the current image of the noise.

In an embodiment of the invention, the image processing module further includes a motion estimation module and an image alignment module. The mobile estimation module is configured to perform motion estimation on the current image and the amplified denoising image. The image alignment module is used to perform image alignment on the current image and the amplified denoising image.

Based on the above, in the image de-noising method and the image de-noising device, the image de-noising device uses the de-noise image after the image is processed by the instant preview image to trigger the shutter. The current image obtained is subjected to noise processing to output the current image of the noise. In this way, when the user presses the shutter button to take a picture, the image de-noising device can perform the de-noise processing without extracting multiple current images, thereby reducing the amount of computation and reducing the processing time. Accordingly, the noise processing method of the embodiment can effectively improve the operation efficiency of the image denoising device outputting the noise-canceled image.

The above described features and advantages of the invention will be apparent from the following description.

20, 40‧‧‧Image de-noising device

202, 402‧‧‧Image capture unit

204, 404‧‧‧ storage unit

206, 406‧‧ ‧ processing unit

208, 408‧‧‧ Instant Preview Module

210, 410‧‧‧ first de-noising module

212, 412‧‧‧ image capture module

214, 414‧‧‧ second de-noising module

416‧‧‧Mobile Estimation Module

418‧‧‧Image Alignment Module

420‧‧‧Image Amplification Module

S102~S114‧‧‧Professional image denoising method steps

S302~S308, S502~S514, S602~S624‧‧‧ steps of the image denoising method of the embodiment

1 is a flow chart of a conventional image denoising method.

FIG. 2 is a block diagram of an image denoising apparatus according to an embodiment of the invention.

FIG. 3 is a flow chart of an image de-noising method according to an embodiment of the invention.

4 is a block diagram of an image denoising apparatus according to an embodiment of the invention.

FIG. 5 is a flow chart of an image de-noising method according to an embodiment of the invention.

FIG. 6 is a schematic diagram of an image denoising method according to an embodiment of the invention.

In the digital camera application, after the power of the digital camera is turned on, it can enter the live view mode, so that the user can compose the current scene, white balance adjustment or tone adjustment through the display screen of the digital camera. action. In this instant preview mode, the digital camera activates the de-noising method on the timeline to progressively preview the image using the previous instant image captured from the previous frame time. The current instant preview image captured by the frame time performs denoising processing, and outputs a denoising image corresponding to the current frame time. Since the noise removal image has met a certain degree of low noise condition, after the shutter of the digital camera is triggered, the digital camera can use the image as a reference image to perform the current image captured. Noise processing saves the time and computational cost of performing noise removal processing. The present invention is an image denoising method and an image denoising device proposed based on the above viewpoint. In order to make the content of the present invention clearer, the following specific embodiments are examples of the invention that can be implemented.

FIG. 2 is a block diagram of an image denoising apparatus according to an embodiment of the invention. Referring to FIG. 2, the image denoising device 20 of the embodiment is, for example, a digital camera, a Digital Single Lens Reflex (DSLR) camera, a Digital Video Camcorder (DVC), or the like. The electronic device such as a smart phone or a tablet that has a video capture function is not limited to the above. The image de-noising device 20 includes an image capturing unit 202, a storage unit 204, and a processing unit 206. In addition, the image de-noising device 20 further includes an instant preview module 208, a first de-noise module 210, an image capturing module 212, and a second de-noising module 214, and the module can be processed by the processing unit 206. To execute.

In detail, the image capturing unit 202 may include components such as a lens, an aperture, a shutter, and a photosensitive element for capturing a plurality of images. The storage unit 204 is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory (Flash memory), hard Discs, or other similar devices or combinations of these devices, can be used to store images and materials. The processing unit 206 can be implemented by software, hardware, or a combination thereof, and is not limited herein. The software is, for example, an operating system, an application software, or a driver. The hardware is, for example, a central processing unit (CPU), or other programmable general purpose or special purpose microprocessor (Microprocessor), digital signal processor (DSP), and can be programmed. Controllers, Application Specific Integrated Circuits (ASICs), and Programmable Logic Devices (PLDs). In this embodiment, at The processing unit 206 can execute various commands or programs on the image de-noising device 20 to control the overall operation of the image de-noising device 20, and execute multiple image processing modules, such as an instant preview module 208, The image denoising method of the embodiment is implemented by processing the image captured by the image capturing unit 202 by using the noise removing module 210, the image capturing module 212, and the second de-noising module 214. .

The instant preview module 208, the first de-noise module 210, the image capturing module 212, and the second de-noising module 214 may be code written by a programming language or a separate chip. For example, the instant preview module 208, the first de-noise module 210, the image capturing module 212, and the second de-noising module 214 may be code written by a programming language, and the code is stored. In storage unit 204, it is executed by processing unit 206. Alternatively, the instant preview module 208, the first de-noise module 210, the image capture module 212, and the second de-noise module 214 are independent wafers composed of one or several logic gates, respectively. The processing unit 206 is coupled to the processing unit 206 to control the respective wafers. The functions of the instant preview module 208, the first de-noise module 210, the image capturing module 212, and the second de-noising module 214 will be described below.

The instant preview module 208 can use the image capturing unit 202 to continuously capture images of multiple instant scenes in a plurality of frame times as an instant preview image, that is, the image capturing unit 202 at each frame time. An instant preview image can be obtained. Specifically, the image denoising apparatus 20 of the embodiment can enter the instant preview mode after the power is turned on, and the instant preview module 208 can continuously use the image capturing unit 202 to capture the instant preview image. In addition, the instant preview module 208 The image can be displayed on the screen (not shown) of the noise device 20 to display an instant preview image, so that the user can preview the captured image content through the screen in real time during shooting. In other words, the instant preview module 208 can sequentially display the instant preview image captured by the image capturing unit 202 on the screen of the image de-noising device 20, so that the user can take a picture on the screen. Scene selection, image capture or processing, and more.

The first de-noising module 210 performs denoising processing on the instant preview image captured by the image capturing unit 202, wherein the first de-noising module 210 is, for example, a time-based progressive method. De-noise processing is performed sequentially using successive instant preview images. Specifically, the first de-noise module 210 refers to the previous instant preview image of the previous frame time to perform denoising processing on the current instant preview image of the current frame time, and previews the image according to the current instant. After the result of the noise processing, the next instant preview image of the next frame time is subjected to the noise processing. In the embodiment, when the first noise removal module 210 performs the denoising process, for example, a motion estimation module (not shown) is used to perform the motion estimation to estimate the difference between the two images. Moving the direction, and using the image alignment module (not shown) to perform image alignment to superimpose the two images, thereby adjusting the data of the two images to produce a better image, and filtering out the preview image. The noise.

The image capturing module 212 uses the image capturing unit 202 to capture the current image. Specifically, when the user presses the image to the shutter button of the noise device 20, the image denoising device 20 can generate a shutter trigger signal, and the image capturing module 212 controls the image according to the shutter trigger signal. The capturing unit 202 enters the current scene Line image capture.

The second de-noise module 214 performs the de-noise processing on the current image captured by the image capturing unit 202 by using the de-noise image outputted by the first de-noising module 210, and outputs the noise-removing signal. Current image. Here, the second de-noising module 214 can also perform the motion estimation by using the motion estimation module when performing the de-noising processing, and perform image alignment by using the image alignment module.

FIG. 3 is a flow chart of an image de-noising method according to an embodiment of the invention. Referring to FIG. 3, the image denoising method in this embodiment is applicable to the image denoising device 20 or the like electronic device having the image capturing unit 202. The detailed steps of the image denoising method of the present embodiment will be described below with reference to the components in FIG. 2.

Referring to FIG. 2 and FIG. 3, first, the instant preview module 208 continuously captures multiple instant preview images by using the image capturing unit 202 (step S302). Here, the image capturing unit 202 performs the framing action sequentially in time in a dynamic form, and the captured image is an instant preview image.

Then, the first de-noise module 210 performs denoising processing on the live preview image, and outputs the de-noise image to the storage unit 204 (step S304). In this embodiment, the first de-noise module 210 sequentially performs two de-noise processing on the two instant preview images captured by the image capturing unit 202 in two consecutive frame times.

It should be noted that before the first de-noising module 210 performs the denoising process, the image de-noising device 20 can first preview the current using the mobile estimation module. The image and the previous instant preview image are subjected to the noise-processed result image to perform the motion estimation, and the image alignment module can also be used to process the current instant preview image and the previous instant preview image after the noise processing. The resulting image is used to perform image alignment. Before the shutter of the image capturing unit 202 has not been driven, the image denoising device 20 repeatedly uses the motion estimation module and the image alignment module to perform the above processing, and when the shutter of the image capturing unit 202 is When driving, the first de-noise module 210 outputs the last instant preview image to perform the de-noise processing result image as a de-noise image.

For example, it is assumed that the instant preview image captured by the first de-noising module 210 at each frame time is P _X , and the result image corresponding to the noise processing of each frame time is S _X . X is a positive integer, and the first de-noise module 210 uses the previous instant preview image P _X-1 captured by the image capturing unit 202 at the previous frame time, and the current frame time is currently The instant preview image P _X performs denoising processing to obtain the result image S _x after the noise removal processing, as the denoising image corresponding to the current frame time. And so on, before the shutter of the image capturing unit 202 has not been driven, the first de-noising module 210 repeatedly performs the above steps to output an instant preview image corresponding to each frame time to perform denoising processing. The resulting image.

Thereafter, the image capturing module 212 receives the shutter trigger signal to capture the current image by the image capturing unit 202 (step S306). For example, when the user presses the image to the shutter button of the noise device 20, the image capturing module 212 receives the shutter trigger signal and uses the image capture command according to the shutter trigger signal. Element 202 captures the current image.

Finally, the second de-noising module 214 accesses the de-noise image in the storage unit 204, and performs de-noising processing on the current image by using the de-noise image to output the de-noised current image (step S308). ). Here, the second de-noising module 214 accesses the storage unit 204, and performs the de-noising processing on the current image by the de-noise image after the noise-removed processing of the instant preview image corresponding to the current frame time. . The image denoising device 20 can use the motion estimation module to perform motion estimation on the current image and the denoising image, and can also perform image alignment on the current image and the denoising image by using the image alignment module. . In addition, the second de-noising module 214 can store the de-noised current image to the storage unit 204 or output to the screen of the image de-noising device 20 for the user to browse.

According to the image denoising method described above, each time the user presses the shutter button to take a photo, since the image denoising device 20 only needs to capture a single image, the noise processing is performed, so the image is mixed. The time for the signal processing device 20 to perform the denoising process on the current image can be greatly reduced. In particular, when the user uses the image denoising device 20 to capture a plurality of continuous shooting images (for example, performing the operation of the handheld night shooting mode), the image denoising device 20 can further reduce the noise processing on the image. The time, so it can achieve the effect of high-speed continuous shooting.

It is worth mentioning that, since the resolution of the instant preview image is usually smaller than that of the current image, in another embodiment of the present invention, when the denoising process is performed on the current image by using the denoising image, The resolution of the noise image is amplified, so that the resolution of the denoising image is the same as the resolution of the current image, and then the solution is solved. The de-noise image after the resolution is amplified and the current image is subjected to de-noise processing to obtain a better noise-canceling effect. A further embodiment will be described in detail below.

4 is a block diagram of an image denoising apparatus according to an embodiment of the invention. Referring to FIG. 4, the image de-noising device 40 includes an image capturing unit 402, a storage unit 404, and a processing unit 406. In addition, the image de-noising device 40 further includes an instant preview module 408, a first de-noise module 410, an image capturing module 412, a second de-noising module 414, a motion estimation module 416, and an image. The module 418 and the image magnifying module 420 are aligned, and the module can be executed by the processing unit 406. The functions of the image capturing unit 402, the storage unit 404, the processing unit 406, the instant preview module 408, the first de-noising module 410, the image capturing module 412, and the second de-noising module 414 are as described above. The image capturing unit 202, the storage unit 204, the processing unit 206, the instant preview module 208, the first de-noise module 210, the image capturing module 212, and the second de-noising module 214 are the same or Similar, so I won't go into details here.

Different from the foregoing embodiment, the image denoising device 40 is further configured with a motion estimation module 416, an image alignment module 418, and an image amplification module 420. The motion estimation module 416 is configured to perform a motion estimation to estimate a moving direction between the two images. The image alignment module 418 is configured to perform image alignment to superimpose the two images. In addition, the image amplifying module 420 is configured to amplify the resolution of the denoising image so that the resolution of the denoising image is the same as the resolution of the current image. The detailed steps of the image denoising method of the present embodiment will be described below with reference to the components in FIG.

FIG. 5 is a flowchart of an image denoising method according to an embodiment of the present invention. Referring to FIG. 4 and FIG. 5 together, the instant preview module 408 may continue to capture by using the image capturing unit 402. A plurality of instant preview images are displayed (step S502). Then, the first de-noising module 410 performs denoising processing on the live preview image, and outputs the de-noise image to the storage unit 404 (step S504). Thereafter, the image capturing module 412 receives the shutter trigger signal to capture the current image by the image capturing unit 402 (step S506).

In this embodiment, the image enlargement module 420 enlarges the resolution of the denoising image to the same resolution as the current image (step S508). Specifically, the image amplifying module 420 can adopt a general image magnifying method, such as polynomial interpolation, edge-directed interpolation, and sample-based super-resolution technology. (exampled-based for super-resolution) and other methods to amplify the resolution of the noise image. The image enlargement method performed by the image magnifying module 420 is exemplified, and the embodiment is not limited thereto.

Finally, the second de-noise module 414 performs the de-noise processing on the current image by using the amplified de-noise image, and outputs the current image to the noise. Specifically, before the second denoising module 414 performs the denoising process, the image denoising device 40 may first use the motion estimation module 416 to perform movement on the current image and the amplified denoising image. Estimated (step S510). Moreover, the image de-noising device 40 can also perform image alignment on the current image and the amplified de-noise image by using the image alignment module 418 (step S512). Finally, the second de-noising module 414 outputs The current image of the noise is removed (step S514). In order to clarify the above-described image denoising method, an embodiment will be described below.

6 is a schematic diagram of an image denoising method according to an embodiment of the present invention. Referring to FIG. 4 and FIG. 6, first, when the image denoising device 40 enters the instant preview image mode (step S602) The instant preview module 408 continuously captures multiple instant preview images by using the image capturing unit 402 (step S604). Then, the motion estimation module 416 performs the motion estimation by performing the motion estimation on the result image of the previous instant preview image and the current instant preview image (step S606), and according to the estimation result, the image is further The alignment module 418 performs image alignment (step S608). Finally, the first de-noise module 410 performs denoising processing on the live preview images that have been subjected to motion estimation and image alignment. The result image of the instant preview image after the noise processing is obtained (step S610). Before the shutter of the image capturing unit 402 has not been triggered, the image de-noising device 40 repeatedly performs the above steps S606-S610, and when the shutter of the image capturing unit 402 is triggered, the first de-noising module 410 The result image of the last instant preview image after performing the noise removal processing is outputted as a noise image.

In this embodiment, the image capturing unit 402 can have a two-stage shutter, and when the user taps the shutter button of the image capturing unit 402, the image capturing module 412 will receive the focus trigger signal to utilize The image capturing unit 402 performs focusing (step S612), and when the user presses the shutter button all the way, the image capturing module 412 receives the shutter triggering signal to perform image scanning on the current screen by the image capturing unit 402. Take to capture the current image (step S614), and store the current image Saved to storage unit 404.

At this time, the image amplifying module 420 enlarges the resolution of the denoising image output by the first de-noising module 410 to the same resolution as the current image (step S616). Then, the motion estimation module 416 performs motion estimation on the current image and the denoising image amplified by the image amplifying module 420 (step S618), and then the image alignment module 418 performs the current image and the amplified image. The noise image performs image alignment (step S620). Finally, the second denoising module 414 performs denoising processing on the current image by using the amplified denoising image to obtain a denoised current image (step S622), and outputs the current denoising current image. Image (step S624).

The image amplifying module 420 enlarges the resolution of the denoising image to the same resolution as the current image, and the image denoising device 40 can compensate for the difference in the number of pixels between the denoising image and the current image, so that the image is made. The denoising device 40 preferably uses the amplified denoising image to perform denoising processing on the current image. Since the image denoising device 40 only needs to capture a single current image to perform the denoising process, the denoising processing time required to denoise the current image can be greatly reduced, thereby improving The efficiency of noise processing.

In summary, in the image denoising method and the image denoising device of the present invention, when the user presses the shutter button to drive the image to the noise device to capture the image, the image is denoised. The device can perform denoising processing by capturing a single current image. The image denoising device performs denoising processing on the current image by using the instant preview image corresponding to the previous frame time to remove the noise image after the noise processing. Therefore, the image denoising device performs denoising processing on the current image. The time can be greatly reduced to improve the efficiency of the noise removal process.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

S302~S308‧‧‧Image Denoising Method Steps

Claims (8)

An image de-noising method is applicable to an electronic device having an image capturing unit, the method comprising the steps of: continuously capturing a plurality of Liveview images by using the image capturing unit; and performing a de-noising signal Processing the instant preview image, outputting a denoising image; receiving a shutter trigger signal to capture a current image by using the image capturing unit; and accessing the denoising image and using the denoising image Performing the denoising process on the current image, outputting the current image of the denoising, wherein performing the denoising process on the instant preview image, and outputting the denoising image comprises: using the instant preview The previous instant preview image in the image performs the denoising process on a current instant preview image, and performs the denoising process on the next instant preview image by using a result image after performing the denoising process; The above steps are repeated, and the last instant preview image is output to perform the denoising processing on the result image as the denoising image. The image de-noising method of claim 1, wherein the instant preview image has a resolution less than the current image. The image denoising method according to claim 2, wherein the denoising process is performed on the current image by using the denoising image, and the step of outputting the denoised current image comprises: zooming in The resolution of the noise image to the same resolution as the current image; The denoising process is performed on the current image by using the amplified denoising image, and the current image of the denoising is output. The image denoising method according to claim 3, wherein the step of performing the denoising processing on the current image by using the amplified denoising image to output the denoised image of the current image The method further includes: performing a motion estimation on the current image and the amplified denoising image; and performing an image alignment on the current image and the amplified denoising image . An image denoising device includes: an image capturing unit that captures a plurality of images; a storage unit that stores the images; and a processing unit that executes a plurality of image processing modules to process the images, The image processing module includes: an instant preview module, wherein the image capturing unit continuously captures multiple instant preview images; and a first de-noising module performs a de-noising process on the instant preview image. Outputting a de-noise image to the storage unit; an image capture module receiving a shutter trigger signal for capturing a current image by the image capture unit; and a second de-noise module for accessing the image The denoising image in the storage unit, and performing the denoising processing on the current image by using the denoising image, and outputting the impurity The current image of the first de-noising module is configured to perform the de-noising processing on a current instant preview image by using the previous instant preview image in the instant preview images, and performing the de-noising processing The result image processed by the message performs the denoising process on the next instant preview image, and repeats the above steps, and outputs the last instant preview image to perform the denoising process as the denoising signal. image. The image denoising device of claim 5, wherein the instant preview image has a resolution less than the current image. The image de-noising device of claim 6, wherein the image processing module further includes: an image magnifying module that amplifies the resolution of the de-noising image to be the same as the current image The resolution is obtained by the second denoising module performing the denoising processing on the current image by using the amplified denoising image to output the denoised current image. The image de-noising device of claim 7, wherein the image processing module further comprises: a motion estimation module, performing a motion estimation on the current image and the amplified And an image alignment module, configured to perform an image alignment on the current image and the amplified denoising image.