JP2011023855A - Apparatus and method for processing image, and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an information processing apparatus which can perform a digital repair for improving accuracy of noise elimination than before. <P>SOLUTION: The apparatus includes: a difference mask creation part 11 which creates a difference mask which carries out masking of an area eliminated as noise from an original moving image by noise elimination processing by comparing the original moving image including one or more object images and a processed moving image obtained by carrying out predetermined noise elimination processing to the original moving image; a mask creation means 12 for detecting an object image which has moved in the region masked by the difference mask from the processed moving image to create a mask for correction which masks an area where at least an detected object image is displayed; and a final moving image creation part 13 which transplants an image of the area which is masked by the correction mask of the original moving image to the processed moving image, and creates a final moving image. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a technique for efficiently removing noise from a moving image.

  In recent years, moving images such as old movies shot with movie films are often converted into digital moving images. Movie films have scratches and dirt on their surfaces due to aging and repeated reproduction. The dust can be removed by cleaning if the elapsed time from adhering to the movie film is short. However, dust that remains attached for a long time cannot be removed by normal cleaning, and the film surface is scraped off. Therefore, it is necessary to perform digital restoration when digitally converting a moving image of a movie film to remove noise caused by scratches or dust.

Noise removal by digital restoration includes a filter method and a pixel transplant method.
In the filter method, a filter that takes an average or median of pixel values (R, G, B luminance values, etc.) with surrounding pixels is used for each pixel. In the filter method, noise is removed in units of pixels. Therefore, it is useful for removing noise having a size close to one pixel, but is not suitable for removing noise having a larger area than that of a pixel. In addition, the resolution of the image is impaired.
In the pixel transplantation method, the operator removes noise by transplanting similar pixels in the frame around or around the noise portion to the noise portion. Although this method can surely remove noise, it requires work for each frame image by an operator, and a great deal of time is consumed.

  There is software that automatically performs such noise removal. For example, the present applicant also manufactures and sells such a conventional digital restoration system. For example, the digital restoration system operates as follows. First, one frame image of a moving image and the frame images before and after that are compared. By comparing, a discontinuous area of one frame image is identified as noise. Thereafter, the identified noise is removed.

  However, in the conventional digital restoration system, the noise and the original object image of the original moving image and the like (in this specification, the background image is also included in the object image, hereinafter referred to as “object image”). Will be determined. Specifically, a fast-moving image in the image is recognized as noise. However, in this case, there is a possibility that a fast-moving object image may also be recognized as noise. For this reason, visual confirmation work is required after processing by software, and the work efficiency cannot be improved. In order to avoid such a situation, the following two methods are used.

  The first method is to change the parameter for recognizing noise according to the intensity of motion for each scene of the video, and to loosen the noise criterion for fast motion scenes, and for slow motion scenes, noise. This is to make stricter criteria. Another method is to perform processing by masking a region where there is a lot of motion so that noise is not judged for the region. However, neither of these methods is an effective measure for shortening the time because it is necessary to check all the moving images once and set the parameters and masks for each scene in detail.

  Patent Document 1 is a noise removal device that is an example of a digital restoration system. In the noise removal device of Patent Document 1, it is determined whether or not the target pixel is an edge pixel that is a pixel of the contour of the object image from the luminance difference between the target pixel and its surrounding pixels. If the edge pixel is an edge pixel, the luminance value of the pixel at the top, bottom, left, and right of the edge pixel is compared with the luminance value of the pixel at the same position as the edge pixel in the preceding and following frames to determine whether the edge pixel is a noise candidate. Determine whether. Noise area candidates are created by integrating adjacent noise candidates for each scanning line of the frame, and the luminance variation of each pixel of the noise area candidate and the luminance variation of each pixel of a certain area adjacent to the noise area candidate before and after. Based on this, it is determined whether the noise candidate area is a noise area. If it is determined as a noise region, the noise is removed by replacing it with the average value of the luminance of pixels at the same position in the previous and subsequent frames.

  In this way, conventionally, when removing noise in a moving image, comparing the frame image of interest with the previous and next frame images, the continuity is the original object image, and the non-continuity is the noise. Distinguished as an ingredient. However, in this method, it is difficult to say that noise and object images are sufficiently selected. For example, a discontinuous shape change such as a garment or car gloss is recognized as a noise component, and the generated image is broken.

JP 2002-135623 A

  In view of the above problems, it is a primary object of the present invention to provide an image processing apparatus capable of digital restoration with improved noise removal accuracy than before.

  The image processing apparatus of the present invention that solves the above problems compares the noise by comparing an original moving image including one or more object images and a processed moving image obtained by performing a predetermined noise removal process on the original moving image. A difference mask generating means for generating a difference mask for masking a region removed as noise from the original moving image by the removal processing, and detecting an object image moved in the region masked by the difference mask from the processed moving image, A correction mask generating means for generating a correction mask for masking at least a region where the detected object image is displayed, and an image of the region masked by the correction mask of the original moving image is transplanted to the processed moving image; And a final moving image generating means for generating a final moving image.

  The noise removal process may be a process using software that automatically performs conventional noise removal. In conventional noise removal, as described above, it is difficult to select noise and object images. In the image processing apparatus of the present invention, the processed moving image is obtained by once applying noise removal to the original moving image. A moving object image is detected using the processed moving image. An object image that has been removed by mistake due to noise is detected as an object image that moves greatly in the processed moving image. Since the noise itself is removed in the processed moving image, it is not detected as an object image. Therefore, by detecting the moving object image, it is possible to specify the object image that has been removed by mistake. The correction mask is generated as a mask image that masks the region of the object image that has been mistakenly removed. By using the correction mask generated in this way, it is possible to repair an object image that has been removed by mistake while removing noise.

  The difference mask generating means is configured to compare the frame images of the original moving image and the processed moving image frame by frame, for example, and generate the difference mask for each frame. The correction mask generating means detects the moved object image by comparing, for example, a frame image of the processed moving image used for generating the difference mask with at least one of the previous and subsequent frame images. Configured. The final moving image generation means, for example, an image of an area masked by the correction mask in the frame image of the original moving image used for generating the difference mask, of the processed moving image used for generating the difference mask. It is configured to be transplanted to the corresponding area of the frame image. In the image processing apparatus having such a configuration, processing is performed frame by frame.

The difference mask generation means, for example, compares the original moving image and the processed moving image to calculate a difference in luminance value for each pixel, and a pixel having the difference equal to or larger than a predetermined value is extracted from the original moving image by the noise removal process. The difference mask may be generated as the removed region. In such a configuration, since the difference mask masks only pixels having a luminance value difference equal to or greater than a predetermined value, it is possible to reduce the processing load when generating the correction mask.
For example, the correction mask unit may be configured to quantify the motion of the object image and detect the object image as the moved object image when the quantified value is equal to or greater than a predetermined value. In such a configuration, for example, only an object image that has moved greatly can be detected. Since the object image that is erroneously removed is detected as an object image that has moved greatly, even with such a configuration, the object image that is erroneously removed can be correctly restored. In addition, since the number of detected object images is reduced, the processing load can be reduced.
For example, the correction mask generation means may be configured to generate the correction mask that masks an area obtained by enlarging the area where the detected object image is displayed by a predetermined amount. With such a configuration, even when an object image that has been mistakenly removed moves greatly in successive frames of the processed moving image, it can be correctly restored.

  The image processing method of the present invention is a method executed by an image processing apparatus connected to a storage device that stores a processed moving image obtained by performing a predetermined noise removal process on an original moving image including one or more object images. A step of generating a difference mask for masking a region removed as noise from the original moving image by the noise removal processing by comparing the original moving image and the processed moving image; and Detecting a moving object image in the masked area, generating a correction mask for masking at least the area where the detected object image is displayed, and an area masked by the correction mask of the original moving image Transplanting the image into the processed moving image to generate a final moving image.

  The computer program according to the present invention includes a computer system connected to a storage device for storing a processed moving image obtained by performing a predetermined noise removal process on an original moving image including one or more object images, the original moving image and the processed moving image. The difference mask generation means for generating a difference mask for masking the area removed as noise from the original moving image by the noise removal processing, and moves within the area masked by the difference mask from the processed moving image. A correction mask generating means for generating a correction mask that masks at least the area where the detected object image is displayed, and an image of the area masked by the correction mask of the original moving image, A computer program for operating as final moving image generating means for generating a final moving image by transplanting to the processed moving image

  As described above, according to the present invention, the final moving image is generated by repairing the moving region of the object image detected by the processed moving image out of the regions from which the image has been removed by the noise removal processing. Since the object image erroneously removed by the noise removal process is detected as an object image moving in the processed moving image, the object image erroneously removed by the noise removal process can be repaired.

It is a functional block diagram of the image processing apparatus of this embodiment. It is an illustration figure of each frame image of an original moving image and a process moving image. It is an illustration figure of a difference mask. It is an illustration figure of the object image detected by motion detection. It is an illustration figure of the object image detected by motion detection. It is an illustration figure of the mask for correction. It is an illustration figure of the mask for correction. It is a processing procedure figure of image processing.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a functional block diagram of the image processing apparatus of the present embodiment.
Each functional block of the image processing apparatus 1 includes, for example, a general-purpose storage device including a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM), an input / output interface, and a mass storage device such as a hard disk. It can be formed by causing a computer system to execute the computer program of the present invention.
The image processing apparatus 1 includes a noise removal unit 10, a difference mask generation unit 11, a correction mask generation unit 12, a final moving image generation unit 13, an original moving image storage unit 14, a processed moving image storage unit 15, and a final moving image storage unit 16. ing. The original moving image storage unit 14, the processed moving image storage unit 15, and the final moving image storage unit 16 are formed in the mass storage device 17. The difference mask generation unit 11, the correction mask generation unit 12, and the final moving image generation unit 13 execute processing by temporarily storing a frame image and various masks described later in the common work area 18. The work area 18 is composed of a RAM, and a frame image and various masks can be sequentially stored and deleted as necessary.

  The image processing apparatus 1 stores an original moving image digitally converted by a known method from a movie film or the like in the original moving image storage unit 14 in advance. The image processing apparatus 1 generates a processed moving image by removing noise from the original moving image by a conventional digital restoration system. In a processed moving image generated by a conventional digital restoration system, there is a possibility that an object image that originally exists in addition to noise is also judged to be noise and removed. The processed moving image is stored in the processed moving image storage unit 15. The image processing apparatus 1 generates a final moving image by restoring an object image that may have been removed from the processed moving image stored in the processed moving image storage unit 15 and stores the final moving image in the final moving image storage unit 16.

  The noise removing unit 10 performs noise removal similar to the conventional one. The noise removing unit 10 reads out an original moving image from the original moving image storage unit 14 and performs noise removal by a digital restoration system manufactured by the applicant, for example, and generates a processed moving image and stores the processed moving image in the processed moving image storage unit 15. The criterion for determining noise by the noise removing unit 10 is preferably the strictest, and all noise is removed even if an object image that originally exists is removed to some extent.

The difference mask generating unit 11 generates a difference mask representing the difference by comparing the original moving image and the processed moving image of the same frame. The differential mask has a mask image that masks the area removed as noise by the noise removing unit 10. The generated difference mask is temporarily stored in the work area 18. The difference mask generation unit 11 generates a difference mask by comparing the original moving image and the processed moving image pixel by pixel for each frame image. The difference mask generation unit 11 stores each frame image of the original moving image and the processed moving image necessary for generating the difference mask in the work area 18.
For example, as shown in FIG. 2, when a difference mask is generated from the frame image of the nth frame (n is a natural number), the frame images of the nth frame of the original moving image and the processed moving image are stored in the work area 18. Let In each frame image, reference numerals 21 and 21 ′ represent object images, and reference numeral 22 represents noise. The frame image of the original moving image may be deleted immediately after the generation of the difference mask, but the frame image of the processed moving image is not deleted immediately because it is used for other processing.

  FIG. 3 is an exemplary diagram of a difference mask generated by the difference mask generation unit 11 from each frame image of the n-th frame original moving image and processed moving image shown in FIG. According to FIG. 2, the object image 21 and the noise 22 are displayed in the frame image of the nth frame of the original moving image, but the object image in which a part of the object image 21 is removed from the frame image of the nth frame of the processed movie. Only 21 'is displayed. A part of the object image 21 and the noise 22 are determined to be noise and are removed. Therefore, a mask image 23 that masks a partial area of the object image 21 and a mask image 24 that masks the area of the noise 22 are displayed in the difference mask of FIG.

  In the mask images 23 and 24, the difference between the luminance value of the corresponding area of the original moving image and the luminance value of the corresponding area of the processed moving image (hereinafter referred to as “luminance difference”) may be recorded for each pixel. In this case, the difference mask generation unit 11 can determine for each pixel whether or not the processed image has changed significantly with respect to the original image due to the luminance difference. For example, the difference mask generation unit 11 can hold a predetermined threshold value in advance and exclude pixels having a luminance difference smaller than the threshold value from the mask image. Even if it is a difference, a pixel having a small luminance difference is not included in the mask image, so that the processing amount of the subsequent processing can be reduced. In addition, since the brightness difference is small, it is unlikely that a person who views the processed moving image feels uncomfortable.

The correction mask generation unit 12 detects a moving object image by comparing the frame image of the processed moving image that is the source of the difference mask with the frame image of the processed moving image before and after the frame image, and the detection result Based on the above, a correction mask is generated. The motion of the object image is detected within the area masked by the difference mask, and there is no need to perform motion detection for other areas. The correction mask is stored in the work area 18.
Since the object image erroneously removed by the noise removing unit 10 exists in the preceding and following processed moving images, the object image is detected as an object image that has moved greatly by detecting the motion of the object image. Since noise does not exist in the preceding and following processed moving images, it is not detected as an object image that has moved greatly due to motion detection of the object image. Therefore, it is possible to specify an object image that has been erroneously removed by detecting a moving object image using the processed moving image.
The correction mask generation unit 12 quantifies the movement of the object image, for example, when detecting the movement of the object image, and detects the object image when the quantified value is equal to or greater than a predetermined value. Thereby, an object image that has moved by a predetermined amount or more is detected. An object image that has moved by a predetermined amount or more is detected as an object image that has moved greatly. An image that masks the object image that has moved greatly becomes a correction mask.

A method for generating a correction mask will be described with reference to FIGS.
The correction mask generation unit 12 performs block matching on the areas masked by the mask images 23 and 24 of the difference mask using the frame images of the (n−1) th frame and the nth frame of the processed moving image, and (n− 1) An object image that has moved significantly within the area masked by the mask images 23 and 24 from the nth frame to the nth frame is detected. FIG. 4 is an illustration of an object image that has moved significantly as a result of block matching in the area masked by the mask images 23 and 24 based on the frame images of the (n-1) -th and n-th frame processed moving images. is there. In FIG. 4, the object image 25 is detected.
Next, the correction mask generation unit 12 performs block matching on the areas masked by the mask images 23 and 24 of the difference mask using the frame images of the nth frame and the (n + 1) th frame of the processed moving image, and the nth frame. To the (n + 1) th frame of the mask image 23, 24, an object image that has moved significantly within the area masked is detected. FIG. 5 is a view showing an example of an object image that has moved greatly as a result of block matching in the area masked by the mask images 23 and 24 based on the frame images of the processed moving images of the nth frame and the (n + 1) th frame. In FIG. 5, the object image 26 is detected.

  The detected object images 25 and 26 shown in FIGS. 4 and 5 are part of the object image 21 of FIG. This part is an object image that is removed by mistake. The correction mask generation unit 12 uses, as a correction mask, an image obtained by superimposing one of FIGS. 4 and 5 or FIGS. 4 and 5 obtained as a result of motion detection. In this way, the correction mask shown in FIG. 6 including the mask image 27 is generated. In this embodiment, the frame image of the nth frame is compared with each of the frame images of the (n−1) and (n + 1) th frames, but this may be either one. This is because the motion of the object image can be detected by comparison with either one of the frame images.

  Although the correction mask shown in FIG. 6 may be used as it is, a mask image 28 obtained by enlarging the mask image 27 once may be used as shown in FIG. By enlarging the mask image once, it is possible to mask even when it is determined that the object image has moved greatly due to noise generated in consecutive frames of the processed moving image.

  The final moving image generation unit 13 transplants the image of the area masked by the correction mask from the original moving image to the processed moving image. As a result, a frame image of the final moving image is generated. The generated frame image is stored in the final moving image storage unit 16. For all frame images, the final moving image is generated by transplanting the image from the original moving image to the processed moving image using the correction mask generated for each frame. In this specification, the transplantation of an image from an original moving image to a processed moving image includes a case of copying an image in addition to the movement of the image. In the case of image copying, the original moving image can be left as it is.

Such an image processing apparatus 1 performs image processing according to the processing procedure shown in FIG.
First, the image processing apparatus 1 performs noise removal processing on the original moving image stored in the original moving image storage unit 14 by the noise removing unit 10 to generate a processed moving image, and stores the processed moving image in the processed moving image storage unit 15 ( Step S10). For example, when the original moving image is a movie, noise is removed at once for all frames of one movie.

  After generating the processed moving image, the image processing apparatus 1 stores the frame image of the original moving image and the processed moving image frame of the same frame in the work area 18, and uses these frame images of the original moving image and the processed moving image to perform a difference mask A difference mask is generated by the generation unit 11 (steps S20 and S30). Thereafter, the correction mask generation unit 12 stores the frame images of the processed moving image before and after the frame used for generating the difference mask in the work area 18, and generates the correction mask from the frame image of the processed moving image and the difference mask. (Step S40). The generation of the difference mask and the correction mask is performed frame by frame.

  When the correction mask is generated, the image processing apparatus 1 causes the final moving image generation unit 13 to generate a frame image of the final moving image (step S50). Thereafter, it is confirmed whether or not the generated frame image is the image of the last frame (step S60). If the image is not the image of the last frame, the process returns to step S20 and the process for the next frame image is performed (step S60). : N). If it is the last frame image, the process is terminated (step S60: Y). Thereby, the final moving image is generated.

  In the conventional digital restoration system, the noise criterion can be changed by changing a parameter for recognizing it as noise. Most of the noise can be removed by making the judgment criteria the strictest. However, in this case, the originally necessary object image is also removed as noise. The image processing apparatus 1 according to the present embodiment first removes noise, and then generates a final moving image in which only noise is removed from the original moving image by supplementing an object image that has been erroneously removed by removing noise. Yes. Therefore, it is possible to surely remove noise while leaving a necessary image as much as possible. Further, since the operator does not directly check the moving image and does not need to change the parameters in detail, the processing time can be greatly shortened.

  DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 10 ... Noise removal part, 11 ... Difference mask production | generation part, 12 ... Correction mask production | generation part, 13 ... Final moving image production | generation part, 14 ... Original moving image memory | storage part, 15 ... Process moving image memory | storage part, 16 ... Final moving image storage unit, 17 ... mass storage device, 18 ... work area, 21, 21 ', 25, 26 ... object image, 22 ... noise, 23, 24, 27, 28 ... mask image

Claims (7)

By comparing an original moving image including one or more object images and a processed moving image obtained by performing a predetermined noise removal process on the original moving image, an area removed as noise from the original moving image by the noise removal processing is determined. Differential mask generating means for generating a differential mask to be masked;
A correction mask generating means for detecting an object image that has moved within the area masked by the difference mask from the processed moving image, and generating a correction mask that masks at least the area where the detected object image is displayed;
Final moving image generation means for generating a final moving image by transplanting an image of an area masked by the correction mask of the original moving image into the processed moving image,
Image processing device. The difference mask generating means is configured to compare each frame image of the original moving image and the processed moving image frame by frame, and generate the difference mask for each frame,
The correction mask generation means detects the moved object image by comparing a frame image of the processed moving image used for generating the difference mask with at least one of the previous and subsequent frame images. Configured,
The final moving image generating means uses the image of the region to be masked by the correction mask in the frame image of the original moving image used for generating the difference mask as a frame of the processed moving image used for generating the difference mask. Configured to be transplanted into the appropriate area of the image,
The image processing apparatus according to claim 1. The difference mask generation means compares the original moving image with the processed moving image, calculates a difference in luminance value for each pixel, and removes pixels having the difference equal to or larger than a predetermined value from the original moving image by the noise removal processing. Generating the difference mask as a region
The image processing apparatus according to claim 1 or 2. The correction mask means quantifies the movement of the object image, and when the quantified value is equal to or greater than a predetermined value, detects the object image as the moved object image.
The image processing apparatus according to claim 1. The correction mask generating means generates the correction mask for masking an area obtained by enlarging the area where the detected object image is displayed by a predetermined amount,
The image processing apparatus according to claim 4. A method executed by an image processing apparatus connected to a storage device that stores a processed moving image obtained by performing a predetermined noise removal process on an original moving image including one or more object images,
Generating a differential mask that masks a region removed as noise from the original moving image by the noise removal processing by comparing the original moving image and the processed moving image;
Detecting an object image that has moved within an area masked by the difference mask from the processed moving image, and generating a correction mask that masks at least an area where the detected object image is displayed;
Transplanting an image of an area masked with the correction mask of the original moving image to the processed moving image, and generating a final moving image,
Image processing method. A computer system to which a storage device for storing a processed moving image obtained by performing predetermined noise removal processing on an original moving image including one or more object images is connected.
A difference mask generating means for generating a difference mask for masking a region removed as noise from the original moving image by the noise removal processing by comparing the original moving image and the processed moving image;
A correction mask generating unit that detects an object image that has moved within the area masked by the difference mask from the processed moving image, and generates a correction mask that masks at least the area where the detected object image is displayed;
Final moving image generating means for generating a final moving image by transplanting an image of an area masked by the correction mask of the original moving image to the processed moving image;
Computer program to operate as.

Images