WO1999055080A1

WO1999055080A1 - Method and equipment for detecting and correcting impulsive faults in a digital video sequence image

Info

Publication number: WO1999055080A1
Application number: PCT/FR1999/000864
Authority: WO
Inventors: Louis Laborelli
Original assignee: Ina - Institut National De L'audiovisuel (Epic)
Priority date: 1998-04-17
Filing date: 1999-04-13
Publication date: 1999-10-28
Also published as: FR2777728A1; FR2777728B1; AU3153699A

Abstract

The invention concerns a method for detecting and correcting impulsive faults occurring on a digital video sequence image to be processed comprising a confirmation function consisting in a matching test between the pixels of two consecutive analysis zones.

Description

METHOD AND EQUIPMENT FOR DETECTION AND CORRECΗON OF IMPULSE DEFECTS IN A DIGITAL VIDEO SEQUENCE IMAGE

The present invention relates to a method for detecting and correcting impulsive faults appearing on an image to be processed of a digital video sequence and equipment for implementing this method. The field of application of such a method relates particularly to the reprocessing and restoration of cinematographic films or videos having undergone a degradation of the support (scratches, dust, local defect of the magnetization, ...). Such operations can be carried out manually by an operator analyzing each image to detect the faults and then treat the faults identified. It is tedious work. The advantage of using the human operator is however to allow the discrimination of actual faults and false faults corresponding to other transient appearances having only the appearance of faults. Solutions aimed at automating the identification of faults and the reprocessing of these faults do not always allow satisfactory discrimination. They thus result in the omission of real defects, and, more serious, in the assimilation to defects of certain normal elements of the image. This latter situation leads to the removal of certain significant details from the image, which is difficult to accept. For example, the presence of dust on the film will obscure the path of the rays and will be manifested by a dark area on the screen which will replace the part of the scene filmed. If this event occurred on the negative or if the colored layer was damaged, this area will be white. Likewise, video documents recorded on tape exhibit temporary signal losses resulting in horizontally oriented areas containing an arbitrary signal. A large variety of shapes and densities is encountered experimentally. Among the solutions known in the prior art, there will be mentioned the European patent EP636299 which describes a method for detecting and eliminating errors. exceeding a specific contrast in digital video signals.

The state of the art of detection techniques is based on a test performed locally, pixel by pixel. It consists in comparing the gray level of the current image to that of the reference in the previous image, and in the following image, these two images being subjected to prior compensation for movement, by means of known methods. If these two comparisons give a high difference simultaneously, the pixel is considered to be part of an erroneous area, based on the fact that the pixels of a normal sequence follow more or less long time trajectories, from an appearance to a disappearance, but with a duration greater than an image. This method has the disadvantage of requiring a perfect match between the movement in the scene and the movement model supposed to be identified. Thus, a movement that is too fast, or with a strong rotation component or a strong local scale variation (expansion or contraction) will pose serious problems for most estimators of translational movement. This erroneous identification will lead to supposing that the current pixel has no reference, neither in the preceding image, nor in the following. The invention aims to improve the discrimination of real faults and therefore aims to extend the known methods of detecting impulsive events, by a posteriori processing, intended to verify this absence of relationship, even in the case of zero knowledge of the movement .

To this end, the invention relates in its most general sense to a method for detecting and correcting impulsive faults appearing on an image to be processed in a digital video sequence, comprising the following steps: - determination for an image I _n of zones Z _{n> 1} correlated to graphic events likely to correspond to defects

- determination for each of the Z _{n #} i of at least one analysis zone Z'i;

- calculation, for each of the i zones Z _n , i of a doubt removal function F by comparison of the descriptor of the zone Z _nιl with the descriptors of the corresponding analysis zones Z'i - modification of the graphical data correlated to the zones Z _n> i for which the function Fi is true characterized in that the doubt removal function is a matching test between characteristics extracted from the pixels of the area _n , i and characteristics extracted from the pixels of the analysis areas' i.

The purpose of modifying the graphic data is to correct the defect confirmed by the function Fi by known image processing solutions. Preferably, invariant descriptors of the spatial permutations are determined for the zones Z _n> i and the zones Z ′ i.

In this patent, the term "analysis area" designates areas deriving in a determined manner from the area correlated to a potential defect. These analysis areas belong to one or more different images of

The image I _n being the subject of the processing. It can be the same image I _{n /} after unconditional correction of all the potential faults identified, before the application of the doubt removal function. It can also be homologous zones belonging to the anterior images I _n -ι and posterior I _{n +} ι to the image I _n .

For the purposes of the present patent, the term “homologous zones” will be understood to mean an area of a first image, an area of a second image whose dimension and / or position results from the application of a predetermined, constant or function of certain determinable parameters by analysis of an enlarged area of the image, such as a global or local motion vector. Is considered as homologous of a zone Z _n of the image l _n a zone Z _{n ±} ι of the image I _{n ±} ι preceding or following, resulting from the application of an operation of:

- dilation

- determination of a bounding box

- displacement (displacement compensation)

- change of scale - combination of all or part of these operations.

An effective approximation of the process consists in cutting the image into blocks and in treating in a global manner all the pixels detected in a block of the grid.

Within the framework of this approximation, the homologous zones can have a form fixed a priori, such as for example a set of blocks obtained by the transformations mentioned above, from the block considered Z _n .

According to a preferred embodiment, said descriptors consist of a histogram of characteristics representative of the pixels of the image areas.

According to a first variant, said descriptors consist of a histogram of the luminance distribution.

According to a second variant, said descriptors consist of a histogram of the color distribution.

According to a third variant, the descriptors consist of a histogram of the textural criteria.

Advantageously, the matching test consists in comparing the descriptor of the zone Z _n> i of the image I _n with the descriptor of the analysis zones Z'i to determine the degree of agreement, the function of doubt removal being true when the degree of agreement is low for the analysis zones Z'i. According to a fourth variant, the descriptors are constituted by a measurement of the quantity of contours present at the border of the zones Z _n> i.

For this fourth variant, it is advantageous for the matching test to be a comparison of a measurement of the quantity of contours present at the border of the zones Z _n> i and of a measurement of the quantity of contours present at the border Z'i zones, the doubt removal function being true when the degree of agreement between these two measurements is strong for the Z'i analysis zones.

According to a first embodiment, the analysis zones Z ′ are homologous zones _{n ±} ι ₍ i belonging to the anterior images I _n -ι and posterior I _{n +} ι and deriving from the zone Z _n, i. Preferably , the zones Z _{n ±} ι, i homologous to the zones Z _n> i are zones whose location in the images In _{± l} of the center of the zone are identical to the coordinates of the center of the zone Z _{n /} i in the images I _{n /} and whose dimensions are greater than the dimensions of the zone Z _n , i.

According to a second embodiment, the analysis zones Z'i are border zones Z _{f / i} of zones Z ^. , i of the image I ' _n corresponding to the image I _n after unconditional correction of the graphic events likely to correspond to defects in the zone Z _{n /} i. According to this second mode of implementation, the correction of the potential faults is applied for the first time, and the descriptors of the images are compared before and after application of this first correction to determine an error raising function Fi for each of the correlated zones to potential defects, to correct and ^'actually confirmed that the defects by the process according to the invention. The invention also relates to a digital image processing device comprising means for storing the image I _n , means for signaling zones Zi liable to contain defects, calculation means for analyzing analysis zones Z 'i associated with zones Z _n i and to determine, for each of the i zones Z _{n <} i a function for removing doubt Fi by comparison of the descriptor of the zone Z _n> i with the descriptors of the corresponding zones Z'i consisting of a test of pairing between the pixels of the zone Z _{n <} i and the pixels of the analysis zones Z'i and means for modifying the graphic data in the zone Z _n , i when the function F is true.

The invention also relates to a device for processing digital images for implementing a method according to the preceding descriptions, characterized in that it comprises means for storing three consecutive images I _n -ι- In and I _{n + l} - of means for signaling zones Zi liable to contain faults, calculation means for analyzing zones Z _{n ±} ι ₍ i corresponding to zones homologous to zones Z _n , i on the images I _{n ±} ι and for determining , for each of the i zones Z _n> i of a doubt removal function Fi by comparison of the descriptor of the zone ^z n, i with the descriptors of the corresponding zones Z _n ± ι _# i consisting of a matching test between the pixels of the zone Z _n , i and the pixels of two zones Z _{n ±} ι, i of the images l _{n ±} ι consecutive and means for modifying the graphic data in the zone Z _{n #} i when the function Fi is true.

The invention will be better understood on reading the description which follows, relating to a nonlimiting example of implementation of the invention.

The method comprises a first step of determining a window around three consecutive images, as well as a map of the detection hypotheses which will have to be confirmed. The general idea is to avoid making a decision that is too local, but to verify that a pairing between all the pixels of the hypothetical spot and a temporal and / or spatial neighborhood belonging to the previous and / or next image, around localization is possible. It suffices to find a pairing with a previous area or an area of the next image to lift the detection hypothesis. In general, the method consists in measuring a certain number of characteristics on the pixels of the spot, such as the distribution of luminance or color.

The number of detectable characteristics is very large and extends for example also to textural criteria.

An area is characterized by a histogram, possibly multidimensional of these characteristics. This method will have the advantage of being invariant by any spatial permutation of the characteristics, and therefore independent of a poor motion estimate.

We will then try to place this histogram in those, obtained in the same way in the previous image, and in the following image. The comparison is carried out, after optional normalization by the size of the initial histogram, by comparing the size of each cell of the histogram, with the corresponding cell in the preceding and following images. Any overshoot leads to a penalty. Each size of a cell in the initial histogram can be the subject of a multiplying factor, before the comparison.

Exceeding a global penalty will lead to classifying the task in the unmatched category. If it is not matched to the future and to the past, it is confirmed in its impulsive zone status. We can simply calculate the histograms on the basis of a grid superimposed on the image. This approximation leads to grouping together all the zones present in a block of the grid in the form of a super zone, which is not necessarily connected. The analysis zones will then be obtained simply in the form of a rectangular zone of several blocks, after application of a geometric relation to the block of the grid, possibly determined by the movement estimated in the sequence of images. The histogram of the analysis zone is then calculated by summing, possibly weighted, the histograms present in all the blocks concerned. This computation can be seen as a convolution independent of each box of the histogram by a low-pass spatial filtering kernel.

This summation can itself be accelerated in the case of uniform weighting, by modifying the summation corresponding to a neighboring window, if the windows share pixels. All spatial low pass filtering techniques can be applied.

For the fraction of the spot present in an element of the grid, the calculation of the inclusion test can be carried out implicitly by counting the number of pixels which do not enter the target histogram. This therefore avoids comparing null zones. Other implementations are possible, in hardware or software form.

The results seem to show a very strong decrease in false alarms over a large area, while small areas that are falsely detected are more difficult to dismiss. These results are dependent on the method for normalizing the histogram and on the comparison function. It is desirable to limit the use of the method to the detection of large surface defects.

A variation of the method consists in this time comparing not the pixel statistics of the initial images, but statistics obtained on the spatial variations of the images, at the borders of the areas in question, before and after correction. The first correction is obtained for example by interpolation compensated by the movement between the previous and next images. This repair can introduce new local variations in the reconstructed image.

Each increase in local variations (contours) of the image, will contribute to a vote in favor of rejection of confirmation of the spot. This comparison is based on the principle that the correction operation should not not to introduce new discontinuities in the image, which were not originally present there, but on the contrary to decrease their number. The histogram of the image is not calculated here, but that of the variations of this image, in order to measure the appearance of new contours at the border of the modification. We will use the knowledge of the initial error detection card, in order to count the contours only in the immediate vicinity of the borders of an initial detection. In the case of an implementation by grid, these histograms can be calculated for each block of the grid independently.

This variation can be applied jointly with the preceding methods, by weighting of the votes, or logical or non-linear operations, on the results of the votes. In the case of an implementation on a grid, the decision will be made on each box independently.

Claims

1 0

R E V E N D I C A T I O N S

1 - Method for detecting and correcting impulsive faults appearing on an image to be processed in a digital video sequence, comprising the following steps:

- determination for an image I _n of zones Z _{n #} i correlated to graphic events likely to correspond to defects

determination for each of the Z _n , i of at least one analysis zone Z'i;

- calculation, for each of the i zones Z _n> i of a doubt removal function Fi by comparison of the descriptor of the zone Z _{n <} i with the descriptors of the corresponding analysis zones Z'i - modification of the correlated graphical data to areas Z _{n #} i for which the function Fi is true characterized in that the doubt removal function is a matching test between the pixels of the area Z _n , i and the pixels of the analysis areas Z'i .

2 - Method for detecting and correcting impulsive faults according to claim 1 characterized in that one determines for the zones Z _n> and the analysis zones Z'i invariant descriptors of the spatial permutations.

3 - A method of detecting and correcting impulsive faults according to claim 2 characterized in that said descriptors consist of a histogram of characteristics representative of the pixels of the image areas.

4 - Method for detecting and correcting impulsive faults according to claim 3 characterized in that said descriptors are constituted by a histogram of the luminance distribution. 1 1

5 - A method of detecting and correcting impulsive faults according to claim 3 or 4 characterized in that said descriptors consist of a histogram of the color distribution or of the joint color / luminance distribution.

6 - Method for detecting and correcting impulsive faults according to claim 3 characterized in that said descriptors consist of a histogram of textural criteria, possibly colored.

7 - Method for detecting and correcting impulsive faults according to any one of claims 4 to 6 characterized in that the matching test consists in comparing the descriptor of the area Z _{n /} i of the image I _n with the descriptor of the analysis zones Z'i to determine the degree of agreement, the doubt removal function being true when the degree of agreement is low for the homologous zones Z'i.

8 - Method for detecting and correcting impulsive faults according to claim 2 characterized in that said descriptors are constituted by a measurement of the quantity of contours in the border area of the area Z _n, i.

9 - Method for detecting and correcting impulsive faults according to claim 8 characterized in that the matching test consists in comparing the descriptor of the area Z _n> i of the image I _n with the descriptor of the analysis areas Z'i to determine the degree of agreement, the doubt removal function being true when the degree of agreement is strong for the homologous zones Z'i.

10 - Method for detecting and correcting impulsive faults according to any one of the preceding claims, characterized in that the analysis zones Z'i 1 2

are homologous zones _{n ±} ι _# i belonging to the anterior images I _n -ι and posterior I _{n +} ι and deriving from the zone Zn, i •

11 - Method for detecting and correcting impulsive faults according to the preceding claim characterized in that the zones Z _n ± ι _# i homologous to the zones Zn, i are zones whose location in the images I _{n ±} ι of the center of the zone are identical to the coordinates of the center of the zone Z _n , i in the images I _n , and whose dimensions are greater than the dimensions of the zone Z _{n <} i.

12 - Method for detecting and correcting impulsive faults according to any one of claims 1 and 6 characterized in that the analysis zones Z'i are border zones Z _f , i determined on the edge of the zones Zn ', i of the image I ' _n corresponding to the image I _n after unconditional correction of the graphic events likely to correspond to defects in the zones Z _{n # i} .

13 - Device for processing digital images for the implementation of a method according to at least one of the preceding claims, characterized in that it comprises means for storing the image I _n , signaling means of areas Zi likely to contain faults, calculation means for analyzing analysis areas Z'i associated with areas Z _n , i and for determining, for each of the i areas Z _n i, a function for removing doubt Fi by comparison of the descriptor of the zone Z _nj i with the descriptors of the corresponding zones Z 'i consisting of a matching test between the pixels of the zone Z _n> i and the pixels of the analysis zones Z'i and means for modifying the graphic data in the zone Z _{n <} i when the function Fi is true. 1 3

14 - Device for processing digital images for the implementation of a method according to claim 1 characterized in that it comprises means for storing three consecutive images I _n -ι <I _n and I _{n + l '} means for signaling zones Zi liable to contain faults, calculation means for analyzing zones Z _{n ±} ι, i corresponding to zones homologous to zones Z _n, i on the images I _{n ±} ι and for determining, for each i zones Z _{n <} i of a doubt removal function Fi by comparison of the descriptor of the zone Zn, i with the descriptors of the corresponding zones Z _{n ±} ι _# i consisting of a matching test between the pixels of the zone Z _n , i and the pixels of two zones Z _{n ±} ι, i of the images I _{n ±} ι consecutive and means for modifying the graphic data in the zone Z _n> i when the function Fi is true.

15 - Device for processing digital images for the implementation of a method according to claim 11 characterized in that it comprises means for signaling zones Zi likely to contain defects, means for calculating an image I ′ _n deriving from the image l _n by the unconditional correction of the potential defects of the image I _n , means for memorizing the image I _n and the image I ′ _n , calculation means for analyzing zones Z ′ i corresponding to the edges of the analysis zones on the image I * _n and to determine, for each of the i zones _{n #} i, a function for removing doubt Fi by comparison of the descriptor of the zone Z _n , i with the descriptors of the corresponding zones Z'i, consisting of a matching test between the pixels of the zone Z _n , i and the pixels of the zone Z 'i of the images I' _n and means for modifying the graphic data in the zone Z _n , i when the function F is true.