EP4260228A1

EP4260228A1 - Method for verifying a digital image

Info

Publication number: EP4260228A1
Application number: EP21830641.3A
Authority: EP
Inventors: Gaël MAHFOUDI
Original assignee: Surys SA
Current assignee: Surys SA
Priority date: 2020-12-08
Filing date: 2021-12-06
Publication date: 2023-10-18
Also published as: FR3117233A1; WO2022122625A1

Abstract

Disclosed is a method for verifying a candidate image, in order to determine whether at least one area thereof has been falsified by image reconstruction, characterised in that the method comprises the steps consisting in: - estimating the reflectance AR(x; y) for the set of pixels (x; y) of the candidate image; - calculating the local variance of the estimated reflectance AR(x, y), for at least one set of pixels (x, y) of the candidate image; - calculating the absolute deviation of the local variance for each pixel (x; y) of the set of pixels (x; y) of the candidate image; - comparing the calculated absolute deviation with a predetermined threshold value; and - if the absolute deviation is less than said predetermined threshold value, then the pixel (x; y) is considered to be an original pixel, otherwise the pixel (x; y) is considered to be a falsified pixel.

Description

Process for verifying a digital image.

Field of invention

The present invention relates to the field of detecting the integrity of a digital image.

For brevity, a digital image will hereinafter be referred to as an "image".

Within the meaning of the present invention, an image is intact if it has not undergone any modification.

The modification, or indiscriminately the falsification, of an original image envisaged here is a painting technique known under the anglicism "Inpainting" and described for example at the following address: https://fr.wikipedia.org/wiki/ lnpainting.

Inpainting is a technique of reconstructing images or filling in missing parts of an image. It is an image synthesis technique which consists of replacing pixels deleted from a predetermined area of the original image by diffusion of adjacent pixels, for so-called "diffusion-based" methods, or by gluing and assembling pixels copied for so-called “examplar-based” methods.

By brevity, hereafter a “falsified” pixel means a pixel modified by inpainting, whether it is diffused or pasted and assembled.

One hears indistinctly inpainting and “image reconstruction technique”.

Unlike other image manipulation techniques, the pixels falsified by inpainting come from the original image and not from another source, it is an intrinsic modification.

Another image manipulation technique is for example that of copied/pasted pixels which consists of an identical duplication of existing pixels. This technique is foreign to the invention because in an inpainting technique, the falsified pixels are not exact duplications of existing pixels. Indeed, unlike copy/paste techniques, when inpainting implements a copy/paste of pixels, the pasted pixels are then also reassembled or rearranged; they are therefore not identically duplicated.

Inpainting is used, for example, to remove all or part of an object or character from an image. The deleted part is replaced by another object, another character, or a background, whether for artistic or fraudulent purposes.

Whatever its purpose, inpainting always involves the creation of a background texture. The user of this technique then works to merge the background texture thus created with the original one so that the modification of the image is the least detectable possible.

There are, for example, inpainting software which are applied to mask a "hole" in an image.

When an image has been processed by inpainting software, it has necessarily undergone prior processing consisting in removing the pixels from an area of the image whose shape and position is defined.

There is therefore a contour corresponding, temporarily, to the border of the area of the deleted pixels.

An inpainting algorithm fills said zone with a set of pixels, the border of which is a transition zone, of the order of a few pixels to a few tens of pixels which separates the original pixels, i.e. the pixels of the original image from the pixels recreated by the inpainting algorithm.

Consequently, the transition zone of such a modified image may appear more blurred than the rest of the image, i.e. the sharpness of the latter may locally decrease.

The present invention aims to cleverly exploit this characteristic.

In this context, the present invention aims, according to a first of its objects, at a method for verifying a candidate image, to determine whether the latter has undergone falsification by image reconstruction.

It is essentially characterized in that it includes steps consisting of:

- determine the estimated reflectance ^A R(x; y) for all the pixels (x; y) of the candidate image,

- for each pixel (x; y) of at least one window having a predetermined shape and predetermined dimensions, centered on said pixel (x, y), and comprising a group of pixels (x; y) of the candidate image, determining the local variance of the estimated reflectance ^A R(x; y) for the group of pixels of said window,

- calculate the absolute deviation of the local variance of the estimated reflectance ^A R(x; y) for each pixel of each group of pixels (x; y) of the candidate image,

- compare the calculated absolute deviation with a predetermined threshold value,

- if the absolute deviation is less than said predetermined threshold value, then said pixel (x; y) is considered as an original pixel, otherwise said pixel (x; y) is considered as a pixel falsified by image reconstruction, and

- emitting a signal representing an alert if the number of pixels (x; y) considered to be falsified by image reconstruction is greater than a predetermined threshold value.

It is also possible to provide a step of displaying a heat map, consisting in assigning a predetermined color value to all of said pixels (x; y) considered to be falsified.

It is also possible to provide a step consisting in assigning a predetermined color value to all of said pixels (x; y) considered as original.

Provision may be made for the step of calculating the absolute deviation to comprise the calculation of the absolute deviations of the local variances of the estimated reflectance ^A R(x; y) for a plurality of groups of pixels (x; y), and that each group corresponds to a respective color.

Provision can be made for the calculation of the absolute deviation to comprise the calculation of a single absolute deviation of the local variance of the estimated reflectance ^A R(x; y) for all of the pixels (x; y) of the candidate image.

There may further be provided a step consisting in dividing the candidate image into a plurality of sub-images, which do not overlap and which may be adjacent or distant, in which the step of calculating the average comprises calculating the average of the estimated reflectance ^A R(x; y) for each sub-image.

Provision can be made for the candidate image to be an image representing all or part of a valuable document, in particular of a data page of an identity document, of a visa, of a check, etc. Provision can be made for the identity document to include an automatic reading zone (MRZ), the method comprising a step consisting in selecting as candidate image at least one of:

- an image of the machine-readable zone (MRZ),

- a field relating to the identity of the bearer of said identity document, and

- a field relating to the identity document such as its date of issue, its date of validity, the authorization (for example for a driving licence), etc...

It is possible to provide a step consisting in increasing the ISO grain of the candidate image beyond a predetermined threshold value, during the capture of said candidate image.

According to another of its objects, the invention relates to a computer program comprising program code instructions for the execution of the steps of the method according to the invention, when said program is executed on a computer.

By computer, we mean a personal computer, but also a tablet, a smartphone or a server.

Other characteristics and advantages of the present invention will appear more clearly on reading the following description given by way of illustrative and non-limiting example and made with reference to the appended figures.

tricks

[Fig. 1] illustrates a candidate image presenting two different textures,

[Fig. 2] illustrates the result of applying a median filter to the image in Figure 1,

[Fig. 3] illustrates a binary segmentation of the image of figure 2,

[Fig. 4] illustrates another candidate image, representing a pen,

[Fig. 5] illustrates the image of figure 4, falsified,

[Fig. 6] illustrates a heat map according to the invention of the candidate image of figure 5,

[Fig. 7] illustrates another candidate image, representing part of a MRZ, falsified,

[Fig. 8] illustrates a heat map according to the invention of the candidate image of figure 7.

- Detailed description -

Within the meaning of the present invention, an image is an image from a photo/video sensor or a synthetic image.

The image is in grayscale or preferably in color, but not in black and white.

It is called a candidate image in that it is a question of determining, thanks to the invention, whether this candidate image is an original image or an image modified by inpainting, that is to say falsified.

Preferably, the candidate image has at least one of the following characteristics:

- sufficient definition, i.e. a total number of image pixels greater than a predetermined threshold value, - a weak compression before the possible falsification, that is to say whose compression ratio is lower than a predetermined threshold value. In other words, the falsified image must not have a high compression rate.

Preferably, when possible, the original image presents an increase in noise, a grain, by increasing the ISO beyond a threshold value, during the capture of the original image, which generates a kind of texture which is very difficult to reproduce by an inpainting algorithm, and which facilitates the detection of a possible falsification.

To simplify the present description, only the case is considered here in which a candidate image is an image of a data page of a valuable document.

Valuable document means any official identity document or security document, i.e. issued by a government or authority (identity card, passport, driver's license, birth certificate, work, visa, etc.) or unofficial (subscription card, loyalty card, show or transport ticket, etc.).

*Reflectance*

In digital form, a candidate image can be modeled as the product of the light intensity and the reflectance of its object.

We can thus define an equation (1) by l(x; y) = L(x; y) * R(x; y), such that:

I is the picture,

(x, y) are the Cartesian coordinates of the pixels of the image I,

L is the light intensity of the lighting source, i.e. the quantity of light (real or simulated for a computer-generated image) reaching the surface of the valuable document at a point corresponding to the coordinate pixel (x, y) of the image, and R is the reflectance, which is the proportion of light reflected from the surface, in this case of the valuable document, from which the original image originated.

We can take the logarithm of equation (1).

We thus obtain equation (2): ln(l (x; y)) = ln(L(x; y)) + ln(R(x; y))

In the frequency domain, filtering an image amounts to performing a mathematical operation of multiplication. And it is faster to calculate the result of a multiplication than a convolution (in the spatial domain).

Equation (2) is therefore advantageously transformed into the frequency domain, by applying a Fourier transform F to it, i.e. equation (2') such that: F(ln( I (x; y))) = F( ln(L(x; y)) + ln(R(x; y)))

The reflectance R being in the high frequencies, we can apply a high-pass filter H in the Fourier domain to equation (2'),

We therefore have: F(ln(l(x; y))) H(u; v) = [F(ln(L(x; y)) + ln(R(x; y)))] H(u v) Let F(ln(l(x; y))) H(u; v) = F(ln(L(x; y)) H(u; v) + ln(R(x; y)) ) H(u; v)

By suppressing the low frequencies thanks to the H filter, we have:

F(ln(l(x; y))) H(u; v) = F(ln(L(x; y)) H(u; v) which makes it possible to obtain equation (3): r (u; v) = F(ln(l(x; y)))H(u; v) with

- (u; v) the frequencies corresponding to the coordinates (x, y), H a high-pass filter in the Fourier domain,

F the Fourier transform,

- r a change of calculation variable such that r(u; v) = F(ln(L(x; y)) H(u; v)

It is thus possible to estimate the reflectance ^A R by calculating the exponential of the inverse Fourier transform of equation (3).

We thus obtain equation (4): ^A R(x; y) = exp(F ¹ (r(u; v)))

The estimated reflectance ^A R(x; y) is an estimate of the surface roughness at pixel (x; y). That is, the more the reflectance ^A R varies around the pixel of coordinates (x; y), the rougher the image texture is in the area of the image around this point.

A reflectance matrix is thus calculated, which gives the value of the estimated reflectance ^A R(x; y) for a pixel (x; y) of the candidate image.

*Local Variance of Reflectance*

Provision is made to calculate the local variance of the estimated reflectance ^A R(x; y) for at least one group of pixels (x; y) of the candidate image.

In this case, for a pixel (x; y), we take a predetermined shape whose dimensions are predetermined, also called "window", and which is centered on said pixel (x, y), and we determine the local variance the estimated reflectance ^A R of said pixel (x; y) within said predetermined shape.

For example, the local variance is calculated for a rectangular window of size NxM pixels centered around the pixel (x, y), and in this case a square of 11 pixels side. For the pixels at the edges of the candidate image, a symmetry technique or another technique known to those skilled in the art can be used.

It is then possible to calculate, for each group of pixels, the variance of the local variances, also called “group variance”.

In a first variant,

Provision is made to calculate a single local variance of the estimated reflectance ^A R(x; y) for a group of pixels which corresponds to all the pixels (x; y) of the candidate image.

In a second variant, provision is made to calculate the local variance of the estimated reflectance ^A R(x; y) for a plurality of groups of pixels (x; y), in which each group corresponds to a respective color for all of the pixels (x; y) of the candidate image.

Each color is determined by RGB values which are within predetermined values.

Thus, for each color, the respective local variance is calculated for all the pixels (x; y) of each group. In a third variant, provision can also be made for each group of pixels to correspond to a block of adjacent pixels two by two, independently of the texture that they represent or of their color.

We can thus divide the candidate image into several adjacent sub-images, or blocks, which do not overlap, and we thus calculate for each block the respective local variances for all the pixels (x; y) of the block. The shape and dimensions of the sub-images are predetermined and may differ from one sub-image to another.

In a fourth variant, provision is made for each group of pixels to correspond to a block of pixels whose pixels are adjacent two by two and represent the same texture, independently of their color; for example by an image segmentation algorithm.

Finally in a fifth variant, a sub-image is extracted from the candidate image and at least one of the 4 variants above is applied to this sub-image, replacing all the pixels (x; y) of the candidate image by all the pixels (x; y) of the sub-image. For example, for a candidate image such as a passport, a sub-image can be a predefined area of the passport, in particular the MRZ as shown in Figure 7.

*Absolute Deviation from Local Variance of Reflectance*

Whatever the above variant, for each pixel (x; y) of a given group of the candidate image, the absolute deviation of the local variance of the estimated reflectance ^A R of said pixel (x; y) is measured. with respect to all the pixels (x; y) of said group of pixels according to the variant implemented.

In a first embodiment, it is then possible, for each pixel (x; y) of a group of pixels, to calculate the difference between:

- the local variance of the estimated reflectance ^A R of each pixel (x; y) of said group of pixels and

- the variance of the estimated reflectance ^A R of all the pixels (x; y) of said group of pixels, that is to say the variance of the group.

Calculating the group variance is optional.

Alternatively, in a second embodiment, it is indeed possible to determine reference values which are group variance values, for all the groups. These reference values are obtained for example in a learning phase, thanks to measurements made on original images. For example, the group variances for images of standardized documents, typically identity documents, vary little from one image to another. We can therefore assign a single reference value per group.

In this case, we can then, for each pixel (x; y) of a group of pixels, calculate the difference between:

- the reference value.

Preferably, the reference values are indexed to a value document type. Preferably, the acquisition conditions of the candidate image are known and the same for the original images during the learning phase. This second embodiment generally allows faster calculations than the first embodiment.

Either of the difference calculations according to the 2 embodiments above makes it possible to determine whether the local variance of a pixel is abnormal with respect to the variance of the group.

Provision may be made to compare the result of the difference calculation with a predetermined threshold value, which makes it possible to binarize the result.

For example, for the second variant, to determine the pixels of the same color, we take all the pixels in a given color slice, for example the green pixels, and for each green pixel, we calculate the absolute deviation of the local variance of the reflectance. And so on for the other colors.

For a given color, the local variance of the reflectance of each pixel (x; y) is therefore compared to the median of the local variances of the reflectance of all the pixels of said color.

Whatever the variant, if the absolute deviation is less than a predetermined threshold value, then said pixel (x; y) is considered as an original pixel, otherwise said pixel (x; y) is considered as an falsified pixel.

It is then possible to display on the candidate image all of said pixels (x; y) considered as falsified pixels with a predetermined RGB value, and thus obtain a heat map.

It is also possible, for the production of a heat map, to assign all the pixels considered as falsified pixels to a predetermined RGB value, possibly the same value. Optionally, moreover, it is possible to allocate all the pixels considered as original pixels to another predetermined RGB value, which makes it possible, by contrast effect, to bring out the pixels considered as falsified pixels.

It is also possible, for the production of a heat map, to allocate all the pixels considered as falsified pixels with an RGB value where the intensity depends on the difference between the local variance of the estimated reflectance ^A R of said pixel (x; y) and the variance of the group.

The greater the difference between the local variance of the estimated reflectance ^A R of said pixel (x; y) and the variance of the group, the greater the probability that said pixel (x; y) is falsified.

To illustrate this principle, it can be provided that the greater the value of the deviation, the greater the intensity assigned to said pixel, or the more the color assigned to it tends towards red. Conversely, the lower the value of the deviation, the lower the intensity assigned to said pixel, or the more the color assigned to it tends towards blue.

The present invention thus makes it possible to locate, on the candidate image, the falsified zone. So, if the falsified area on a passport is part of the background, it may be an artifact. On the other hand, if the falsified area on a passport is part of the MRZ or a field relating to the identity of the bearer, then the candidate image has certainly been falsified.

In addition, thanks to the heat map, it is possible to determine the position of the falsified area in the image.

For this purpose, it is also possible to provide a step of selecting areas of predetermined shape and position, for example the MRZ of a passport, or areas of alphanumeric fields whose position is known, for example the surname, first name, date of birth etc. of the bearer of the identity document. It is then possible to treat only those zones which are the most likely to be modified, which makes it possible to save treatment time.

* Examples *

Figure 1 illustrates a candidate image presenting two different textures: a fine texture in its left part and a smoother texture in its right part.

By applying a median filter to the image of figure 1, we obtain figure 2. The median filter makes it possible to reduce the noise.

Figure 3 is obtained by binarizing figure 2, in this case with an Otsu binarization.

Figure 3 clearly illustrates that the texture on the left is overall sharper than the one on the right, and that the texture on the right, the white areas on a black background, actually correspond to sharper areas, see the corresponding areas in Figure 2.

This principle set out in FIGS. 1 to 3 is used to detect the sharpness transition zones on a candidate image.

For example, figure 4 represents a pen. This figure has been tampered with inpainting by deleting this one, as shown in figure 5.

The entire tampered zone can be identified on the heat map according to the invention illustrated in Figure 6.

This principle has been applied to identity documents, in this case to a passport.

For example, in the field of identity documents, some frauds consist of falsifying an image of the data page by deleting a text area and then rewriting another text in this area.

This type of falsification is sometimes used in the field of remote check-in for boarding an air flight, an image of the identity document is sent to a control center. The present invention makes it possible, for example, to verify that this image has not been digitally retouched.

Figure 7 illustrates a portion of the machine-readable zone (MRZ) that has been tampered with by inpainting. The falsification consisted of deleting the name of the bearer of the document and replacing it with the fictitious name “OLIVERA” followed by a “<” character.

This falsification is clearly detected by the method according to the invention, whose heat map corresponding to Figure 7 is illustrated in Figure 8.

The invention is not limited to the embodiments previously described.

Provision can be made to eliminate from calculations of variance and average, pixels that are too bright or too dark, i.e. pixels whose light intensity value is greater than a predetermined threshold value or less than another predetermined threshold value.

Whatever the variant, for a given pixel (x; y), the higher the variance of the estimated reflectance ^A R, the sharper the image in the local neighborhood of said pixel (x; y). Alternatively to the calculation of the mathematical variance of the estimated reflectance ^A R, it is possible to implement a calculation of the gradient of the estimated reflectance ^A R, according to a set of at least one predetermined direction with respect to said pixel (x; y).

The falsification of an original image generates contours which are not necessarily visible on the candidate image. It is also possible to implement an edge detection step on the heat map and on the original image.

We can then compare the contours of the heat map and the original image to detect contours which are on the heat map and which are not on the original image. All the pixels contained in the closed contour(s) thus detected are considered to be falsified pixels.

Thanks to the invention, it is possible to exercise a sort of self-control of a candidate image, that is to say that the candidate image is sufficient unto itself, which makes it possible to free from a possible reference database, which will typically include an original image to which the candidate image would be compared.

Claims

[Claim 1] Method for verifying a candidate image, to determine whether the latter has undergone falsification by image reconstruction, characterized in that the method comprises steps consisting of:

- calculate an absolute deviation of the local variance of the estimated reflectance ^A R(x; y) for each pixel of each group of pixels (x; y) of the candidate image,

- if the absolute deviation is less than said predetermined threshold value, then consider said pixel (x; y) as an original pixel, otherwise consider said pixel (x; y) as a pixel falsified by image reconstruction, and

[Claim 2] Method according to claim 1, further comprising a step of displaying a heat map, consisting in assigning a predetermined color value to all of said pixels (x; y) considered to be falsified.

[Claim 3] A method according to any preceding claim, further comprising a step of assigning a predetermined color value to all of said pixels (x; y) considered to be original.

[Claim 4] A method according to any preceding claim, wherein the step of calculating the absolute deviation comprises calculating the absolute deviations of the local variances of the estimated reflectance ^A R(x; y) for a plurality of groups of pixels (x; y), in which each group corresponds to a respective color.

[Claim 5] A method according to any of claims 1 to 3, wherein the step of calculating the absolute deviation comprises calculating a single absolute deviation of the local variance of the estimated reflectance ^A R(x; y ) for all pixels (x; y) of the candidate image.

[Claim 6] A method according to any preceding claim, further comprising a step of dividing the candidate image into a plurality of sub-images, which are non-overlapping and which may be adjacent or distant, wherein the The step of calculating the absolute deviation comprises calculating the absolute deviations of the local variances of the estimated reflectance ^A R(x; y) for each sub-image.

[Claim 7] A method according to any preceding claim, wherein the candidate image is an image representing all or part of a data page of a document of value.

[Claim 8] A method according to claim 7, wherein the document of value is an identity document comprises a machine-readable zone (MRZ), the method comprising a step consisting in selecting as candidate image at least one among:

- an image of the machine-readable zone (MRZ),

- a field relating to the identity of the bearer of said identity document, and - a field relating to the identity document.

[Claim 9] A method according to any preceding claim, comprising a step of increasing the ISO grain of the candidate image above a predetermined threshold value, when capturing said candidate image.

[Claim 10] A computer program comprising program code instructions for carrying out the steps of the method according to any preceding claim, when said program is executed on a computer.