FR3060180A1 - DIGITAL IMAGE PROCESSING METHOD - Google Patents

Abstract

The invention relates to a method for processing an initial image comprising a subject and a background, comprising: - if the initial image is in color, transforming (100) at least one of the channels of the initial image into an image in grayscale, and - record said grayscale image in a memory. It is essentially characterized by: determining (110) the contrast of the grayscale image in order to obtain a matrix comprising a set of coefficients each corresponding to a pixel (P) of the grayscale image, comparing (140) the value of each coefficient with a possibly respective threshold value (Th), to obtain a binarized image, - selecting (145) the coefficients greater than the threshold value, in order to obtain a binary mask comprising a set of outlines, - fill (150) the inside of the contours of the binary mask homogeneously.

Description

Holder (s): CYCLOPUS Simplified joint-stock company.

Extension request (s)

Agent (s): INNOVATION COMPETENCE GROUP.

loT) PROCESS FOR DIGITAL IMAGE PROCESSING.

FR 3 060 180 - A1 _ The invention relates to a method for processing an initial image comprising a subject and a background, comprising:

- if the initial image is in color, transform (100) at least one of the channels of the initial image into a grayscale image, and

- Save said grayscale image in a memory.

It is essentially characterized by:

- determining (110) the contrast of the gray level image in order to obtain a matrix comprising a set of coefficients each corresponding to a pixel (P) of the gray level image,

- compare (140) the value of each coefficient with a threshold value (Th), possibly respective, to obtain a binarized image,

- select (145) the coefficients greater than the threshold value, in order to obtain a binary mask comprising a set of contours,

- fill (150) the inside of the contours of the binary mask evenly.

I 100 * 110 I ¹³⁰ I -►! 140 p pl 145 | -►! 150 P 180 I ¹²⁰ I I ¹⁹⁰ I

zx:

160 zxz

170 i

DIGITAL IMAGE PROCESSING METHOD.

FIELD OF THE INVENTION 5

The present invention relates to the field of digital image processing, hereinafter "image" for the sake of brevity, taken by a shooting device, typically a camera or a camera, which is why we mean here without distinction "image" and " shooting ”, the image processing being based on matrix calculations.

A video includes a plurality of sequential images. Unless specified otherwise, the term "image" is used here to mean either a photographic image or an individual image of a video.

To acquire images, the shooting device comprises an optical device (for example a set of at least one lens), a digital sensor (for example a CMOS / CCD / photodiodes sensor). Typically, an electronic device is also provided comprising a computer (typically a processor and for example a DSP), which performs at least one image processing algorithm, said electronic device being able to be embedded

20 in the shooting device or in a separate computer.

The processing according to the present invention consists at least of detecting the background and / or the subject of an initial image, in order then to be able for example to separate the subject from the background on which it is located, whether in an individual image. or for all the images in a video sequence, in particular with a view to merging the subject with the background of another image, known as the background image, to add a virtual object between the subject and the background, or even by deleting objects standing in front of the subject (foreground).

The background image can be an image of a video sequence.

By definition, here the term "subject" means: any element located in the plane of development of the optical system having produced the image, "background" (or background by anglicism) all the other parts of the image, in front behind the subject, "scene" the set of real elements (subject (s), object (s), etc.) detected by the sensor of the shooting device and represented in the image.

In general, the subject of the scene has different optical properties from those of the background.

5For example, in the field of video surveillance, the subject corresponds to all the objects (people, cars, etc.) that move on the scene, and the background is a fixed environment (street, interior of the building, etc.).

In video, the interest of background detection is to be able to film a subject in any environment, and for example to then integrate the subject on another background to create another environment (other background and / or addition of information , additional animations, etc.) by video editing.

Traditionally, the subject extraction operation, in order to be automated, requires shooting in a studio equipped with a uniform color background, generally green or blue. It is then in real time or in post-production that the background is "deleted" by computer, by eliminating all the pixels of the chosen uniform color. This technique has the drawback of requiring very controlled shooting conditions (uniformity of the background, of the lighting, minimum distance of the subject from the background).

In addition, this technique can cause processing errors or imperfections in certain cases, for example it can eliminate objects having at least locally a tint of the same color as the uniform color chosen and require various operations including colorimetric processing.

In addition, the post-production operation is often at least partially manual, it can be long and tedious, and requires powerful calculation tools to be able to be performed in real time.

SUMMARY OF THE INVENTION

More specifically, the invention relates, according to a first of its objects, to a method of processing a set of at least one initial digital image comprising a subject and a background, the method comprising steps consisting in:

- if the initial image is in color, transform (100) at least one of the channels of the initial image into a grayscale image into grayscale, and save said grayscale image in a memory.

It is essentially characterized in that it further comprises steps consisting in: determining (110) the contrast of the image in gray levels, by calculating the level of local sharpness for each pixel of the image in levels of gray, in order to obtain a contrasting image matrix comprising a set of coefficients, each coefficient of the matrix of the contrasting image corresponding to a pixel (P) of the image in gray levels, compare (140) the value of each coefficient of the matrix of the contrasted image at a threshold value (Th), possibly respective, recorded in a memory, to obtain a binarized image, select (145) the coefficients of the matrix of the contrasted image whose value is greater than the threshold value, in order to obtain a binary mask comprising a set of contours, and to fill (150) the inside of the contours of the binary mask in a homogeneous manner.

It is also possible to provide a step (180) for filtering the contours of the binary mask.

Preferably, the step (180) of filtering the contours of the binary mask comprises a step (190) of guided filtering in which the initial image in gray levels serves as a guide image on the binary mask, to obtain a mask in grayscale.

In one embodiment, the set of at least one initial digital image is a video sequence, the method further comprising a step (200) of temporal filtering of a predefined number of individual sequential images of the video.

Preferably, the contrast determination step (110) comprises a step (130)

25 consisting in calculating the matrix (c ⁿ ) resulting from the convolution of each of the pixels of the matrix (j ⁿ ) of the gray level image by a Laplacian kernel, the method optionally further comprising and prior to step ( 130), a step (120) of low-pass filtering of the gray level image.

We can predict that the threshold value (Th) is predetermined.

The method can also comprise a step consisting in:

extract (160) the subject from the initial image by applying the binary mask or the grayscale mask to the initial image.

The method can also comprise a step of mixing by superimposing the initial image and the background image, performed as a function of the value of the coefficients of the mask matrix, as follows:

for the mask matrix coefficients whose value is 1, the pixel intensity values 5 of the subject of the initial image are retained without modification, and the pixel intensity values of the subject of the background image are replaced by said intensity values of the pixels of the subject of the initial image, for the coefficients of the mask matrix whose value is 0, the intensity values of the pixels of the subject of the initial image are replaced by the intensity values of the pixels of the subject of the background image, and for the coefficients of the mask matrix whose value is between 0 and 1, the intensity of the pixels of the image resulting from said mixing is composed of the values pixels of the subject of the initial image multiplied by the coefficients of the mask plus the values of the pixels of the background image multiplied by the inverse coefficients of the mask.

We can predict that the filling step (150) includes:

a filling operation which consists in filling the inside of the contours of the mask, and optionally also:

a dilation operation which consists in enlarging the lines of the binarized image according to a predefined magnification.

According to another of its objects, the invention relates to a computer program comprising program code instructions for executing the steps of the method according to the invention, when said program is executed on a computer.

According to another of its objects, the invention relates to a computer memory medium in which the computer program according to the invention is recorded.

According to another of its objects, the invention relates to an optical device, comprising an optical objective and a memory, in which the memory comprises the computer program according to the invention.

The invention proposed here is an automatic method of image processing, including video which therefore differs from at least partially manual methods of separating a subject.

35vs. background that exist in particular in photo editing software.

The invention proposed here allows a large number of applications, and is particularly relevant for several types of applications, for which the requirements in terms of image quality, processing time and trimming precision are high.

Advantageously, the present invention implements a passive solution, that is to say that it does not require active distance / depth measurement sensors, which distinguishes the present invention from all active systems, for example at based on infrared sensors, which can also be disturbed by other infrared radiation, eg, radiation from the sun.

Furthermore, according to the invention, only one objective can be used, which distinguishes the present invention from all stereovision systems. Likewise, a single image (single focus) can be used, which makes the present invention easy to implement.

The present invention also eliminates the need for homogeneous backgrounds (for example green or blue) used in television or cinema. The subject can therefore wear attributes / clothes colored in green.

Finally, the present invention can be implemented both indoors and outdoors.

Other characteristics and advantages of the present invention will appear more clearly on reading the following description given by way of illustrative and nonlimiting example and made with reference to the appended figures.

DESCRIPTION OF THE DRAWINGS FIG. 1A illustrates an initial image according to the invention, FIG. 1B illustrates a final image according to the invention, FIG. 2A illustrates a grayscale image according to the invention, in this case the face 30 of the FIG. 1A, FIG. 2B illustrates an image filtered according to the invention, in this case the image of FIG. 2A filtered by a low-pass filter, FIG. 2C illustrates a contrasted image according to the invention, in the present case the image of FIG. 2B transformed by a Laplace operator, FIG. 3A illustrates a binarized image, in this case the image of FIG. 2C, according to a first threshold value, FIG. 3B illustrates a binarized image, in in this case the image of FIG. 2C, according to a second threshold value, FIG. 3C illustrates a binarized image, in this case the image of FIG. 2C, according to a third threshold value, FIG. 3D illustrates a binarized image, in this case the image of figure 2 C, according to a fourth threshold value, FIG. 4A illustrates the result of an operation of dilating the contours of a mask of a binarized image, in this case that of FIG. 3D, FIG. 4B illustrates the result of an image mask processing operation, in this case that of FIG. 4A, FIG. 4C illustrates the result of a filling operation performed on the mask 10O of an image, in this case that of the FIG. 4B, FIG. 4D illustrates the result of a processing operation applied to the mask of an image, in this case that of FIG. 4C, FIG. 4E illustrates the result of a processing operation applied to the mask of an image, in this case that of FIG. 4D, FIG. 5A illustrates the result of the application of the mask of FIG. 4E to the initial image of FIG. 2A combined with another background than the background of the image Figure 5B illustrates the result of applying the mask of FIG. 6A to the initial image of FIG. 2A combined with the same background as the background of the image of FIG. 5A, FIG. 6A illustrates the result of a filtering algorithm guided on the mask of FIG. 204E using the guide image of FIG. 2A, FIG. 7A illustrates the point spreading function, or variation of the diameter of the blur ε, as a function of the distance between an optical device and a subject, said optical device having a predetermined focal length, FIG. 7B is a photograph illustrating FIG. 7A, in this case 3 subjects (apples) 25 arranged at 3 distinct distances from the same optical device, FIG. 8 illustrates an optical diagram representing the estimation of blur defocus ε as a function of the distance from the focal plane, and FIG. 9 illustrates an embodiment of the method according to the invention.

DETAILED DESCRIPTION

Principle

In the present description, a subject is assumed to be placed in the focusing plane of the lens of a shooting device. The shooting device can be of standard type, single channel (that is to say no stereo vision).

The image of the scene including the subject is called the "initial image", Figure IA. It is for example recorded in a memory, typically that of the shooting device or sent to a remote memory. The processing can also be carried out in streaming mode or streaming by anglicism, without going through the recording phase of 5video.

An initial image is for example an individual image from a video sequence or an image from a camera. Unless otherwise specified, only an individual image from a video sequence will be referred to.

As illustrated in Figure 7, we define by:

D, the distance to the lens, i.e. the distance between the camera and the subject,

DS, a so-called “safety” distance, configurable, in front and behind the subject, f, the focal length of the lens used by the shooting device.

The distance D and the focal length f are known. The depth of field is known or determined. Preferably, the safety distance DS is less than or equal to the depth

20 of field.

By "before" is meant in front of the subject, and in particular between the camera and the subject.

By “rear” or “beyond”, we mean behind the subject, therefore beyond the distance D and possibly beyond the safety distance DS.

The present invention makes it possible to delete all the elements of the initial image lying beyond the safety distance DS of the subject, as described below.

To this end, the principle of the invention is based on the segmentation of the initial image using the distinct optical properties of the subject and of the background in said initial image.

The subject being in the focusing plane located at a distance D from the optical sensor, it is characterized by a very high contrast. Conversely, the background has a very low contrast.

The present invention cleverly takes advantage of this characteristic and makes it possible to obtain an image processing capable of accurately segmenting the areas of high contrast and of low contrast in the initial image, and this all the more with an optical device capable to have a big difference in contrast between the subject and the background.

The present invention takes an initial image as input and transforms it into an image

5 final, an example of which is illustrated in FIG. 1B, processed, of resolution less than or equal to that of the initial image, in which the subject of the initial image is outlined with a mask, and in which the background of the image initial is deleted or replaced by another background.

Shades of grey

The initial image can be in color or grayscale.

If the initial image of a video is in color, then this means that each initial individual image n of the video is composed of three color channels: red R ⁿ , green G ⁿ and 15 blue B ⁿ where for the digital processing of l image, each channel is a matrix. Each matrix includes a set of coefficients.

Thus, each pixel P of coordinates (x _p , y _p ) of the initial image is represented by a vector (triplet) of value [R ⁿ (xp, yp), G ⁿ (xp, y _p ), B ⁿ ( x _p , y _p )]. For a color image, there are

20 therefore a correspondence between a triplet of coefficients and a pixel.

However, the background suppression principle proposed here is not based on color information, and it is therefore redundant to process these three channels (or pixel arrays).

To avoid this redundancy, it is preferable to treat only one channel. All the calculations described below can thus only be carried out for this single channel, which allows the calculation speed to be increased and the processing time to be reduced.

For a grayscale image, there is therefore a correspondence between a pixel of

30said image and a coefficient of its corresponding matrix. Each pixel P is characterized by its position (x, y) and its intensity value for each channel R (x, y), G (x, y) and B (x, y). We can therefore consider here without distinction a pixel or its corresponding coefficient.

If the initial image is in color, we therefore plan a step 100 consisting in transforming

35at least one of the channels in the initial grayscale image.

Preferably, this step is implemented by any suitable software known on the market. For example, we can use the standard formula which, for each pixel, replaces the intensity value triplet Γ of an initial individual image n of the video by a single value with a certain proportion of the intensity of each channel :

l ⁿ (xp, yp) = 0.2989 * R ⁿ (xp, y _p ) + 0.5870 * G ⁿ (xp, yp) + 0.1140 * B ⁿ (xp, y _p ) (1).

At the end of this step, the initial image of the video is in the form of a matrix Γ whose values are between 0 and 1, each value corresponding to a gray level.

An example of a grayscale image is illustrated in Figure 2A. By simplification is meant by "gray" matrix, the matrix corresponding to the grayscale image.

Contrast

At the grayscale image stage, it is not yet possible to know whether a pixel in the grayscale image matches the subject or the background.

Indeed, if we limit ourselves to measuring the light intensity alone of a pixel, this single measurement is not enough to determine whether this pixel corresponds to a point of an object belonging to the subject or to the background.

It is possible to get rid of this problem.

For this purpose, we assume that the more a pixel is contrasted (or sharp), the more the probability that it corresponds to the subject. Conversely, the less a pixel is contrasted (or sharp), the lower the probability that it corresponds to the subject. By "contrasting pixel" is meant that the area around a given pixel is contrasted, that is to say has for example a contrast gradient greater than a threshold value.

Indeed, at the optical level, the level of contrast of an object in an image depends on the position of this real object relative to the focus plane. The further the object is away from the plane of focus (in particular behind) of the subject, the more the contrast degrades, which results in an observation of blurring in the image of said object, as illustrated in FIG. 7B.

We therefore aim to replace a gray level with a level of contrast or sharpness for each pixel.

To this end, a step 110 of determining the contrast is provided which consists in calculating the local level of sharpness for each pixel P, that is to say in transforming the matrix Γ (χ _ρ , y _p ) of the image into gray levels in a matrix C ⁿ (x _p , y _p ) of the same size as the image in gray levels Γ (χ _Ρ , y _P ), also in gray levels, and whose values represent the level of sharpness local for each pixel P.

For the contrast determination step 110, provision is made to use a metric of 5 contrast.

The level of local sharpness for each pixel P is a value between 0 and 1, calculated according to a function of a brightness gradient in one or more directions predetermined from pixel P. The more the gradient is strong the more the image is locally clear.

By simplification, here therefore is meant by "contrasting pixel" the value of the function of a contrast gradient around said pixel in at least one predetermined direction.

We can provide a contrast metric of the point spread function type. There are also many contrast metric algorithms which make it possible to carry out this transformation, for example a review of these methods is presented at the address httpsT / wwvnesea ^^ For example a value 1 means a highly contrasted pixel and a value 0 means a pixel not contrasted, the intermediate values corresponding to a gradual contrast between these two extremes.

Another possibility to measure the contrast (or, conversely, the amount of blurring), is to make an estimate of the spreading function of the FEP point (or PSF for Point Spread Function in English) for an image. It is evaluated locally around each pixel and represented by a small matrix (e.g., 15x15, 21x21). The shape of the FEP represents 251a shape of the blur of the image, and the characteristics of this shape (like the width, the inclination or others) can be used to evaluate the contrast of each pixel of the image. However, a disadvantage of this contrast measurement method is that it is necessary to take into account the optical design and the sensor of the optical system, and carry out a set of at least one camera calibration in order to be able to obtain the different levels of contrast and bind

30these levels of contrast with the characteristics of VET.

Preferably the transformation used here is a Laplace operator, which is an omnidirectional function. This method is simple, therefore fast in computation time, and efficient, and the only one described in detail here.

However, Laplace’s operator is sensitive to acquisition noises that are inevitable in cameras. Also, to neglect the influence of noise, a low-pass filter 120 is advantageously applied beforehand, which makes it possible to smooth the image obtained.

273

For the sake of clarity, the image in levels

The low-pass filtering is carried out by applying a convolution filter with a kernel G, which consists in calculating the matrix / ⁿ resulting from the convolution of each of the pixels of the gray matrix Γ (χ _ρ , y _p ) by the nucleus G, let T ⁿ = conv (j ⁿ , G).

In this case, the nucleus G is a Gaussian nucleus.

| x ² + y ²

In this case, we use a discrete form of the Gaussian nucleus G (x, y) = -e ^2σ ~

2πσ ² which is in the form of a matrix of a predefined size, in this case square, for example a 3 * 3 matrix; a 5 * 5 matrix, etc.

For purely illustrative purposes, the nucleus G can be presented in the discrete form:

7 4 1 '

26 16 4

41 26 7

26 16 4

7 4 1 of gray thus transformed is called image

15 "filtered".

An example of an image filtered by a low-pass filter is illustrated in Figure 2B. You can see that the filtered image is slightly blurred. The level of this blur depends on the filter, in particular on the parameters of its kernel.

After the low-pass filtering, the transformation by a Laplace operator is also carried out by a convolution, but with another kernel, and preferably in discrete form.

d ² 1 d ² 1

In this case, we use the nucleus L (x, y) = + —— to calculate 130 the matrix c σχ oy

25resulting from the convolution of each of the pixels of the matrix j ⁿ of the gray level image by the Laplacian kernel L, that is: c ⁿ = conv (i ⁿ , L)

Two typical examples of discrete and square Laplacian nuclei, in this case 3 * 3, are as follows:

“0 -1 O “ -1 -1 -1 -1 4 -1 OR L = -1 8 -1 0 - 4 0 -1 -1 -1

The choice of a Laplacian nucleus or another allows to evaluate the contrast using the vertical, horizontal, or diagonal pixels, and to detect the details of the subject with different levels of contrast

The filtered image thus transformed is called the "contrast measurement" image and below "contrast image" for simplicity. An example of a contrasting image is shown in Figure 2C.

Thresholding

After the contrast determination step, a thresholding step 140 is provided, which consists in comparing the value of each pixel P of the matrix of the contrasted image with a threshold value Th recorded in a memory.

The threshold value Th can be predetermined. It can be identical for all the pixels or different for at least two pixels, that is to say that each pixel corresponds to a respective threshold value Th.

The threshold value Th is chosen so as to simulate the value of the sharpness limit.

The comparison with the threshold value makes it possible to obtain a binary result: either the result is positive, or the result is negative.

If the result is positive, that is to say that the value of the matrix of the contrasting image corresponding to a given pixel P is greater than the threshold value, then it is considered that said pixel P is sufficiently contrasted and is part from subject.

Conversely, if the result is negative, that is to say that the value of the matrix of the image

30 contrast corresponding to a given pixel P is less than the threshold value, then it is considered that said pixel P is insufficiently contrasted and is part of the background.

We can therefore determine the degree of pixel belonging P of the contrasting image to the subject or background.

There are several possible options for the thresholding step.

In a first variant, the simplest one, a uniform threshold value is applied to the entire contrasted image.

The thresholding step makes it possible to transform the matrix C ⁿ into a binary matrix B ⁿ such that:

B ⁿ = l if C ⁿ > Th, and B ⁿ = 0 if C ⁿ <Th.

For the sake of clarity, the contrasted image thus transformed is called the "binarized" image. Examples of binarized images are illustrated in Figures 3A, 3B and 3C.

FIG. 3A represents the contrasting image of FIG. 2C binarized according to a first threshold value Th1.

FIG. 3B represents the contrasting image of FIG. 2C binarized according to a second threshold value Th2.

FIG. 3C represents the contrasting image of FIG. 2C binarized according to a third threshold value Th3.

FIG. 4C represents the contrasting image of FIG. 2C binarized according to a fourth threshold value Th4.

In FIGS. 3A, 3B and 3C, the white lines correspond to B ⁿ = 1 for which it is considered that the pixels correspond to the subject, and the black lines correspond to B ⁿ = 0 for which it is considered that the pixels correspond to the background.

In this case, Thl <Th2 <Th3. The higher the Th threshold value, the lower the number of pixels considered to be sharp in the contrast image.

In a second, more sophisticated variant, an adaptive threshold value is applied, that is to say that the threshold value is not the same for all the pixels of the image. For example, an Otsu threshold can be implemented, as described in the publication by Nobuyuki Otsu, "Athreshold selection method from gray-level histograms", IEEE Trans. Sys., Man., Cyber., Vol.

359, 1979, p. 62-66.

From the thresholding step, it is possible to determine the contours of the binarized image, which makes it possible to define a mask, comprising contours.

Morphological treatment

Since the contrast measurement is based on a gradient, it is all the more reliable than

5the initial image shows textured areas, that is to say local areas with a strong gradient, where by "strong" is meant above a predetermined threshold value.

Indeed, it happens that the zones of the initial image having homogeneous or quasi-homogeneous intensities, that is to say a weak gradient, where by “weak” one understands lower than a predetermined threshold value, do not emerge after the thresholding step.

Now, we aim to obtain a full binary mask: everything that is outside the outline of the mask is considered to be the background, and everything that is located inside the outline of the mask is considered to be the subject, the outline having a high level of contrast.

Consequently, it may be useful to implement an additional morphological processing step, to fill 150 the zones situated inside the contours of the mask in a homogeneous manner.

The details of the mathematical morphology operations used are conventional and are well described in various bibliographic sources on image processing, for example in the book on mathematical morphology [Image analysis and mathematical morphology, by J. Serra. Academie Press, London, 1982],

Thus, in one embodiment, the step of mathematical morphology, applied to the binarized image, comprises at least one of the operations summarized below, described sequentially.

Dilation

We can foresee a dilation operation which consists in enlarging the features of the binarized image, according to a predefined magnification. This operation closes the contours obtained after the thresholding step.

Figure 4A illustrates an example of a dilation operation performed on the image

35binarized from Figure 3D.

It may be advantageous to implement other processing operations which aim to ensure that the contours of the subject mask always remain closed, as illustrated in FIG. 4B, which illustrates the result of a processing operation applied in FIG. 4A.

Once the contours of the subject mask are still closed, we can then provide a filling operation.

Filling

A filling operation is provided which consists in filling the inside of the contours of the mask, in this case with a binary value, and in this case with the binary value corresponding to the subject.

For example, Figure 4C illustrates the result of a filling operation applied to Figure 4B.

At this stage, the overall shape of the subject is clearly detected by the contours of the binary mask.

Another mask processing operation can be provided. For example, FIG. 4D 20 illustrates the result of such a processing operation applied to the mask of FIG. 4C.

FIG. 4E illustrates the result of another processing operation applied to the mask of FIG. 4D.

The configuration of the morphological processing depends on the resolution of the initial image and the quality of contours detected by Laplace filtering including thresholding. The configuration therefore advantageously depends on the quality of the optics of the optical device and its adjustment.

The sequence of operations is only one of the possible embodiments of morphological processing, but for the sake of brevity the only one described here.

In this case after the opening operation, a binary mask is obtained, illustrated in FIG. 4E, which represents a subject / background separation where in this case the binary values of the matrix 35 of the mask include two values: a binary value which corresponds to the background, in this case in black pixels, and the other binary value which corresponds to the subject, in this case in white pixels.

This mask can be enough for certain applications, because it allows to cut the subject from the background to a first level.

Contour filtering

Depending on the subject, typically depending on the degree of detail of the subject, and the quality sought, the application of the mask obtained after the morphological processing step to the initial image can lead to more or less satisfactory results.

For example, the application of the mask of FIG. 4E to the combination of the initial image of FIG. 2A and another background (in this case a mountain) results in the result illustrated in FIG. 5A, which does not appear therefrom. sufficiently realistic, the photomontage remains coarse, in particular at the level of the hair in the region of the ears.

To further improve the result, it is possible to provide a step of filtering the contours 180, which makes it possible to have more precise and precise contours of the subject.

For this step, we use the binary mask obtained at the end of the morphological processing (i.e. a binary matrix of the same size as that of the initial image) and the initial image

20grey.

In one embodiment, the contour filtering method consists in implementing a guided filtering algorithm 190, as described for example in the publication “Guided Image Filtering”, by Kaiming He, Jian Sun, and Xiaoou Tang, in ECCV 2010.

This filtering makes it possible to reinforce the contours of the filter thanks to the contours of the guide image which are detected by gradient calculations. The size of the filter makes it possible to control the size of the region on the guide image which is taken into account to assess the importance of the contours.

This postprocessing makes it possible to have a quality of contours superior to that of the simple binary mask (obtained at the end of the morphological processing) and the mask obtained is in gray levels and of the same size as the initial binary mask.

For example, Figure 6A illustrates the result of a filtering algorithm guided on the

35 mask of FIG. 4E thanks to the guide image of FIG. 2A. The initial gray image serves as a guide image and makes it possible to refine the mask of FIG. 4E.

The mask resulting from the guided filtering step is no longer a binary mask but a grayscale mask.

It is clear that the mask resulting from the guided filtering step illustrated in FIG. 6A is 5much more precise and fine than the mask resulting from the morphological processing step illustrated in FIG. 4E.

Then, by applying the mask resulting from the guided filtering step illustrated in FIG. 6A to the grayed-out initial image illustrated in FIG. 2A, the image illustrated in FIG. 5B is obtained. Which, in comparison with FIG. 5A, appears much more realistic.

Overall treatment

All the steps and operations previously described can be applied to each individual frame of a video sequence.

However, it can happen in this case that it generates jitter or jitter effects by anglicism, visible on the edges of the subject, because the mask is not exactly the same from one individual image to another.

Time filtering

To limit this effect, one can provide, in addition to the preceding steps, a time filtering step 200, which makes it possible to obtain a fluidity effect when watching the video.

25treated.

For this purpose, provision is made to select a "window", that is to say a predefined number of sequential individual images of the video. Each individual image therefore corresponds to a respective mask, obtained by morphological treatment or by guided filtering.

We can predict that the temporal filtering includes at least one of the following two levels.

The first level is a median filtering applied by pixel on the selected window, which avoids mask jumps and outliers from one individual image to another.

The second level is an average filtering applied per pixel on the selected window, to add fluidity to the processing results.

For the first or second level of filtering, we take the position (x, y) of a given pixel P 5 and the mask values of the same position (x, y) on a predetermined set of neighboring individual images.

It will be noted that the temporal filtering is applied to the masks of the individual images of the video but not to the individual images themselves.

Thanks to this feature, the intensity values of each individual image, like the color combinations of each individual image, are not changed.

Format

We plan to select or extract 160 the subject from the initial image (grayed out or not) using the mask obtained thanks to morphological processing or guided filtering, and then mix said extracted subject with another image, called "background image".

The background image can be arbitrary. It may have a different size from that of the original image and a different resolution from that of the initial image. We can plan a scaling of the background image.

The value of a coefficient, between 0 and 1, in the mask matrix indicates the

25 intensity level of the subject to reappear on the final image.

Thus the mixing can be carried out as a function of the value of the coefficients of the mask matrix, as follows:

for the coefficients of the mask matrix whose value is 1, the intensity values of the subject of the initial image are taken without modification, which amounts to replacing 170 the background of the initial image by corresponding pixels of l background image, for the coefficients of the mask matrix whose value is 0, the intensity values of the background of the background image are taken without modification, and

- for the coefficients of the mask matrix whose value is between 0 and 1, the intensity of the pixels considered is mixed with the corresponding coefficients of the mask. This means that the intensity of the final image is made up of the values of the initial image (those of the subject) multiplied by the mask coefficients plus the values of the background image multiplied by the inverse mask coefficients.

By reverse mask, it is meant that the matrix of said reverse mask comprises complementary coefficients of the coefficients of the matrix of the mask obtained thanks to the morphological treatment or to the guided filtering. By complementary coefficients, we mean that the sum of a coefficient of the mask matrix obtained thanks to morphological processing or guided filtering, and the corresponding coefficient of the inverse mask matrix is equal to l.

Preferably the mixture of the extracted subject with the background of the background image is carried out by transparency simulation or alpha blending by anglicism.

The present invention can be implemented in post-processing, for example for prerecorded videos. For example, the post-processing program is a Matlab script (registered trademark) and the video is in HD format (1920x1088).

The present invention can also be implemented in real time, for example for streaming or streaming by anglicism.

We can provide an additional post-analysis different from the morphological treatment already described. This analysis imposes certain a priori, or selection criteria, such as for example the size of the subject, its position on the initial image, its shape, the maximum quantity of subjects to be extracted, etc. which allows you to filter the initial image by leaving only the binary mask on the

25objects matching the selection criteria and deleting all other objects.

Safety distance

The contrast determination step makes it possible to calculate a matrix whose values 30 correspond, for each pixel of the contrasted image, to a degree of sharpness or of contrast thereof.

We can estimate the defocus blur, or blur level, by calculating the spreading function of the FEP point, which represents, for a real object in a given scene, the variation of the size ε in pixels of said object in an image of said scene as a function of the distance between said object and the objective which made it possible to produce the image.

In other words, the further an object is located from the focus plane, the higher its FEP, so the more blurred the representation of this object in the image.

There is however a limit distance to be exceeded to consider that an object behind or 5 in front of the subject (placed in the focus plane) is blurred. There is therefore a zone of sharpness ZN, or depth of field, in which the focus plane is included.

We can therefore define a depth, called DS safety distance, preferably less than or equal to the depth of field, calculated from the focusing plane towards the rear of the subject, and such that all objects positioned outside this safety distance

DS are considered sufficiently blurred and will be removed from the final image.

The greater the DS safety distance, the greater the risk that parasitic objects, that is to say, having no interest in the subject, for example, will be included in the final image.

15 is great.

To reduce the DS safety distance, which amounts to having a steep FEP slope, the characteristics of the optical device can be modified (aperture, optical quality, better focusing, pixel size, etc.). Typically, the shorter the distance from the subject (with reference to the shooting device) and the safety distance, the shorter the focal length of the lens of the shooting device and vice versa, which leads to short distances, to specific and complex optical systems.

Optical optimization

An optical device, typically a camera or camera (including a communicating object, and in particular any smart phone, tablet, etc.), comprises an optical lens and a sensor. Generally, the optical device also includes a memory for storing the shots, and a computer.

It is possible to implement the present invention either locally on the optical device, or remotely on a remote machine to which the initial image or images are sent, said machine comprising a memory for storing the initial images, and a computer.

The sensor of the optical device is preferably arranged in the focal plane. The relative position of the subject and the optical device is such that the subject is positioned in the depth of field ZN of the optical objective. For example, the optical lens is focused on the subject.

We can define by:

- ZN_AV the zone of sharpness before, that is to say the space of the zone of sharpness in front of the subject (supposed punctual and placed in the plane of focusing), and ZN_AR the zone of rear sharpness, it that is to say the space of the zone of sharpness behind the subject (assumed to be punctual and placed in the focus plane).

Preferably, a safety distance DS is chosen such that DS <ZN_AR.

For a subject of thickness EP (or depth along the optical axis of the optical objective), we preferably choose, we preferably choose a safety distance DS such as ZN_AR> DS> EP. Indeed, any element of the scene included in the safety distance is considered

15as belonging to the subject.

Preferably a safety distance DS_AV is provided in front of the subject and a safety distance DS_AR behind the subject, the safety distance DS_AV in front of the subject being able to be of value different from the safety distance DS_AR behind the subject.

Preferably, provision is made to configure the optical device so that the blurring gradient along the optical axis is greater than a predetermined threshold value.

The present invention is not limited to the embodiments previously described.

Claims (10)

1. Method for processing a set of at least one initial digital image comprising a subject and a background, the method comprising steps consisting in: if the initial image is in color, transform (100) at least one of the channels of the image 5 initial in grayscale, and save said grayscale image in a memory, characterized in that it further comprises, steps consisting in: determining (110) the contrast of the grayscale image, calculating the level of local sharpness for each pixel of the grayscale image, in order to obtain a matrix 10 of a contrasting image comprising a set of coefficients, each coefficient of the matrix of the contrasting image corresponding to a pixel (P) of the grayscale image, compare (140) the value of each coefficient of the matrix of the image contrasted with a threshold value (Th), possibly respective, recorded in a memory, to obtain a binarized image, 15 - select (145) the coefficients of the matrix of the contrasting image whose value is greater than the threshold value, in order to obtain a binary mask comprising a set of contours, and fill (150) the interior of the contours of the homogeneous binary mask. 20 2. Method according to claim 1, further comprising a step (180) of filtering the contours of the binary mask. 3. Method according to claim 2, in which the step (180) of filtering the contours of the binary mask comprises a step (190) of guided filtering in which the initial image in 25 grayscale is used as a guide image on the binary mask, to obtain a greyscale mask. 4. Method according to any one of the preceding claims, in which the set of at least one initial digital image is a video sequence, the method 30 further comprising a step (200) of temporal filtering of a predefined number of sequential individual images of the video. 5. Method according to any one of the preceding claims, in which the step (110) of determining the contrast comprises a step (130) consisting in calculating the matrix ( 35c) resulting from the convolution of each of the pixels of the matrix (j ⁿ ) of the gray level image by a Laplacian kernel, the method optionally further comprising, before step (130), a step (120 ) low-pass grayscale image filtering. 6. Method according to any one of the preceding claims, in which the threshold value (Th) is predetermined, the method further comprising a step consisting in: extract (160) the subject from the initial image by applying the binary mask or the mask 5 in gray levels on the initial image, the method optionally further comprising a step of mixing by superposition of the initial image and the background image, carried out as a function of the value of the coefficients of the mask matrix, as following : for the mask matrix coefficients whose value is 1, the pixel intensity values 10 of the subject of the initial image are retained without modification, and the pixel intensity values of the subject of the background image are replaced by said intensity values of the pixels of the subject of the initial image, for the coefficients of the mask matrix whose value is 0, the intensity values of the pixels of the subject of the initial image are replaced by the values 15 of intensity of the pixels of the subject of the background image, and for the coefficients of the mask matrix whose value is between 0 and 1, the intensity of the pixels of the image resulting from said mixing is composed of the values pixels of the subject of the initial image multiplied by the mask coefficients plus the values of the pixels of the background image multiplied by the 20 inverse coefficients of the mask. 7. Method according to any one of the preceding claims, in which the filling step (150) comprises: a filling operation which consists in filling the inside of the contours of the mask, 25and optionally also: a dilation operation which consists in enlarging the lines of the binarized image according to a predefined magnification. 8. Computer program including program code instructions for 301 execution of the process steps according to any one of the preceding claims, when said program is executed on a computer. 9. A computer memory medium in which the computer program according to claim 8 is stored. 10. An optical device, comprising an optical objective and a memory, in which the memory comprises the computer program according to claim 8. 1/4 Sharpness limit DS FIGURE 7A