FR2927447A1 - PROCESS FOR RESTORING A BLURRED IMAGE ACQUIRED BY MEANS OF A CAMERA EQUIPPING A COMMUNICATION TERMINAL. - Google Patents

Abstract

The method according to the invention makes it possible to restore a blurred image acquired by means of a camera fitted to a communication terminal. It consists in acquiring an image by means of the camera, extracting the luminance information from the image and creating a new image in grayscale, the coarse approximation I <0> of an image restored by means of an accentuation filtering (FAC) of the observed image Iobs, the search for a degradation operator K <n> such that it is the minimum of a convex functional J (I < n>, K) in K, the search for a restored image I <n + 1> such that it is the minimum of a convex functional J (I, K <n>) in I, the possible processing of l 'image I <n + 1> by an edge enhancement filter (FAC) and repeating the search and processing steps until | I <n + 1> - I <n> | becomes small enough (convergence).

Description

The present invention relates to a method for restoring a blurred image acquired by means of a camera fitted to a communication terminal. It applies more particularly to the restoration of a blurred image acquired by means of a camera fitted to a device provided with the means necessary for: transmitting an image and receiving information in return, carrying out the processing autonomously. Its object in particular is to improve the interpretation of the information contained in said image, whether this interpretation is carried out by an individual or automatically by a machine.

In general, it is known that the cameras equipping a device of the type mentioned above are often manufactured with fixed focal lengths (that is to say without autofocus or macro mode) and / or a fixed aperture, only the speed. 25 shutter is ordered. In addition, the inexpensive sensors used are often very limited in low light conditions. When the object is close to the camera, the resulting photo is often blurry, noisy and difficult to interpret. Indeed, although it is possible to equip the cameras with an autofocus device "or" macro "mode, this is seldom the case because of the additional cost or the additional bulk which results. 2 The object of the invention is more particularly to improve the sharpness of categories of images where it is fundamental to be able to interpret the information contained in the imaged object, (it being understood that it makes it possible to process any type of image). 'images).

Therefore, it is necessary to restore the blurred image as faithfully as possible to the initial data in order to guarantee its good interpretability.

The human eye is sensitive to the outlines of objects in images. A blurred image sees its contours attenuated, making the task of interpreting the objects or symbols contained very difficult, if not impossible. The same is true for a machine, where it is also impossible to interpret symbols contained in an image that is too fuzzy. Indeed, these devices usually use local high frequency characteristics to classify or recognize symbols.

This is the case in particular, for example, in character recognition applications, or even in decoding of ld / 2d bar codes. The object of the invention is more precisely the reconstruction of a “digital” image v from the sole knowledge of the observed image u. In the following, we will consider an image u E] 18N1 'N2 where NI and N2 are two natural numbers. It will be assumed that this digital image is obtained from a digital image v, using a measuring device.

We also suppose that y EL (R2), (L (R2) being the set of functions defined in R2 of (summable) squares and that the (possible) degradation introduced by the measuring apparatus follows the model: 20 -3- u. (X, y) = (h * v) (x, y) + h (x, y), (equation 1) where * denotes the convolution product, (x, y) E {0, .... Nl ù 1} x {0, ..., N2 ù 1}, h EL (R2) is a spatially invariant convolution kernel and b is a random variable defining the noise.

The problem that the invention proposes to solve is to reconstruct the image v from the sole knowledge of the observed image u. This problem falls into the category of inverse problems (see [Guy. Demoment, Inverse problems: theory and applications, Spring-Rolls, Lorient, 1997; Albert Tarantola and Bernard Valette, Inverse problems = quest for information, Journal of Geophysics 50 (1982 ), 159ù170; PC Sabatier, Introduction to inverse problems, Applied Inverse applied Problems (PC Sabatier, ed.), Springer-Verlag, Berlin, Germany, 1978, pp. 2ù26]).

It turns out that these "reverse" problems relating to the reconstruction of an image from knowledge of the observed image are usually badly posed. In addition, the convolution kernel h is also unknown (blind deconvolution) (see [Deepa Kundur and Dimitrios Hatzinakos, Blind image deconvolution, IEEE Signal processing magazine (1996), 43ù64; Schultz, Multiframe blind deconvolution of astronomical images, Journal of the Optical Society of America 10 (1993), no. 5; David Donoho, On minimum entropy deconvolution, Applied Time Series Analysis ii (New York) (David D. Findley, ed.), Academic Press, March 1981, pp. 565- 608]).

A quick analysis of this problem shows that: in the presence of noise, an exact deconvolution is impossible; we do not find the original image; only an approximation is possible, and this even if the kernel h is perfectly known, most often, this solution is not unique, it is therefore necessary to impose constraints to reduce the field of solutions and / or to add assumptions and an a priori on the image to be restored, where we must estimate both the blur kernel h and the image v; an exhaustive search would give an approximate solution to the problem of equation 1 with a computation cost necessarily too high.

The invention proposes an approximate solution for determining the torque v and h quickly and with a better compromise between cost, robustness and portability. To this end, it proposes a method making it possible in particular to restore a blurred image, acquired by means of a camera fitted to a communication terminal, with a view to improving the human or automatic interpretation of the imaged symbols, this method comprising the following steps : the acquisition of at least one image by means of a camera fitted to the communication terminal; the possible transmission of the image to a remote calculation means, it being understood that the communication terminal is provided with the means necessary for the transmission of an image by any means and media whatsoever, and also provided with means of processing information in return ; extracting the luminance information from the input image and creating a new image in gray levels Ip; the possible reduction in the size of the lobs image, carried out to reduce the calculation times; the rough approximation I ~ of an image restored by means of an edge enhancement filtering technique (FAC) of the observed image lobs; the search for a degradation operator Kn such that it is the minimum of a convex functional J (In, K) in K; the search for a restored image In + 1 such that it is the minimum of a convex functional J (I, Kn) in I; the possible processing of the image In + 1 by a contour accentuation filter (FAC); 2025 the repetition of the steps of searching for a degradation operator, of searching for a restored image In + 1 and of possible processing of the restored image until convergence, that is to say when IIn + 1 ù In1 becomes small enough.

The previously defined method could use a spatially invariant operator

and a convex criterion J (I, K) of the form: J (I, K) = Ep ((I * K) (xy) -Io (x, y)) + (Vf (x, y)) + ~ Ee (VK (x, y)). x, y x.y x, y (equation 15) where pO,) and i /,> O are derivable and strictly convex functions. I and K realize a global minimum of the criterion J. The function pO can be either a quadratic function such that p (u) = n2, or a function such that p (u) = n, or all other functions approaching the behavior of w, and nevertheless derivable in 0. The function 00 can be either a quadratic function such that 0 (u) = 112, or any other functions approaching the behavior of lu and nevertheless derivable in O. Moreover, the method according to the invention could include a joint estimate of two images such that I = h + Iutz the estimate in which the equation I2 = IZ * K is true at least on the support of K, the equation JO then becoming:

J (h, ht ~ le, K) p (Iz (x, y) + ('useful * K) (x, y) ù Io (x, y)) - ha E w (ohutile (x, y)) + p 'sL' (VK (x, y)) x, yx, y æ, y (equation 16) Advantageously, the fuzziness degradation operator K can be parameterized and depend only on a few parameters. Thus, in the case of motion blur, the number of parameters will be reduced to 2 and in the case of defocus blur, the number of parameters will be reduced to 1. In the case where K is set, it will not it will be more necessary to take account of the regularization imposed on K and ut = O.

Moreover, the search for the parameters of the fuzzy operator K may involve a dichotomy technique such as, for example, but not necessarily, the Brent method. An embodiment of this method will be described below, by way of nonlimiting example. In this example, this method iteratively estimates an image pair v and a blurring operator h comprising the following steps: 1. The filtering of the observed image u by a contour accentuation filter 15 (FAC) independent of l 'blur operator h makes it possible to obtain an image v °. 2. We compute an estimate of the fine fuzzy operator from vn and u. 3. We estimate the restored image yn + 1 at iteration n, between hn and u. 4. To accelerate the convergence of the process, the image vn + 1 is processed by the contour enhancement filter 20 (FAC). 5. We repeat the steps from the calculation of the estimate of the fuzzy operator, until convergence - that is, until I vn + l - vn l is sufficiently small. . It turns out, however, that in practice, a joint estimation such as that described above strongly depends on the initialization. Indeed, it is obvious that there are many image / operator pairs which are solutions of equation 1. The use of a contour accentuation filter (FAC) has the double advantage of leading to a rough estimate. of the pair of solutions (v, h) sufficiently close to the final solution to guarantee convergence towards an acceptable solution and also to accelerate the overall process. Although the method can have recourse to any contour accentuation filter (for example of the "unsharp masking" type), it is obvious that the performances are directly dependent on the capacity of this filter to approach a satisfactory coarse solution. .

The invention provides the use of shock filters as a type of edge enhancement (FAC) filter. The use of concepts and techniques developed for the computation of solutions of Equation 1 by solving nonlinear partial differential equations (PDEs) of hyperbolic type for image processing problems has been proposed by L. Rudin (see [L. Rudin, Shock filters, Rockwell International Science Center Annual DARPA TR 22 (1984), no. 2, 387-405]), who was the first to introduce the notion of shock filter in this field. These filters are operators whose application allows the development in the restored signal of phenomena analogous to the shock waves known in fluid mechanics (see [to3em, Images, numerical analysis of singularities and shock filters., Ph.D. thesis, Computer Science Department, CalTech, Pasadena, CA, 1987; S. Osher and LI Rudin, Feature-oriented image enhancement using shock filters, SIAM J. Numer. Anal. 27 (1990), no. 4, 919-940] for more precision. ).

The approach developed by Rudin for the resolution of the problem of image restoration uses an EDF 'scheme whose solution v (t,: r ,, y) has as initial condition the restored image v (0 . x, y).

The 2D model proposed by Rudin is: av at v (0, x, y) av an ùMvT (G (v)), for t> 0; (x, y) E S2, v (x. y); (x. y) E f2, t = 0, 0 on the edge aQ of S2. (equation 2)

where F (s) is a function of the variable s such that F (0) = 0 and sign (s) F (s)> 0 where sign () is the sign function, G (v) denotes a nonlinear elliptic operator of the second order, allowing the extraction of contours. The performance of this type of model lies in the judicious choice of the function F and the operator (v).

Rudin's choice and the most obvious for F is the sign function defined by: Isis> 0, F (s) = algorithm 0sis <0.

Now, it is a question of choosing the elliptic operator G, knowing that one wishes to develop the shocks in the direction orthogonal to the gradient of the image, Rudin proposed: (v) =: = pvTHZ, Vv,

where uT is the transpose of u and Hz1 is the Hessian matrix associated with v (matrix composed of the second derivatives of v), y being the direction of the gradient.

To simplify, the classical numerical diagram for this kind of filter in the 1D case is: vk + 1 vk Ot Dvli 'F (D2I), for At> 0; i E {1, .., N1}, (equation 3)

25 where D and D2 are discrete derivation operators of order 1 and 2 respectively, satisfying: 15 7n (A + vi,. _V2 A 0_v. 2 =) .. D2Z ,. - + AX (0x,) 2, where the function • rri is given by: m (x, y) = F (x) 2 F (y) min (lxH'y and L = (ui i ù ni). , the CFL condition (for the 1D case) is:

Lx At <--- 2 The main properties of shock filters are: 10 the shocks develop at the inflection points (the zeros of the derivatives of order 2), the local extrema are time invariant, no new local extremum n 'is created, the scheme keeps the total variation, 15 the (weak) solution at equilibrium is piecewise constant (with discontinuities at the inflection points), the model approaches deconvolution without requiring knowledge of the convolution kernel h . 20 However, this model is not very resistant to noise. Theoretically, in the DC domain, any additive white noise can add an infinite number of inflection points, disrupting the process completely. In order to resolve the difficulty related to noise, the method according to the invention uses

25 a family of shock filters where the activation of these filters is weighted according to the region of the image. These weights are of greater amplitude near a point of inflection and less in a more regular region. For this, the first part of equation 2 is replaced by: 5 t = 2 arctan (atv, ~, ~ Ver-10- + 'v,; - e), t> 0, in Q, (equation 4) where v denotes the derivatives of order 2 in the direction parallel to the gradient. This anisotropic diffusion term makes it possible to reduce the noise during the process, ry> 0 is the penalty parameter associated with this term. The arctan function. weights the amplitude according to the region, finally a is a real parameter controlling the acuity of the slope in the neighborhood of zero. The introduction of time in the second term of equality makes it possible to reduce the effect of the shock at the start of the process. Thus, it is the diffusion term (reducing noise) which is predominant, which will considerably eliminate the number of false inflection points. In order to solve equation 1 in h and v, we use a functional J defined by: J (v, h) = / p ((v * h) (x, y) ù u (x, y)) dx dy + f 45 (Vv (x, y)) dx dy + F ~ J 2, ", (Vh (x, y)) dr dy. (equation 5)

where p0, 00, z / 'O are convex, positive functions satisfying p (0) = 0, ç (0) = 0, /,' (0) = 0. S2 is the domain of the image.

This equation consists of three terms: the first expresses the error made on the model. This term is also called the data fit term. For a quadratic pO function, this term is the least squares criterion. The second and the third terms express for their part the a priori which one supposes on the image v and the operator h. These two terms are means of controlling the noise in the image v and stabilize the problem.

By construction, J is convex in v for fixed h and also convex in h for fixed v. In the following, J is defined as being a criterion will be convex, and the solution of equation 1 satisfies: 2927447 -11- (v, h) = In fi, kJ (I, k) = arg min (arg min J (I, k). K I By abuse of notation and for simplicity, the pair (v, h), solution of equation 1, is called the minimum of criterion J. Finding this solution consists in looking for this minimum. is then necessary to have recourse to gradient descent techniques, i.e. to iteratively and alternately search for the minimum of J using its partial derivatives with respect to h and v 0 av to J = h (ùx, --y) * P ((v * h.) (x, y) ù n (x, 'y)) + A c (P (Vv (x, y))) (equation 6) 10 c ~ J = v (x, ùy) * p ((v * h) (x, y) -u (x, y)) + It - (/ '(Vh (x, y))). Oh, ah. (equation 7) A simple but expensive procedure to reach the minimum of J in h and in v 15 can be written in the form:, vi + 1 ù a 0J (v Dv ûh (vi + l, hi). 20 Where, a and / 3 are positive and sufficiently small constants. It could also be advantageous to resort to other optimization techniques always in the spirit of improving the rate of convergence. For this purpose, we prefer methods of the conjugate gradient type (see [R. Marucci, RM Mercereau, and RW Shafer, Constrained iterative deconvolution using a conjugate-gradient algorithm, Proceedings of the International Conference on Acoustic, Speech and Signal Processing (Paris ), 1982, pp. 1845û1848; R. Fletcher and CM Reeves, (equation 8) (equation 9) - 12 - Function minimization by conjugate gradients, Comp. J. 7 (1964), 149û157]) or better still of the quasi type -Newton such as the LBFGS ([J. Nocedal and SJ Wright, Numerical Optimization, Springer-Verlag, New York, 1999]).

As mentioned previously, the use of a quadratic pO function amounts to expressing a least squares criterion on the data fit term. However, it is also advantageous to use a more robust convex function for this term, that is to say, a function which penalizes model errors less heavily than Tikhonov-type quadratic penalties (see [Jerry Eriksson, Optimization and regularization of nonlinear least squares problems, Ph.D. thesis, Umeâ University, Sweden, June 1996; to3em, Solutions of ill-posed problems, Winston, Washington DC, 1977; A. Tikhonov and V. Arsenin, Methods of solving ill-posed problems, MIR Publishing, Moscow, 1976]).

A simple robust norm amounts to using a function p (x) = lx 'and its derivative (in the sense of distributions) p' (x) = sign (x). However, this standard poses problems of convergence around 0 because of its non-derivability in the usual sense at this point. Other functions can then be used imitating a quadratic behavior below a threshold T, and a linear behavior above. Without this constituting a restriction of the method, we can have recourse to a Lorentzian norm by setting: _ 2x p (x) = log 1 + 1 2 (x T) 2 'P (x) 2T2 + x2. In the same way, the two penalization terms which relate to h and v are penalizations of the Tikhonov type when the functions 00 and eO are chosen quadratic. However, such a quadratic penalization on the image 'v is too strong a priori, in the sense that it imposes a too regular solution v. In order to preserve the contours of the image to be restored, and without this constituting a restriction of the method, one can resort to techniques of penalization by total variation.

5 The penalization by total variation amounts to posing: VT ('= fp (Vv) dx dy f 1Vv dx dy. (Equation 10) However this equation does not take into account the singular points (that is to say the points canceling out the spatial gradient of v). The majority of authors use a regularized version, for example [R. Acar and C. Vogel, Analysis of bounded variation penalty method for ill posed problems, Inverse Problems 10 (1994), no. 6, 1217û1229; to3em, Fast, robust total variation-based reconstruction of noisy, blurred images, IEEE Transactions on Image Processing IP-7 (1998), no. 6, 813û823; Antonin Chambolle and Pierre-Louis Lions, Image recovery via total 15 variation minimization and related problems, Numerische Mathematik 76 (1997), 167û188; to3em, A computational algorithm for minimizing total variation in image restoration, IEEE Transactions on Image Processing 5 (1996), 987û995; Yuying Li and Fadil Santosa, An affine scaling algorithm for minimizing total variation in image enh ancement, Tech. report, IEEE Image processing; R. V. 20 Vogel and M. E. Oman, Iterative methods, for total variation denoising, SIAM Journal of Scientific Computing 17 (1996), no. 1, 227-238; L. Rudin, Stanley Osher, and C. Fatemi, Nonlinear total variation based noise removal algorithm, Physica D 60 (1992), 259? 268] to reduce degeneration in smooth regions of the image where the spatial gradient is zero. Thus, we can define a regularized version of the total variation, such as: p (Vv) dx dy = 1: \ / vx + vÿ + s d.x, dy. (equation 11) where s is a sufficiently small positive real. I. 2927447 - 14 - The derivative of equation 11 in v is then written: OVT (v) vxx. (S + vy2) ù 2.vx.vy.vxy + vyy. (S + vx) (s + v2r + vy) (equation 12) It should be noted that a reasoning analogous to that on v can be held concerning the penalization of the operator h, but which will not be detailed here. In many applications, the lighting conditions for the shot are not under control. Also, the image obtained by shooting may reveal variable illumination. This variable illumination does not represent a limitation of the

10 restoration method according to the invention. However, the management of variable illumination poses great difficulties for image analysis algorithms. Also, the invention proposes a different writing of equation 5 to deal with this particular case by imposing the following decomposition: v = z + w (equation 13) It is assumed that z contains the useful part of the signal to be restored while w contains weak variations of the signal such that Îv (x ,, y) = (h * w) (x, y) is true at least in the neighborhood of (x. y) and on the support of h. By combining equations 5 and 13, we obtain the new criterion J3 such that: Js (z. W, h) - / P ((z * h) (a ', y) + u ^ (a', y) ù 2c (a., Y)) d: r dy + À J ç> (Vz (a ,, y da. Dy + yf (Vh (.r, y) dr dy. Sl S2 (equation 14) 25 This new writing brings the advantage of carrying a more adequate a priori on the useful part of the signal z. It is then possible to choose: û to fix w then to estimate alternately and iteratively z and h as before, or then - to estimate jointly z, w and h. 15 20 2927447 - 15 - Whatever the case, the penalty or optimization methods based on the criterion of equation 5 are also applicable for equation 14. In fact, it will not be no distinction is made between these two equations later.On the assumption that the nature of the blur is known in advance (example: atmospheric blur, motion blur, defocus blur, etc.), h can be expressed parametrically and with few degrees of freedom.

10 Once h has been parameterized, it then appears that the minimization of J by h amounts to that of J by the parameters 9 of h. Therefore, simpler and faster optimization techniques can be considered.

For example, in the case of a defocus blur, h is modeled by a disk 15 of radius re-centered at the origin, so J (v, h, (r)) becomes J (v, r). In this case, finding the minimum of J, when v is fixed, amounts to finding the minimum of a convex function with one variable, and can be achieved by dichotomy without using the derivative of J by r. Golden search-type methods are therefore indicated and in particular the Brent method (see [RP Brent, Algorithms for minimization without 20 derivatives, ch. 7, pp. 308-313, Prentice-Hall. Englewood Cliffs, NJ, 1973]) ). 5

Claims (19)

Claims 1. A method making it possible in particular to restore a blurred image, acquired by means of a camera fitted to a communication terminal, in order to improve the human or automatic interpretation of the imaged symbols, this communication terminal being provided with means for transmitting an image. image and processing of feedback information, characterized in that it comprises the following steps: the acquisition of at least one image by means of the camera fitted to the communication terminal 10; extracting the luminance information from the input image and creating a new grayscale image 10; coarse approximation I ° of an image restored by means of an edge enhancement filtering technique (FAC) of the observed image lobs; The search for a degradation operator Kn such that it is the minimum of a convex functional J (In, K) in K; the search for a restored image In + 1 such that it is the minimum of a convex functional J (I, Kn) in I; repeating the steps of searching for a degradation operator, of searching for a restored image In + 1 until convergence, that is to say when 1P-F1 ù In becomes sufficiently small. 2. Method according to claim 1, characterized in that following the step of acquiring the aforesaid image, it comprises the transmission of this image by the communication terminal to a remote computing means. - 3. Method according to claim 1, characterized in that it comprises a reduction in the size of the lobs image to reduce the calculation times. 4. Method according to claim 1, characterized in that it comprises a processing step ln + l by a contour accentuation filter (FAC). 5. Method according to claim 1, characterized in that it uses a spatially invariant operator and a convex criterion J (I, K) of the form: J (I, K) _ E p 1 * K) (x, Y ) ù Io (x, Y)) + a (vl (x, Y)) + -, vK (x, Y)) x, y x. x, y where p (), 0 () and 1,) () are derivable and strictly convex functions. I and K 15 achieve an overall minimum of criterion J. 6. Method according to claim 5, characterized in that the function p () is a quadratic function such that p (n) _? 1,2. 7. Method according to claim 5, characterized in that the function pO is a function such that p (u) = I41, I. 8. Method according to claim 5, characterized in that the function p () consists of a function other than a quadratic function such that p (u) = u2 and that a function such that p (u) _ him. , and which approaches the behavior of it and nevertheless derivable in O. 20 2927447 -18- 9. Method according to claim 5, characterized in that the function G5 () is either a quadratic function such that çb (u) =? 1,2. 5 10. Method according to claim 5, characterized in that the function 0 () is a function other than a quadratic function such that 0 (u) = u2 and approximating the behavior of lu, i and nevertheless derivable at 0. 10 11. Method according to claim 1, characterized in that it comprises a joint estimation of two images such that I = Ii + _rutile estimation in which the equation Ii = Ii * K is true at least on the support of K, l 'equation J () then becoming:, 1 (-Tt, ruttiie, K) = p (Mx- y) + (II.we * K) (x, y) -10 (x, y)) + a 4 ( vlu.t, le (x, y)) + p (VK (x, y)) x, yx, y "y 12. Method according to claim 1, characterized in that the degradation operator K is configurable and depends on a limited number of parameters. 20 13. The method of claim 12, characterized in that the degradation operator K depends on two parameters in the case of motion blur. 14. Method according to claim 12, characterized in that the degradation operator K depends on a parameter in the case of a defocusing blur. 2927447 -19- 15. The method of claim 12, characterized in that when the blur degradation operator K is configured, it is no longer necessary to take account of the regularization imposed on said operator and the coefficient p is equal to zero. 16. Method according to one of the preceding claims, characterized in that it comprises the search for the parameters of the degradation operator K by a dichotomy technique. 10 17. The method of claim 11, characterized in that the search for the above parameters is carried out by the Brent method. 18. The method of claim 1, characterized in that the contour accentuation filter (FAC) belongs to the family of Shock filters. 19. Method according to claim 1, characterized in that the search for the minima of the functional J (I, K) is carried out by means of a gradient descent technique taking into account the derivative of the equation J (I , K) with respect to both of its variables. 5