EP3170303B1 - Method for processing high-frequency movements in an optronic system, optronic system, computer program product and storage means - Google Patents

Description

The present invention relates to a method and a device for processing high frequency movements in an optronic system and an optronic system implementing said device and said method.

An optronic system such as a camera, binoculars, a telescope, a viewfinder, a gyro-stabilized ball (BGS) equipping an airborne observation system, generally uses image stabilization techniques. These image stabilization techniques make it possible to reduce motion blur in the images induced by more or less voluntary movements of the optronic system. Image stabilization techniques therefore make it possible to obtain an improvement in the images in terms of quality and precision.

Image stabilization techniques can use optical stabilization methods. Optical stabilization methods consist in stabilizing an image acquisition by varying an optical path followed by a light beam representative of a scene towards a sensitive surface such as a film or a sensor. Optical stabilization methods include, for example, lens stabilization methods and sensor stabilization methods. Known methods of objective stabilization use floating lenses moving orthogonally with respect to an optical axis of the objective using electromagnets. Movements are detected by gyrometers detecting horizontal and vertical movements and can thus be compensated by controlling the position of the floating lenses using electromagnets. The sensor stabilization methods are dedicated to digital devices. Each image requiring a longer or shorter acquisition time depending on light conditions, these methods consist of moving an image sensor during the acquisition time of an image so as to compensate for the movements of a digital device .

BGS generally use optical stabilization techniques such as lens stabilization where it is the lens and the sensor as a whole that are moved.

The Fig. 1A schematically shows an optronic system 10, such as for example a camera, consisting of an optical element comprising for example two groups of lenses, such as the groups of lenses 101 and 103, a mobile element 102 generally implemented by a lens floating, an image sensor 104 such as a CCD (charge-coupled device: “Charge-Coupled Device”) or CMOS (complementary metal-oxide semiconductor: “Complementary Metal-Oxide-Semiconductor” in English Anglo-Saxon terminology). The lens groups 101 and 103, the movable element 102 and the image sensor 104 are perpendicular to an optical axis 105. The lens groups 101 and 103 and the movable element 102 converge a light beam 100 towards the sensor. images 104. A movement sensor 106, implemented for example by a gyrometer, determines movements of the optronic system and transmits this information to a movement compensation device 107 modifying the position of the mobile element 102 so as to compensate for the movements of the optronic system 10. The movement compensation device 107 is implemented for example by motors or by electromagnets. The image sensor 104 produces images from the received light beam 100. The images are produced at a signal acquisition frequency (or image acquisition frequency) of the order of a few tens of Hertz and are transmitted in the direction of a display or a memory.

Image stabilization techniques, such as those used in cameras or BGS, make it possible to compensate for low frequency movements and medium frequency movements. On the other hand, these image stabilization techniques are generally not very effective in compensating for high frequency movements. Indeed, high frequency movements are very fast movements that only high precision movement sensors are able to capture. Furthermore, an optronic system is considered to be a rigid structure undergoing an overall movement when low and medium frequency movements are applied to this system. On the other hand, when a movement of high frequencies is applied to an optronic system, this system is considered as a deformable structure, on which local micromovements are applied. An application of conventional image stabilization techniques to the processing of high frequency movements could consist in introducing several high precision sensors into optronic system structures. However, this solution leads to an increase in the complexity and cost of manufacturing optronic systems.

Some optronic systems that do not aim to deliver high precision images can afford not to process high frequency movements. For these systems, it is considered acceptable that after compensation for low and medium frequency movements by conventional image stabilization techniques, uncompensated residual high frequency movements may remain and induce sharpness problems in the images. images.

The document US2003 / 0197787 A1 discloses a camera comprising an image sensor and an image stabilization system comprising a sensor for detecting movement of the camera.

The document US 5,218,442 discloses a camera comprising a film and an image stabilization system comprising a high frequency motion detection sensor. Unlike the present invention, these documents do not make use of a conversion matrix to determine values representative of the movement of the image being acquired by the image sensor.

The situation is very different for optronic systems requiring high precision such as a BGS or a camera equipped with a telephoto lens. Indeed, in this case, an image of poor quality could cause a misinterpretation of the contents of the image.

The object of the invention is to solve the problems mentioned above. The invention aims in particular to provide a simple and efficient method and device for processing high frequency movements undergone by an optronic system and an optronic system implementing this method and this device for processing high frequency movements.

To this end, according to a first aspect of the present invention, the present The invention relates to a method of processing high frequency movements in an optronic system as defined by claim 1. Embodiments of this method are defined by claims 2 to 7.

In this way, very weak movements can be picked up and compensated.

According to a second aspect of the invention, the invention relates to an optronic system as defined by claim 8. An embodiment of this system is defined by claim 9.

According to a third aspect of the invention, the invention relates to a computer program product as defined by claim 10.

According to a fourth aspect of the invention, the invention relates to storage means as defined by claim 11.

• La Fig. 1A représente schématiquement un exemple de système optronique classique,
• La Fig. 1B représente schématiquement un exemple de système optronique mettant en œuvre un dispositif et un procédé de traitement apte à traiter des mouvements de hautes fréquences du système optronique,
• La Fig. 2 illustre schématiquement un exemple d'architecture matérielle d'un dispositif de traitement apte à traiter les mouvements de hautes fréquences du système optronique,
• La Fig. 3 représente schématiquement une photodiode multi-éléments,
• La Fig. 4 représente schématiquement un exemple de procédé de traitement de mouvements de hautes fréquences d'un système optronique mis en œuvre par le dispositif de traitement,
• La Fig. 5 représente schématiquement un exemple de procédé de détermination d'une matrice de conversion utilisée dans le procédé de traitement de mouvements de hautes fréquences,
• La Fig. 6 illustre une mise en correspondance de pixels d'une image avec des quadrants d'une photodiode multi-éléments utilisée dans le procédé de traitement de mouvements de hautes fréquences.

The characteristics of the invention mentioned above, as well as others, will emerge more clearly on reading the following description of an exemplary embodiment, said description being given in relation to the accompanying drawings, among which:

• The Fig. 1A schematically represents an example of a classical optronic system,
• The Fig. 1B schematically represents an example of an optronic system implementing a device and a processing method capable of processing high frequency movements of the optronic system,
• The Fig. 2 schematically illustrates an example of the hardware architecture of a processing device capable of processing the high frequency movements of the optronic system,
• The Fig. 3 schematically represents a multi-element photodiode,
• The Fig. 4 schematically represents an example of a method for processing high frequency movements of an optronic system implemented by the processing device,
• The Fig. 5 schematically represents an example of a method for determining a conversion matrix used in the method for processing high frequency movements,
• The Fig. 6 illustrates a mapping of pixels of an image to quadrants of a multi-element photodiode used in the high frequency motion processing method.

The Fig. 1B schematically represents an example of an optronic system 11 implementing a device and a processing method capable of processing the high frequency movements of the optronic system 11. The optronic system 11 comprises elements identical to the optronic system 10 of the Fig. 1A , each identical element being represented by the same reference. The optronic system 11 nevertheless comprises additional elements allowing the processing of high frequency movements. The optronic system 11 comprises a device 108 making it possible to redirect a part of the light beam 100 towards a high frequency optical movement sensor 109. The device 108 may for example be a semi-reflecting device. The high-frequency motion sensor 109 may for example be a multi-element photodiode (“multi-element photodiode” in English terminology, which we call “ME photodiode” hereinafter), such as a two, four or four photodiode. eight quadrants, a position-sensing photodiode (“position-sensing photodiode” in English terminology) or a matrix sensor comprising very few pixels but operating at high frequency. The high-frequency movement sensor 109 is able to supply values representative of a movement with a signal acquisition frequency of the order of a few KHz. It is assumed here that the image sensor and the high frequency motion sensor operate in similar and preferably identical light wavelength ranges, just as these two sensors have similar and preferably identical spectral responses in these ranges. of wavelengths.

The Fig. 3 schematically represents an ME photodiode. ME photodiodes are devices commonly used for focusing in laser readers such as CD or DVD players or in satellites for slaving laser lines of sight. An ME photodiode operates in most cases with a signal acquisition frequency of a few KHz. An ME photodiode is generally circular in shape, as shown in Fig. 3 , or square. An ME photodiode is made up of several quadrants. The Fig. 3 represents a photodiode ME comprising four quadrants denoted A, B, C and D. Each quadrant comprises a sensor which we call mega-pixels hereinafter. Each mega-pixel produces a signal when struck by a light beam. We denote by S _A , S _B , S _C and S _D the signals produced respectively by the mega-pixels of quadrants A, B, C and D. Furthermore, a photodiode ME produces two signals Δ X and Δ Y. The signals Δ X and Δ Y are representative of a movement in a scene corresponding to the light beam received by the photodiode ME.

$$\mathrm{\Delta }Y=\frac{S_A+S_B-S_C-S_D}{S_A+S_B+S_C+S_D};$$

The values of the signals Δ X and Δ Y are linked to the values of the signals S _A , S _B , S _C and S _D by the following relationships: $$\mathrm{\Delta }X=\frac{-S_{\mathrm{AT}}+S_B-S_{\mathrm{VS}}+S_D}{S_{\mathrm{AT}}+S_B+S_{\mathrm{VS}}+S_D};$$

$$\mathrm{\Delta }Y=\frac{S_{\mathrm{AT}}+S_B-S_{\mathrm{VS}}-S_D}{S_{\mathrm{AT}}+S_B+S_{\mathrm{VS}}+S_D};$$

ME photodiodes are commonly used in optronic systems for tracking an object in a scene. The objects tracked are generally objects of known shapes, such as for example laser pointers generated by laser designators. ME photodiodes have a barycenter corresponding to a reference position of a tracked object. In the ME photodiode represented schematically by the Fig. 3 , the barycenter of the photodiode ME corresponds to the reference 300. As long as a tracked object is positioned on the barycenter of the photodiode ME, the photodiode ME produces zero signals Δ X and Δ Y. As soon as the tracked object moves away from the barycenter of the photodiode ME, at least one of the signals Δ X and Δ Y becomes non-zero, which subsequently makes it possible to reset the optronic system on the tracked object.

In the optronic system 11, the light beam 100 is transmitted simultaneously towards the image sensor 104 and the high frequency movement sensor 109, each sensor receiving a part of the light beam 100. In this way, high frequency movements can affecting the acquisition of images by the image sensor 104 are detected and values representative of these movements can be measured by the high frequency motion sensor 109 with a signal acquisition frequency of the order of a few KHz.

The values representative of the movement measured by high-frequency movement sensors cannot, in general, be used directly by movement compensation devices because they depend on the content of the scene. In the case of the optronic system 11, the values representative of the movement are transmitted to a processing device 110 determining the movement values that can be used by the movement compensation device 107. These movement values that can be used by the movement compensation device 107 are representative of movements in the images acquired by the image sensor 104 and are measured in numbers of pixels. We will call these movements "pixel movements" hereafter. There is a direct relationship between pixel movements and movements of the optronic system. The movements of an optronic system that interest us here are angular movements. An optical system has a focal length f known by construction The distance focal length f is a parameter making it possible to make the link between pixel movement and angular movement.

The values representative of the pixel movements can therefore be used directly by the movement compensation device 107 to compensate for the movements of the optronic system. In this way, the processing device 110 controls the motion compensation device 107 based on the values of the motion representative signals Δ X and Δ Y produced by the ME photodiode 109. When the high frequency motion sensor 109 is a photodiode ME, there is a relationship between the values of the signals Δ X and Δ Y and the values representative of the pixel movements.

où M est une matrice de conversion carrée 2x2 dépendant d'un contenu de la scène visée.When the representative values of the pixel movements are expressed as a horizontal movement value Δ x and a vertical movement value Δy , the relationship between the signals Δ X and Δ Y and the values representative of the pixel movements is the next : $$\left(\begin{array}{c}\mathrm{\Delta }X\\ \mathrm{\Delta }Y\end{array}\right)=M.\left(\begin{array}{c}\mathrm{\Delta }x\\ \mathrm{\Delta }y\end{array}\right)$$

where M is a 2x2 square conversion matrix depending on a content of the target scene.

où M' est une matrice de conversion carrée 2x2 qui dépend d'un contenu de la scène visée.However, the values of the pixel movements could just as easily be expressed in the form of polar coordinates comprising a direction of movement θ and an amplitude of movement p. In this case the relation would be as follows: $$\left(\begin{array}{c}\mathrm{\Delta }X\\ \mathrm{\Delta }Y\end{array}\right)=M\prime .\left(\begin{array}{c}\rho \\ \theta \end{array}\right)$$

where M ' is a 2x2 square conversion matrix which depends on a content of the target scene.

When the values representative of pixel movements are determined by the processing device 110, they are transmitted to the movement compensation device 107 so that it can compensate for these pixel movements. In one embodiment, the frequency of transmission of the values representative of the pixel movements to the motion compensation device 107 is equal to the signal acquisition frequency of the high-frequency motion sensor 109 (ie the signal acquisition frequency of the ME photodiode). In this way, the motion compensation device 107 can compensate for movements of high frequencies. Note that the motion compensation device 107 always receives information representative of low and medium frequency movements from the motion sensor 106. Thus, the motion compensation device 107 can compensate for low, medium and high movements. frequencies.

The Fig. 4 schematically represents an example of a method of processing high frequency movements of the optronic system 11 implemented by the processing device 110. This method comprises obtaining by the processing device 110 at least one value of a representative signal of movements of the optronic system 11. When the high-frequency movement sensor 109 is a photodiode ME, the processing device 110 obtains two values of a signal representative of pixel movements. As we have seen above, a pixel movement is representative of a movement of the optronic system. In a step 401, the processing device 110 obtains values of the signals Δ X and Δ Y from the high frequency motion sensor 109.

où M ^-1 est l'inverse de la matrice de conversion M.During a step 402, the processing device 110 determines the values representative of the pixel movements as follows: $$\left(\begin{array}{c}\mathrm{\Delta }x\\ \mathrm{\Delta }y\end{array}\right)=M^{-1}.\left(\begin{array}{c}\mathrm{\Delta }X\\ \mathrm{\Delta }Y\end{array}\right)$$

where M ^-1 is the inverse of the conversion matrix M.

où M' ^-1 est l'inverse de la matrice de conversion M'.When the representative values of pixel movements are expressed in polar coordinates, the following relation applies: $$\left(\begin{array}{c}\rho \\ \theta \end{array}\right)={\left(M\prime \right)}^{-1}.\left(\begin{array}{c}\mathrm{\Delta }X\\ \mathrm{\Delta }Y\end{array}\right)$$

where M ' ^-1 is the inverse of the conversion matrix M '.

The conversion matrix M (resp. M ') is assumed to be known by the processing device 110 during step 402. We will describe below in relation to the Fig. 5 a method for determining the conversion matrix M (resp. M ′) periodically implemented by the processing device 110.

In a step 403, the processing device 110 transmits the values representative of the pixel movements thus calculated to the movement compensation device 107 so that it can implement feedback in the optronic system to compensate for the calculated pixel movement. In this embodiment, the transmission of the values representative of the pixel movements follows the signal acquisition frequency of the photodiode ME. The feedback can therefore be implemented at the signal acquisition frequency of the photodiode. High frequency feedback is therefore obtained.

The processing device 110 then waits for reception of new values of the signals Δ X and Δ Y from the high-frequency motion sensor 109. When new values of the signals Δ X and Δ Y are received, the device for processing again implements step 401.

The method for determining values representative of pixel movements described in relation to the Fig. 4 requires knowledge of the conversion matrix M (resp. M '). In optronic devices for object tracking or focusing, the conversion matrix M (resp. M ′ ) is generally a known constant matrix. The invention addresses the case of uncalibrated optronic devices such as cameras or BGSs. In this case, the conversion matrix M (resp. M ') changes over time and depends on the scene on which the photodiode ME is focused. It is then necessary to determine the conversion matrix M (resp. M ′) and to update this matrix again to take into account the changes in the scene on which the photodiode ME is focused.

The Fig. 5 illustrates an example of a method of determining the conversion matrix M (resp. M ') periodically implemented by the processing device 110. In one embodiment, the method of determining the conversion matrix M (resp. M ') is implemented by the processing device 110 during each acquisition of an image by the image sensor 104. An image acquired by the image sensor 104 is called “original image” hereinafter.

In a step 500, an original image acquired by the image sensor 104 is obtained by the processing device 110. In one embodiment, the processing device 110 uses this original image as a reference image when determining the conversion matrix M (resp. M ' ) .

In a step 501, the processing device 110 simulates the signals ΔX _ref and Δ Y _ref that the photodiode ME would provide if it were subjected to a light beam corresponding to the reference image according to a simulation method which we will explain below. .

In a step 502, a variable n is initialized to zero.

During steps 503 to 507, the processing device 110 applies movements of predetermined values to the reference image to obtain a set of displaced images and, for each displaced image, simulates the signals Δ X and Δ Y that the photodiode ME if it were subjected to a light beam corresponding to the displaced image. These steps are detailed below.

During step 503, a pixel movement of a predetermined value comprising a horizontal movement value Δx _s ( n ) and a vertical movement value Δy _s ( n ) (resp. A movement direction value θ _s ( n) ) and a displacement amplitude value ρ _s ( n ) in the case of a movement expressed in polar coordinates) is obtained by the processing device 110. This predetermined movement value is obtained for example from a list of values of predetermined movements stored in a memory of the processing device 110.

In a step 504, a displaced image I (n) is created by displacing the pixels of the reference image by the value of the pixel movement ( Δx _s ( n ) , Δy _s ( n )) (resp. ( Θ _s ( n ), ρ _s ( n ))).

In a step 505, the processing device implements a method of simulating the values of the signals ΔX _s (n) and ΔY _s ( n ) that the photodiode ME would provide if it were subjected to a light beam corresponding to the displaced image I (n). This simulation process is explained below. During this step, the processing device 110 determines the values of the signals S _A , S _B , S _C and S _D.

$$\mathrm{\Delta }{Y}_s\left(n\right)=\frac{S_A+S_B-S_C-S_D}{S_A+S_B+S_C+S_D}-\mathrm{\Delta }{Y}_{\mathit{ref}};$$

The values of the signals ΔX _s ( n ) and ΔY _s ( n ) are then calculated as follows: $$\mathrm{\Delta }{X}_s\left(\mathrm{not}\right)=\frac{-S_{\mathrm{AT}}+S_B-S_{\mathrm{VS}}+S_D}{S_{\mathrm{AT}}+S_B+S_{\mathrm{VS}}+S_D}-\mathrm{\Delta }{X}_{\mathit{ref}};$$

$$\mathrm{\Delta }{Y}_s\left(\mathrm{not}\right)=\frac{S_{\mathrm{AT}}+S_B-S_{\mathrm{VS}}-S_D}{S_{\mathrm{AT}}+S_B+S_{\mathrm{VS}}+S_D}-\mathrm{\Delta }{Y}_{\mathit{ref}};$$

During a step 506, the variable n is incremented by one unit. During a step 507, the variable n is compared with a constant N which we explain later. When the variable n is less than N, the processing device 110 creates a new displaced image I (n) by returning to step 503. A predetermined pixel movement value different from any other predetermined pixel movement value already used for measurements. displaced images I (n) created previously is then obtained by the processing device 110.

If the variable n is equal to the constant N, step 507 is followed by a step 508 during which the conversion matrix M is determined.

ou $$\left(\begin{array}{c}\mathrm{\Delta }{X}_s\left(n\right)\\ \mathrm{\Delta }{Y}_s\left(n\right)\end{array}\right)=M\prime .\left(\begin{array}{c}{\rho }_s\left(n\right)\\ {\theta }_s\left(n\right)\end{array}\right)=\left(\begin{array}{cc}a{\prime }_{11}& a{\prime }_{21}\\ a{\prime }_{12}& a{\prime }_{22}\end{array}\right).\left(\begin{array}{c}{\rho }_s\left(n\right)\\ {\theta }_s\left(n\right)\end{array}\right)$$

en coordonnées polaires.The constant N fixes the number of images I (n) necessary for the computation of the conversion matrix M (resp. M '). The conversion matrix M (resp. M ') being a 2x2 matrix, it has four coefficients. The coefficients of the conversion matrix M (resp. M ' ) form a set of four unknowns to be determined. For each displaced image I (n), the following relation applies: $$\left(\begin{array}{c}\mathrm{\Delta }{X}_s\left(\mathrm{not}\right)\\ \mathrm{\Delta }{Y}_s\left(\mathrm{not}\right)\end{array}\right)=M.\left(\begin{array}{c}\mathrm{\Delta }{x}_s\left(\mathrm{not}\right)\\ \mathrm{\Delta }{y}_s\left(\mathrm{not}\right)\end{array}\right)=\left(\begin{array}{cc}{\mathrm{at}}_{11}& {\mathrm{at}}_{21}\\ {\mathrm{at}}_{12}& {\mathrm{at}}_{22}\end{array}\right).\left(\begin{array}{c}\mathrm{\Delta }{x}_s\left(\mathrm{not}\right)\\ \mathrm{\Delta }{y}_s\left(\mathrm{not}\right)\end{array}\right)$$

or $$\left(\begin{array}{c}\mathrm{\Delta }{X}_s\left(\mathrm{not}\right)\\ \mathrm{\Delta }{Y}_s\left(\mathrm{not}\right)\end{array}\right)=M\prime .\left(\begin{array}{c}{\rho }_s\left(\mathrm{not}\right)\\ {\theta }_s\left(\mathrm{not}\right)\end{array}\right)=\left(\begin{array}{cc}\mathrm{at}{\prime }_{11}& \mathrm{at}{\prime }_{21}\\ \mathrm{at}{\prime }_{12}& \mathrm{at}{\prime }_{22}\end{array}\right).\left(\begin{array}{c}{\rho }_s\left(\mathrm{not}\right)\\ {\theta }_s\left(\mathrm{not}\right)\end{array}\right)$$

in polar coordinates.

This relation therefore provides two equations for each image I (n). Knowing for each image I (n) the values of the signals ΔX _s ( n ) and ΔY _s ( n ) and the values of the corresponding pixel movement ( Δx _s ( n ), Δy _s ( n )) (resp. ( Θ _s ( n ) , ρ _s ( n ))) , it is necessary and sufficient to have two images I (n) to be able to calculate the four coefficients of the conversion matrix M (resp. M ' ) . In theory, it is therefore sufficient to fix the constant N to the value two to determine the conversion matrix M (resp. M '). However, to avoid obtaining noisy values of the coefficient of the conversion matrix M (resp. M ′) , it is preferable to fix the constant N at a value greater than two.

We then obtain a system of 2N ( N > 2) equations with four unknowns which can be solved in a conventional manner by a linear regression during step 508.

As soon as it is determined, the conversion matrix M (resp. M ′) is used by the processing device 110 during the step of determining pixel movements 402.

As we have seen above, the method for determining the matrix M (resp. M ' ) comprises during steps 501 and 505 a method for simulating the values of the signals ΔX _ref and ΔY _ref and of the signals ΔX _s ( n ) and ΔY _s ( n ) that the photodiode ME would provide if it were subjected to a light beam corresponding respectively to the reference image or to the displaced image I (n). Let be an image Î which can be a reference image or a displaced image I (n). During the implementation of the simulation method, the processing device 110 implements correspond each dial of the photodiode ME with a set of pixels of the image Î . For each dial, a sum of the values of the pixels of the image Î corresponding to the dial is calculated. The value of a signal S _i ( i ∈ { A , B , C , D }) is then equal to the sum of the values of the pixels of the image Î calculated on the corresponding dial. The Fig. 6 represents an example of the matching of pixels of an image Î comprising 100 pixels with a four quadrant photodiode. In this example, the value of the signal S _A is calculated as the sum of the values of the pixels p1 to p15 corresponding to the dial A.

In the embodiment of the optronic system 11 described in relation to the Fig. 1B , a single motion compensation device is used to process the low and mid frequency movements perceived by the motion sensor 106 and the high frequency movements perceived by the high frequency motion sensor 109. In another embodiment, at least two motion compensation devices are used. A first device processes the low and medium frequency movements perceived by the motion sensor 106 and a second device processes the high frequency movements perceived by the high frequency motion sensor 109.

In one embodiment, the conversion matrix M (resp. M ' ) is periodically updated by the method for determining said matrix with a frequency lower than the image acquisition frequency of the image sensor 104 The frequency of implementation of the method for determining the conversion matrix M (resp. M ′) can also be adaptive, as a function for example of statistics on the evolution of the values of the signals Δ X and ΔY. The frequency can for example be increased when the statistics show that the optronic system is in a period of strong movements and decreased when the statistics show that the optronic system is in a period of weak movements.

In one embodiment, when creating the displaced images, a reference image resulting from a sub-pixel interpolation of the original image obtained by the image sensor 104 is used rather than the original image directly. The sub-pixel interpolation used can be half, quarter or eighth pixel interpolation. In this way, movements of high frequencies of low amplitude can also be processed by the movement compensation device 107. The sub-pixel interpolation can be adaptive as a function for example of statistics on the evolution of the values of the signals ΔX and Δ Y. Interpolation can be suppressed when statistics show that the optronic system is in a period of strong movements. Half or quarter pixel interpolation can be used when statistics show that the optronic system is in a period of medium motion. An eighth pixel interpolation can be used when statistics show the optronic system is in a period of weak motion.

In one embodiment, the transmission frequency of the values representative of the pixel movements to the movement compensation device 107 is lower than the signal acquisition frequency of the high-frequency motion sensor 109 (ie the signal acquisition frequency of the ME photodiode) while remaining higher than the signal acquisition frequency of the image sensor 104. The transmission frequency of the values representative of the pixel movements to the motion compensation device 107 can be set adaptively based on statistics on the signals Δ X and Δ Y. For example, when the statistics show that the signals Δ X and Δ Y vary little over time, the frequency of transmission of the values representative of the pixel movements to the motion compensation device 107 can be reduced. On the other hand, when the statistics show that the signals Δ X and Δ Y vary rapidly over time, the frequency of transmission of the values representative of the pixel movements to the motion compensation device 107 can be increased.

Until then, we have assumed that the images generated by the image sensor have only one component. These images can for example be grayscale images.

In one embodiment, the images generated by the image sensor 104 are multi-component images such as RGB images. In this case, each pixel of an image generated by the image sensor comprises three components.

During steps 501 and 505, the processing device 110 simulates the signal that the ME photodiode 109 would generate if it were subjected to a light signal corresponding to a given image. During this calculation, the sum of the values of the pixels included in each dial of the photodiode is calculated. In the case of pixels comprising several components, the value of a pixel is given by a linear combination of the components. For example, for a pixel comprising a red component (R), a green component (G) and a blue B component (B), the pixel p- value used in the simulation is: $$p=\mathrm{at}.R+b.G+\mathrm{vs}.B$$

The coefficients a , b and c depend both on spectral response characteristics of the image sensor 104 and of the ME photodiode 109. The coefficients a, b and c are assumed to be known by construction.

In one embodiment, the high frequency sensor 109 and the image sensor 104 have different spectral responses. It is important that the spectral responses of the two sensors are as close as possible and ideally that these spectral responses are identical. If, for example, there is a red zone in a scene and the image sensor 104 is not sensitive to this color, it is difficult to correctly simulate the signals produced by the ME 109 photodiode since the image obtained by the image sensor 104 does not contain this information.

In order to correct this situation, a correction filter is inserted in front of the image sensor 104. The function of the correction filter is to correct for differences in the spectral responses of the two sensors. The spectral response of a sensor is represented by a quantum efficiency curve. The corrective filter is adjusted so that the quantum efficiency curve of the image sensor 104 after filtering of the light beam 100 by the correcting filter is as close as possible to the quantum efficiency curve of the high frequency sensor 109. The correcting filter can be implemented by a color filter used to obtain a coarse fit on which is added a stack of thin layers to obtain a fine fit of the filter.

In one embodiment, the correction filter is placed in front of the ME 109 photodiode.

In one embodiment, a correction filter is placed in front of the ME photodiode 109 and in front of the image sensor 104.

In one embodiment, the ME photodiode is replaced by a position detecting photodiode. A position-detecting photodiode provides the position of a barycenter of an image called a "photometric barycenter". A movement of the image causes a movement of its barycenter which causes variations in a signal produced by the position detector photodiode.

$$\mathrm{\Delta }\mathrm{Y}=\mathrm{\Sigma }\left({\mathrm{p}}_{\mathrm{i}}\ast {\mathrm{y}}_{\mathrm{i}}\right)/{\mathrm{\Sigma p}}_{\mathrm{i}}$$

ou i est une variable variant de « 1 » à N_p The method of processing high frequency movements described in relation to the Fig. 4 and the method for determining the conversion matrix M remain the same. However, the values of the signals ΔX and ΔY calculated in simulation during steps 501 and 505 are coordinates of the barycenter of the image. If an image contains N _p pixels, each pixel having a value pi and coordinates ( x _i , y _i ) , then the coordinates of the barycenter are: $$\mathrm{\Delta }\mathrm{X}=\mathrm{\Sigma }\left({\mathrm{p}}_{\mathrm{i}}\ast {\mathrm{x}}_{\mathrm{i}}\right)/{\mathrm{\Sigma p}}_{\mathrm{i}}$$

$$\mathrm{\Delta }\mathrm{Y}=\mathrm{\Sigma }\left({\mathrm{p}}_{\mathrm{i}}\ast {\mathrm{y}}_{\mathrm{i}}\right)/{\mathrm{\Sigma p}}_{\mathrm{i}}$$

where i is a variable varying from "1" to N _p

The Fig. 2 schematically illustrates an example of the hardware architecture of the processing device 110. The processing device 110 comprises, connected by a communication bus 1105: a processor or CPU (“Central Processing Unit” in English) 1100; a random access memory RAM (“Random Access Memory” in English) 1101; a ROM (“Read Only Memory”) 1102; a storage unit 1103 or a storage media reader, such as an SD ("Secure Digital") card reader or USB ("Universal Serial Bus") key reader or an HDD ("Hard Disk Drive ”in English); at least one interface 1104 making it possible to exchange data with other devices. The interface 1104 allows for example the processing device 110 to receive signal values Δ X and Δ Y from the high frequency sensor 109 and original images from the image sensor 104.

Processor 1100 is capable of executing instructions loaded into RAM 1101 from ROM 1102, external memory (not shown), storage media, or a communications network. When processing device 110 is powered on, processor 1100 is able to read instructions from RAM 1101 and execute them. These instructions form a computer program causing the implementation, by the processor 1100, of all or part of the algorithms and steps described in relation to the processing device 110 and the Figs. 4 and 5 .

All or part of the algorithms and steps described above can thus be implemented in software form by executing a set of instructions by a programmable machine, such as a DSP (“Digital Signal Processor” in English) or a microcontroller, or be implemented in hardware form by a machine or dedicated component, such as an FPGA (“Field-Programmable Gate Array”) or an ASIC (“Application-Specific Integrated Circuit”).

Claims (11)

1. Method for processing high-frequency movements in an optronic system comprising an image sensor (104) operating with a first signal-acquisition frequency, called the image frequency, each image obtained by the image sensor being representative of a scene, characterized in that the method comprises the following steps:

   obtaining (401) values of signals representative of a movement of said scene from a high-frequency movement sensor (109), the high-frequency movement sensor generating values of signals representative of movements in said scene with a second signal-acquisition frequency, called the movement frequency, higher than the image frequency;

   determining (402) values representative of a movement in an image being acquired by the image sensor from the values of the signals representative of the movement of said scene, the determination of the values representative of the movement in the image being acquired by the image sensor comprising a matrix operation between the values of the signals representative of the movement of said scene and a conversion matrix;

   transmitting (403) the values representative of the movement in the image being acquired by the image sensor that are thus determined to a movement-compensating device (107) in order that said movement-compensating device may provide a feedback in the optoelectronic system in order to compensate for the movement in the image being acquired by the image sensor, the feedback being provided in the optoelectronic system with a frequency lower than or equal to the movement frequency and consisting in modifying a position of a movable element (102) contributing to make a light beam (100) converge toward the image sensor (104), the image sensor (104) producing each image from said light beam (100);

   an inverse matrix of the conversion matrix being determined via a determining method comprising the following steps:

   - obtaining (500) a reference image from an original image acquired by the image sensor,

   - simulating (501) first values of the signals representative of a movement, these values corresponding to those obtained by the high-frequency movement sensor when the high-frequency movement sensor is subjected to a light beam corresponding to the reference image,

   - applying (504) movements of preset movement values to the reference image to obtain a set of displaced images,

   - for each displaced image of the set of displaced images, simulating (505) second values of the signals representative of a movement, these values corresponding to those obtained by the high-frequency movement sensor when the high-frequency movement sensor is subjected to a light beam corresponding to the displaced image,

   - determining (508) the inverse matrix of the conversion matrix on the basis of the first and second values of the signals representative of a movement, which values are obtained by the high-frequency movement sensor, and of the preset movement values.
2. Method according to Claim 1, characterized in that the high-frequency movement sensor is a multi-element photodiode.
3. Method according to Claim 1 or 2, characterized in that the conversion matrix is updated periodically at a frequency lower than or equal to the image frequency.
4. Method according to Claim 1, 2 or 3, characterized in that, for each displaced reference image, the preset movement values comprise a horizontal-movement value and a vertical-movement value.
5. Method according to Claim 1, 2 or 3, characterized in that, for each displaced reference image, the preset movement values comprise a movement-direction value and a movement-amplitude value.
6. Method according to any one of the preceding claims, characterized in that the set of displaced reference images comprises at least two images.
7. Method according to any one of the preceding claims, characterized in that the reference image results from an application of a sub-pixel interpolation to the original image obtained by the image sensor.
8. Optronic system comprising an image sensor (104) operating with a first signal-acquisition frequency, each image obtained by the image sensor being representative of a scene, and a movement-compensating device (107), characterized in that the system comprises:

   a high-frequency movement sensor (109) that generates values of signals representative of movements in said scene with a second signal-acquisition frequency higher than the first signal-acquisition frequency; and

   a device (110) for processing high-frequency movements, said movement-processing device (110) comprising:

   means for obtaining (401) values of signals representative of a movement of said scene from said high-frequency movement sensor (109),

   means for determining (402) values representative of a movement in an image being acquired by the image sensor from the values of the signals representative of the movement of said scene, the means for determining the values representative of the movement in the image being acquired by the image sensor comprising a means for implementing a matrix operation between the values of the signals representative of the movement of said scene and a conversion matrix;

   means for transmitting (403) the values representative of the movement in the image being acquired by the image sensor that are thus determined to said movement-compensating device (107) in order that said movement-compensating device may provide a feedback in the optoelectronic system in order to compensate for the movement in the image being acquired by the image sensor, the feedback being provided in the optoelectronic system with a frequency lower than or equal to the movement frequency and consisting in modifying a position of a movable element (102) contributing to make a light beam (100) converge toward the image sensor (104), the image sensor (104) producing each image from said light beam (100) ;

   the device furthermore comprising means for determining an inverse matrix of the conversion matrix, comprising:

   - means for obtaining (500) a reference image from an original image acquired by the image sensor,

   - means for simulating (501) first values of the signals representative of a movement, these values corresponding to those obtained by the high-frequency movement sensor when the high-frequency movement sensor is subjected to a light beam corresponding to the reference image,

   - means for applying (504) movements of preset movement values to the reference image to obtain a set of displaced images,

   - means for simulating (505), for each displaced image of the set of displaced images, second values of the signals representative of a movement, these values corresponding to those obtained by the high-frequency movement sensor when the high-frequency movement sensor is subjected to a light beam corresponding to the displaced image, and,

   - means for determining (508) the inverse matrix of the conversion matrix on the basis of the first and second values of the signals representative of a movement, which values are obtained by the high-frequency movement sensor, and of the preset movement values.
9. System according to Claim 8, wherein a corrective filter is inserted in front of the image sensor and/or in front of the high-frequency sensor in order to compensate for a difference between a spectral response of the image sensor and a spectral response of the high-frequency movement sensor.
10. Computer-program product, characterized in that it contains instructions to implement, by means of a device (110) for processing high-frequency movements belonging to a system according to Claim 8, the method according to any one of Claims 1 to 7, when said program is executed by a processor of said device.
11. Storing means, characterized in that they store a computer program containing instructions to implement, by means of a device (110) for processing high-frequency movements belonging to a system according to Claim 8, the method according to any one of Claims 1 to 7 when said program is executed by a processor of said device.

Images