FR3123486A1 - Process for constructing an image from a variable focal length optical device. - Google Patents

Abstract

Process for constructing an image from a variable focal length optical device. Method for acquiring images of a first surface (12) of a human body using an image pickup device (20) comprising variable focal length optics (40), said method comprising the steps of: acquisitions of a plurality of images using the image pickup device (20), said acquisitions being carried out by varying the focal length (df) of the variable focal length optics (40) according to a control law;constructing a sharp image of a portion of the surface (12) of the human body from the plurality of acquired images of said surface (12),estimating the depth map of the local geometry of the first surface (12). Figure for abstract: Fig.1

Description

Process for constructing an image from a variable focal length optical device.

Field of invention

The field of the invention relates to that of optical devices with variable focal length making it possible to acquire images of a body. In particular, the field of the invention relates to the field of liquid lens optical devices making it possible to acquire high-resolution images of a human body and to exploit these images. The field of the invention finds particular application in guiding a robot arm at the end of which an optical device is configured to acquire images of the skin of an individual.

State of the art

Currently, to acquire sharp images with good resolution of an area of a body that one wishes to analyze, optics with a shallow depth of field are generally used. For example, in the context of dermatological applications, the quality of the image is essential for the a posteriori analysis of the surface of the skin. Thus, reduced-field optics are favored to highlight the definition of the colors and roughness of a surface of the skin.

However, these macrophotographic lenses which have a relatively shallow depth of field imply that the sharp area of the acquired image is also small. However, this solution is incompatible with the acquisition of curved areas in which the depth of field used makes certain parts of the vicinity of a point blurred since they leave the area of sharpness. It is therefore necessary to carry out acquisitions in the vicinity of this point either by modifying the shot or by modifying the optics to extend the acquisition zone by a larger surface. However, it is difficult to configure the shooting settings without first knowing the surface topology of the body whose images are to be acquired. This lack of knowledge makes the operations complex and time-consuming. Moreover, the different shooting configurations used are an obstacle to the restitution of a complete and sufficiently explicit portion of the surface.

The invention overcomes these drawbacks.

According to a first aspect, the invention relates to a method for acquiring images of a first surface of a human body using an image pickup device comprising variable focal length optics, said method comprising the steps of:

• acquisitions of a plurality of images using the image pickup device, said acquisitions being carried out by varying the focal length of the variable focal length optics according to a control law;
• construction of a sharp image of a portion of the surface of the human body from the plurality of images acquired from said surface,
• estimation of the depth map of the local geometry of the first surface.

One advantage is to allow the reconstruction of a clear image of a human body in order to allow, for example, to categorize dermatological patterns such as moles.

According to one embodiment, the construction of an image of a portion of the surface of the body comprises:

• cropping the images of the plurality of images from estimated sharpness information to generate sharp areas and blurred areas;
• assembling the sharp areas of said images to construct a sharp image of the surface of the portion of the body.

One advantage is to allow reconstruction of a complex surface of the human body by making the most of the optics and the images acquired over the entire area considered.

According to one embodiment, the method comprises measuring or estimating a distance between the variable focal length optics and a first point on the surface of the body, the control law taking into account said measured or estimated distance between the varifocal optics and the first point for modifying the focal length of the optics between at least two images acquired in the vicinity of the point on the surface of the body.

One advantage is to increase the performance of the control law by reducing the range of values of the variation of the focal distance. Furthermore, a distance measurement also allows a better estimation of the sharpness of the images.

According to one embodiment, the measurement or estimation of the distance between the variable focal length optics and a first point on the surface of the body is carried out:

• by a distance measurement per point and/or;
• by a distance measurement combined with known topological information of the body model of the body surface and/or;
• by known topological information of the body model of the surface of the body and position information of the optics with respect to the 3D model and/or;
• by a distance measurement combined with a measurement of the local geometry of the imaged area of the body surface and/or,
• by measuring the local geometry of the imaged area of the body surface and information on the position of the optics in relation to the 3D model.

An advantage is to obtain a better estimate by combining different estimates or measurements of the distance between the varifocal optics and a first point on the surface of the body so as to increase the performance and/or the quality of the result.

According to one embodiment, the measurement, calculation or estimation of the distance between the variable focal length optics and a first point on the surface of the body is carried out either by a measurement or by an estimation:

• using a distance measuring device;
• from sharpness information of all the acquired images and a blur and/or sharpness estimation algorithm.

According to one embodiment, the method comprises the steps of:

• Receiving a 3D model of at least a part of a body, said 3D model modeling at least a first surface of said body;
• Recovery of local topological data from the neighborhood of at least the first point, said local topological data being extracted from the 3D model or calculated from the 3D model;
• The acquisitions of a plurality of images being carried out by varying the focal length of the variable focal length optics according to a control law also taking into account the local topological data of the vicinity of the first point of the first surface.

An advantage is to know a topological information of the body which allows on the one hand to optimize the control law in calculation and in time and on the other hand to effectively guide a mobile device if necessary.

According to one embodiment, the first surface of the 3D model is a surface of a human body.

According to one embodiment, the first surface of the 3D model is a cloud of points, in particular a connected graph oriented in space.

An advantage is to have easily exploitable metrics to address and superimpose the sharp 3D images reconstructed on the body. Another advantage is to allow easy planning of guidance trajectories on the surface of the body. Another advantage is to make it possible to know the local depths of field of a region of the surface with precision to control the control law of the focal length of the optics.

According to one embodiment, the method comprises the steps of:

• Generation of a treatment trajectory on the surface of the 3D model by means of a computer, said treatment trajectory comprising at least two points on the surface of the 3D model;
• Construction of an image of a portion of the body surface in the vicinity of the treatment trajectory.

One advantage is to automatically generate a complete image of a human body by programming a mobile device such as a robot arm in advance.

According to one embodiment, the range of values of the focal length of the variable focal length optics is defined according to the local topological data of the 3D model in the vicinity of the first point of the processing trajectory.

According to one embodiment, the acquired images are processed so as to generate sharpness gradient information between the pixels of the same image, said gradients being exploited so as to generate alignment instructions between the successive acquired images, said realigned images to produce a set of image portions covering the same pixel areas, the sharpness or blur information or of these portions being compared to select the sharpest pixels, said selected pixels being merged to construct a 2D image or Composite 3D.

An advantage is to obtain a sharpness map of each image in order to collect and select the set of sharpest pixels of an image.

According to one embodiment, the construction of the image of a portion of the surface of the body comprising the cutting and the assembly of the images is carried out using a learning algorithm, of the neural network type, trained so as to producing a sharp 3D output image from a plurality of input acquired images.

One advantage is to generate sharp images quickly thanks to an artificial intelligence algorithm.

According to one embodiment, the image processing algorithm estimating the sharpness or the blur of each point of the images of the plurality of images is implemented by a first learning function, for example an automatic learning method.

According to one embodiment, the image processing algorithm comprises a selection and a cutting of each group of pixels of each image having a sharpness greater than a predefined threshold or a sharpness greater than the sharpness of pixels by at least one another image of the same area.

According to one embodiment, the method comprises a step of estimating the value of the distance between the variable focal length optics and at least a first point of the processing path from an estimate of the sharpness of points of the images of the plurality of images, said estimation being carried out using an image processing algorithm.

An advantage is to make it possible to generate a depth map by software means that do not require additional equipment. In the case where equipment making it possible to generate a depth map is used, the estimation step carried out by software means makes it possible, for example, to refine the measurement of a device.

According to one embodiment, the method comprises an estimation of the relative depth between the pixels of the same image and a calibration of the measurement of at least one pixel to generate absolute depth information of at least one pixel.

According to one embodiment, the variable focal length optics are arranged on a guide element configured to move a lens.

According to one embodiment, the method comprises, between the steps of generating a processing trajectory and measuring the distance, the steps of:

• Calculation of a guide trajectory for the variable focal length optics;
• Generation of a kinematics of the guiding element so that a point of the optics with variable focal length is slaved on the guiding trajectory;
• Activation of the kinematics of the guiding element to traverse the guiding trajectory, said guiding trajectory being calculated in real time from the 3D surface model.

According to one embodiment, the method includes a step of correcting the guidance trajectory from the real-time reconstruction of the 3D model.

According to another aspect, the invention relates to an image pickup device characterized in that it comprises variable focal length optics for acquiring a plurality of images of the surface of an individual's body using of the image pickup device, said acquisitions being made by varying the focal distance of the variable focal length optics according to a control law, said device comprising at least one computer configured to implement an image processing algorithm to generate a sharp 3D image.

According to one embodiment, the image pickup device comprises means for measuring a distance between the variable focal length optics and a first point on the surface of the body, said acquisitions being carried out by varying the focal length of the variable-focal lens according to a control law taking into account the distance between the variable-focal lens and the first point.

According to one embodiment, the image pickup device further comprises an interface for receiving a 3D model of at least one part of a human body, said model modeling at least a first surface of the body and a mobile platform, said computer(s) being configured for:

• Retrieving local topological data from the neighborhood of at least the first point, said local topological data being extracted from the 3D model or calculated from the 3D model;
• Control the acquisition of a plurality of images by the image pickup device, said acquisitions being carried out by varying the focal length of the variable focal length optics according to a control law taking into account the distance between the varifocal optics and the first point, and local topological data of the neighborhood of the first point of the first surface.

According to one embodiment, the image pickup device comprises a device for measuring the distance between the variable focal length optics and the first point configured to measure in real time a distance between the variable focal length optics and a first point on the surface of the body and means for measuring the orientation of the optics to deduce viewing angle information, said orientation and said distance being used by a computer of the device to correct or transform the acquired image.

According to one embodiment, the image pickup device comprises a guide element comprising variable focal length optics, the computer(s) being configured for:

• Generate a treatment trajectory on the surface of the 3D model, said treatment trajectory comprising at least two points on the surface of the 3D model;
• Calculate a guide trajectory of the varifocal optics;
• Generate a kinematics of the guide element so that a point of the variable focal length optics is slaved on the guide path;
• Activate a kinematics of the guiding element to traverse the guiding trajectory, said guiding trajectory being calculated in real time from the 3D surface model.

According to one embodiment, the variable focal length optic is a liquid lens.

According to another aspect, the invention relates to a mobile platform comprising an image capture device of the invention, said platform comprising a command interface for receiving guidance instructions calculated in particular from the body model calculated in real time .

According to one embodiment, the mobile platform is:

• a robot arm having a plurality of degrees of freedom;
• A drone comprising optics forming an image capture device;
• A guide rail on which the imaging device is movable or:
• A reception cabin for a human subject comprising means for guiding an image-taking device,
• A platform comprising a fixed part accommodating the image capture device a mobile part causing an individual to move vis-à-vis the fixed part.

According to one aspect, the invention relates to a gun-type device comprising a tip holding the image pickup device according to the invention.

Brief description of figures

Other characteristics and advantages of the invention will become apparent on reading the following detailed description, with reference to the appended figures, which illustrate:

: an example of a guidance device configured to acquire images of the surface of a body and to reconstruct a sharp image of at least a portion of the surface of the body;

: an example of an image pickup device of the invention guided along a guide trajectory to acquire images in the vicinity of a trajectory on the surface of a body;

: an example of a guidance device configured to acquire images of the surface of a human body such as a patient;

: an example of a flowchart representing the steps of the image acquisition process for their processing in order to reconstruct a clear image of all or part of a surface of a body.

The illustrates an image acquisition device 20 of the invention comprising variable focal length optics 40. An image acquisition device 20, also called an image pickup device, is configured to acquire a plurality of images at the neighborhood of at least one point located on the surface of a body. The invention finds particular application in dermatology and the analysis of images of an individual's skin. However, the invention is not limited to this application and can be implemented in other fields. Finally, the invention is particularly advantageous when image acquisition devices 20 comprise optics suitable for macrophotography, or even microphotography, such as optics having a large aperture.

One objective of the invention is to reconstruct the photograph of the surface of a body so that each portion of the reconstructed three-dimensional image is as sharp as possible.

In the remainder of the description, the term “body” will mean the body of a human. However, the invention is not limited to this embodiment, the invention can be implemented for any type of body having a three-dimensional surface of which it is sought to represent at least a portion as faithfully as possible.

The acquisition parameters, including the number of images, the dimensions of the images and the resolution of the images can be predefined or defined according to a displacement configuration of the acquisition device 20, for example when the latter is arranged on a mobile platform moving around the body.

The invention advantageously makes it possible to acquire a set of images in the vicinity of a point by varying the focal distance of the acquisition optics according to a control law taking into account the distance between the variable focal length optics and the target point.

The operation can thus be repeated on a plurality of points of a surface of a body in order to acquire several sequences each comprising a plurality of images in the vicinity of a plurality of points. This method can be carried out continuously on a set of points on a surface to cover all or part of the surface of a body or this method can be carried out along a trajectory on the surface of the body in order to acquire images in the vicinity of points defining the trajectory. Thus, step by step, the entire surface of a body can be treated.

Varifocal optics

According to one embodiment, the variable focal length optics 40 is advantageously a variable focal length lens, also called an adaptive lens. According to one embodiment, the lens used is a lens deformable by piezoelectric actuation. According to another embodiment, the lens is a liquid crystal lens. The latter use a change in the optical index of the liquid crystals in order to modify the focal length of the lens. Finally, another embodiment can be implemented from a liquid lens whose geometry is controllable by micro-fluidic parameters or mechanical parameters such as parameters defining the geometry of a membrane retaining a liquid or even a combination of the two. According to one example, a liquid lens comprises a circular membrane made of electroactive polymer. The central part of the membrane forms the lens and the peripheral part is surmounted by an annular electrode. By applying an electric voltage between this electrode and the substrate, an electrostatic pressure is created, which tends to move the fluid towards the central part, inflating the membrane, and thus modifying the curvature of the lens. These devices can be controlled with a pilot voltage.

These parameters can be adjusted, configured or controlled from a digital setpoint from a sensor or a computer or any other electronic device.

According to another embodiment, the variable focal length optic comprises a magnetic actuator making it possible to move an optical element, such as a lens, to modify the focal length. According to one example, the optical element moves for example in a tube, the position in the tube defining a focal distance specific to the optical configuration of the lens. The magnetic actuator can be controlled by an electromagnetic field generated from a dedicated component. The field can be adjusted, configured or controlled from a digital control setpoint. This digital setpoint can be generated by a control law from a computer or a sensor or any other electronic equipment.

According to another embodiment, the variable focal length optic comprises a mechanical actuator making it possible to move an optical element, such as a lens, to modify the focal length. The actuator can be controlled by an electronic component making it possible to receive a digital instruction and to generate a mechanical instruction. In this case, this digital setpoint can also be generated by a control law from a computer or a sensor or any other electronic equipment.

An advantage of using variable focal length optics is to overcome the shortcomings of macrophotography which acquires images with a relatively shallow depth of field which remains incompatible with the acquisition of images in a portion of curved surface for example on the calf or the arm. Another advantage is to have very low response times, for example a response time of less than 25ms.

Consequently, the use of variable focal length optics, for example in addition to optics with a larger aperture, makes it possible to acquire sufficient precision of the images of a surface of a body so as to reconstitute a clear global image by an image reconstruction algorithm.

In order to guide the shooting automatically in the vicinity of an aiming point, the invention makes it possible to automatically vary the focal length of an optic according to a control law in order to reconstitute a sharp image of the vicinity of the targeted point.

Liquid lens case

In the case of a liquid lens, the controller controls the numerical value of the focal length of the lens. In this example, the numerical value makes it possible to vary the shape of the lens.

According to one embodiment, the control of the liquid lens is carried out by means of a continuous signal such as a trigonometric function, for example a sine function or a linear function, for example a function whose shape is in -saw. Such control makes it possible to optimize the stabilization of the medium constituting the liquid lens. Such a control signal makes it possible to minimize the nonlinear effects and to reduce the relaxation times or the transient phases of modification of the medium of the liquid lens. Such a variation of the focal length of the lens is moreover consistent with the evolution of the curvature of the zone in the vicinity of the target point P ₁ . Indeed, there is no supposed discontinuity, except effect of artefact or effects of edges of the surface of the body, thus a law of control of the focal can be regular and continuous so as to make correspond the variations of surface topologies with variations in lens focal length.

According to one example, the image acquisition by the liquid lens 20 is carried out according to a predefined sampling over the entire range of focal lengths addressable by the lens. In this embodiment, the number of images acquired is constant according to a speed of movement of the imaging device. One advantage of this solution is to collect the greatest number of images possible in order to obtain a stratification of the images according to the different focal lengths of the lens. An advantage of this solution is to obtain a regular depth map of all the points defining pixels of the acquired image.

Picture

According to one embodiment, the acquired image is a color image in the visible spectrum. The frequency range of the acquired image can in this case be the visible, that is to say for wavelengths between 380 nm and 700 nm.

According to modes where ranges extend on either side of the visible spectrum, it is possible to acquire images in a broader spectrum from dedicated equipment. According to one example, the images acquired are acquired in an infrared range, that is to say for wavelengths between 700 nm and 1000 nm. According to another example, the images acquired are acquired in an ultraviolet range, that is to say for wavelengths between 10 nm and 380 nm.

According to one embodiment, the image is a multispectral image. In the latter case, spectral imaging equipment can be used. A configuration of such equipment with a controllable variable focal length can be used in the context of the invention. The equipment used for the acquisition can be configured to acquire images whose spectrum is notably distributed over several spaced frequency bands. An interest is to obtain an image over a wider band of frequencies including for example infrared, near infrared and ultraviolet.

According to another example which can be combined with the previous modes, the acquired image is a hyperspectral image. In the latter case, an imaging spectrometer, also called an imaging spectrometer or an imager associated with a spectrometer, makes it possible to generate a hyperspectral image. The equipment used for the acquisition can be configured to acquire images whose spectrum is notably distributed over several contiguous frequency bands. The equipment used can be configured with a controllable variable focal length. An interest is to obtain an image over a wider band of frequencies with a high degree of resolution including for example infrared, near infrared and ultraviolet.

According to one embodiment, the equipment is a chemical imager or a chemical imaging device configured to create an image from an acquired frequency spectrum such as a spectrometer and spatial and temporal information.

Means of measuring distance

According to one embodiment, the image pickup device 20 comprises a means for measuring the distance between the optical device with variable focal length 40 and a first point P ₁ of the surface of the body.

This means for measuring the distance can be an active physical device such as an emitted laser arranged close to the variable focal length optics 40. The emission of the laser beam makes it possible to measure the distance to the center of the acquired image. Other devices for measuring the distance between a point on the surface of the body and the variable focal length optics 20 can be used, for example a radioelectric rangefinder, an optical rangefinder or even an acoustic rangefinder. Other examples of distance measuring devices can be used in the context of the invention, such as a Lidar, a Radar, a Sonar or even a stadimetric rangefinder.

In order to carry out the distance measurement, the invention can implement a single-point measurement device such as a laser or a multipoint measurement system such as a laser, a so-called “matrix Time-of-Flight” device. , a lidar, an active/passive stereoscopic device, or even a device comprising a projection of structured light.

According to an alternative, or according to a complementary mode, the means for measuring the distance between the variable focal length optics 40 and a first point P ₁ of the surface of the body is software means implementing a computer processing the images acquired to extract sharpness or blur information therefrom in order to reconstruct from this datum information characterizing the distance between a point P ₁ of the surface and the optics. An exemplary embodiment may be an algorithm detecting the sharpest pixel(s) of an image in order to deduce therefrom information on the distance between a point P ₁ of the surface and the optics, knowing moreover the focal length of the optics used. The gradients between the sharpest pixels and the other pixels can also be exploited in order to consolidate the measurement of the distance between a point P ₁ of the surface and the optics. The focal distance information is then used to process the evaluation of the distance between the variable focal length optics and a point P ₁ on the surface of the body. All of the shades of sharpness or blur between the pixels can then be used to consolidate the calculation of the distance between a point P ₁ of the surface and the optics or to define a local depth map in the vicinity of the point P ₁ sight or a point whose sharpness is considered to be the highest.

According to an exemplary embodiment, an algorithm can be implemented by a learning function, of the function type driven by automatic learning, such as a deep learning algorithm. In this embodiment, a CNN-like convolutional neural network can be used. The learning function makes it possible in particular to detect the pixel or pixels supposed to be the sharpest in the image and to determine a distance between a point P ₁ of the surface and the optics.

To this end, a sharpness map, also called “defocus map” in English terminology, can be established for each of the images in the image stack by varying the focal length of the lens. Such a sharpness map makes it possible to encode blur or sharpness information, for example on a standardized scale of 0 to 1.

When a classical method is implemented, the sharpness map of an area is constructed by considering the sharpest pixels of each image.

Distance measurement methods from acquired images

According to an exemplary embodiment, the distance measured by a sensor can be combined with another measurement of the distance obtained either by another sensor or by another method making it possible to evaluate the distance.

Among the other methods, a first method consists in evaluating the distance between the variable focal length optics 40 and a point P ₁ of the surface of the body from the acquired images and by an algorithm for estimating the sharpness of different points of the image in order to reconstruct distance information between at least one point of the surface and the optics. Indeed, in this method the differences in sharpness of different points of different images having been acquired with different focal lengths make it possible to obtain an estimate of the distance of each of these points with respect to the optics. This estimate may or may not be combined with another method or data from a sensor. This estimate can be used to define an input of a control law Lc of the focal length of the optics 40.

Among the methods, one of them can be implemented if a priori knowledge of the volume or of the surface 12 of the body is known. According to an exemplary embodiment, the invention comprises a component making it possible to generate a body model, in particular of the three-dimensional surface of this body. This surface can be a surface defined in space or it can be defined by a cloud of points or a mesh. The cloud of points can be advantageously connected and oriented in space, we speak of spatial orientation in space. When the cloud of points is oriented, it is possible to locate the surface of the body model in space, for example within a predetermined reference frame R ₀ . When the body model is located in a frame of reference R ₀ and the image acquisition device is also, it is then possible to calculate a distance between a point P ₁ of the surface 12 of the body and the optics at variable focal length 40 and to use this distance to implement a control law Lc automatically controlling the focal length df of the variable focal length device 20.

From the following data: the position and the orientation of the body in 3D in R ₀ , the knowledge of the body model and the position of the optical device in R ₀ , it is possible to calculate in real time a control law of the optics according to the relative position of the latter with respect to the body.

According to a first case, this control law can be calculated in real time without having to plan a trajectory of the optics beforehand. This embodiment is particularly advantageous in the case of optics embedded in a mobile device of the pistol type, that is to say removable according to the movements of an operator's hand for example.

According to a second case, this control law can be calculated for example by planning a preliminary trajectory of the optics. This embodiment is particularly interesting in the case of an optics embedded in a robotic device of the type an articulated robot arm and more generally a device embedded in any removable platform.

Using a body template

When the invention implements an automatic generation of the body model of an individual, the latter can be generated from an optical device capturing partial views or complete views of the body of the individual. The partial views can then be used to generate a body model, for example from an algorithm implemented by a learning function, such as a function having been trained by automatic learning. An example is a convolutional neural network. Such a function makes it possible to generate in real time 3D body models of an individual from acquired partial views. Such a possibility makes it possible to calculate topological data in the vicinity of a target point P ₁ of the surface of the body during the scanning of a trajectory by the imaging device 20.

Topological descriptor

The knowledge of the topology can come from different devices or methods for calculating at least one local topological descriptor at the targeted point P ₁ .

According to a first embodiment, the descriptor comes from a calculation carried out in real time from an acquired image. This calculation can be performed with a second optical device configured with an aperture, a depth of field or a focal length allowing local analysis of the depth map in the vicinity of the point P ₁ . According to this same example, the second optical device can comprise optics in an infrared range. According to another example of a descriptor calculated in real time, a device comprising a projection of images whose deformation is calculated makes it possible to calculate a depth map. Another example is the use of structured light.

According to a second embodiment, the descriptor comes from a real-time calculation from an image extracted from a surface model of the human body which is oriented and positioned in space. Such a body model can for example be generated in real time from partial or total knowledge of a patient's body. A priori knowledge of the trajectory that has been planned and of the patient's body model for which local images of the skin are to be obtained and finally knowledge of the imaging device 20 and its arrangement vis-à-vis the bodies make it possible to generate a set of local descriptors a priori before the image acquisition process in the vicinity of the targeted points P ₁ . Such a technique makes it possible to configure the control law Lc of the variable focal length optics without depending on a real time constraint. Furthermore, this technique makes it possible to save calculation time in real time and makes it possible to optimize the best ratio of images necessary for the reconstruction of a clear image of all or part of the human body for which a clear image is desired.

According to one embodiment, at least one local shape descriptor and/or one global shape descriptor is calculated. A descriptor can be of the type: “Wave Kernel Signature” (WKS) or “Heat Kernel Signature” (HKS) or “Gaussian Curvature”. For example, the HKS descriptor is notably based on a solution of the heat equation, it is based on the Laplace – Beltrami operator. An advantage is to define shape descriptors from a spectral analysis of the region in the vicinity of a point. An advantage is the simplified use of standardized equations or operators whose numerical processing can be optimized.

According to one example, the calculation of the vector normal to the surface at the targeted point P ₁ also makes it possible to define a local shape descriptor.

According to another example, the control law is calculated in real time on the body in such a way that the focal length of the optic scans the entire depth range of the points of the mesh of a local zone considered, it that is to say between the most distant points of the optics and the closest points of the optics. Thus, the a priori knowledge of the local topology due to the knowledge of the body model makes it possible to control the control law in an optimal manner according to the inspected zone. In other words, from the cloud of points of a local area of the body model, the invention makes it possible to extract a range of focal length values making it possible to control the control law of the optics.

When at least one shape descriptor is calculated, the value of the latter makes it possible to generate a range of values of the focal length and to define a sampling of images taken from the variable focal length optics within this range. This sampling can be predefined or calculated in real time depending on the specific case.

According to an example when several shape descriptors are calculated locally, a shape coefficient is calculated in order to establish a simple correspondence rule between the shape coefficient and a range of variation of the focal length.

Generation of a trajectory

According to one example, the invention comprises guiding an image pickup device 20 along a trajectory or a surface to be traversed on the surface of an individual's body. The method therefore makes it possible to process a plurality of target points P ₁ along a trajectory. The trajectory can be planned initially on the surface of a calculated body model of an individual and then this trajectory can be recalculated or deformed during the guiding of the imaging device 20 according to the movements or movements of the individual. . In this case, the position in space of at least one point of the surface can be recalculated in order to control the path of movement of the imaging device.

In this example, the topological descriptors can be recalculated at the points of the servo path, for example if the movement involves a deformation of the surface of the body. According to another case, the values of the descriptors in first approximation can be kept at the points located on the trajectory having changed position.

According to one example, a shape detection algorithm can be implemented. This algorithm can be, for example, implemented to recognize postures {sitting, standing, arms raised, etc.} or even typical movements. According to an alternative, an operator can initiate a change of imaging modes imposing a recalculation of the body model and its position in space.

Control law

The device of the invention comprises an electronic controller making it possible to control the device with variable focal length 40. One advantage of the implementation of a control law Lc is to vary the focal distance at each acquisition of images in the vicinity of a point P ₁ of a trajectory 30 of the surface of the body while guiding the imaging device 20 along a trajectory to be traveled 32.

According to a first embodiment, the control law Lc can be planned in advance, that is to say before the image capture. This embodiment is possible when the body model of an individual is known and when the guiding trajectory of the imaging device 20 is planned. In this configuration, all the variables of the control law can be precomputed so that all the images to be acquired can be in order to form the final composite image. This embodiment can be combined with local modifications of the position of the body and therefore of the trajectory of the targeted points and therefore of the guiding trajectory of the imaging device 20. In this case, the control law does not is not modified since the configuration of the system {body – imaging device} remains unchanged.

According to a second embodiment, the control law Lc is calculated in real time from the data acquired in real time. Thus, a trajectory can be planned on the surface of the body, but the distance between the imaging device 20 and the surface of the body is not known a priori or even the local topology of the vicinity of the targeted points is not known. unknown and must be discovered in real time. When the control law Lc is controlled in real time, a real-time calculation is carried out to control the variable focal length optics and the position of the image pickup device 20 so that all of the images necessary for the reconstruction of a clear composite image is achieved.

The control law Lc can be configured so that the image pickup device 20 scans a set of points to acquire sharp images of a complete area of all or part of a surface of the body. In order to optimize the number of images acquired and the range of values of the focal length of the variable focal length optics, the control law Lc can take into account a topology datum in the vicinity of a point P1 from which it is desired to acquire images.

When the surface 12 of the body is relatively flat in the vicinity of the point P ₁ , the points in the vicinity of the point P1 are in a reduced range of focal length values. Within this range, the focal length of the optics is controlled to acquire sharp images close to the point P1. Typically, if the range of focal length values makes it possible to acquire sharp high definition images in a depth of field of 1mm, the number of images to be acquired to obtain a set of sharp images in the vicinity of the point P1 can be reduced. . The capture of images is limited to the variation of the focal length over a range of reduced focal length values. Such an area can correspond to that of a flat stomach of a fit person and not overweight.

When the surface 12 of the body is relatively curved in the vicinity of the point P ₁ , the points in the vicinity of the point P ₁ are in a wide range of focal length values. Within this wider range than the reduced range, the focal length of the optics is controlled to acquire sharp images close to the point P1. In order to acquire sharp images over a greater depth of field, for example 20 mm, then the range of values of the focal length is configured so as to be widened and the number of images acquired in this zone is greater than in the previous case. Such a zone may correspond to that of a calf or an arm.

Some areas may have even greater curvature such as the bent elbow, nose, ears or toes.

The control law Lc can also have an influence on the speed of movement of the imaging device 20 depending on the number of images to be acquired locally in the vicinity of a point P ₁ and more generally for all the points P ₁ of a trajectory. Finally, this control law Lc can be configured a priori or in real time according to the desired recovery rate of the images acquired from the surface of the human body. The coverage rate can be a percentage value defining the proportion of the image surface covered, for example on the edges of the images in order to reconstruct an overall image. The recovery rate can also be calculated so that it defines a statistic of proportion of sharp images at a given precision factor with a given error factor.

The recovery rate can also be defined step by step between two successive acquisitions. For example, the overlap rate can be a percentage of the proportion of a repeated image within an image based on a previously acquired image.

An advantage of this last configuration is to obtain an area covered with two acquired images obtained with different focal lengths. In this embodiment, it is possible to reconstruct a gradient to generate a reconstructed image which will be sharp over a larger dimension than each image acquired separately.

As the curvature of the area increases, image acquisition can be performed more quickly with sampling of a larger range of the focal length of the varifocal optics, the movement of the imaging device 20 can slow down and the overlap can be configured to be more important between two successive images.

Thus, according to an example, the control law Lc can be configured with the following variables:

• The movement speed V _d is selected from a speed range [V _{d min} ; V _{d max} ];
• The image overlap rate T _R is selected from a range of image overlap rates [T _min ; T _max ];
• The focal length of the varifocal optics F _V is selected from a range of focal lengths [F _min ; F _max ];
• The topological descriptor or the local shape coefficient calculated from several local topological descriptors Cf F _V is selected from a range of shape coefficients [C _min ; C _max ];
• The distance df between the variable focal length optics and the aiming point P ₁ of the surface 12 of the body is selected from a distance range [d _min ; d _max ].
• The acquisition speed or delay between two successive acquisitions T _A is selected from a range of durations [T _min ; T _max ].

In an example mode, the viewing angle and image size are considered predefined. However, according to other examples, the dimensions of images and the viewing angles may vary as parameters.

According to different embodiments, one or more of these variables are used to define or parameterize the control law Lc making it possible to acquire the images. The control law Lc can itself be a complex control law depending on several control laws Lc1, Lc2, etc… depending on the consideration of the input variables.

For example, for a point of the trajectory comprising a plurality of target points P ₁ , one can write for example that the focal length is defined by a first control law Lc1 defined by a relation of the type: Fv = Lc1(df, Cf) . And for a given trajectory, the delay between two acquisitions T _A and the displacement speed V _d of the imaging device 20 are controlled according to a second control law Lc2 defined by a relation of the type: {V _{d ;} T _A }= Lc2(T _R , Cf}.

Building a sharp image

According to one embodiment, the invention comprises a step of constructing a sharp image of all or part of the surface 12 of the body 10 from the plurality of images acquired of said surface 12. This image is a composite image formed from a plurality of portions of images acquired by the image pickup device 20.

In order to select the portions of images or the groups of pixels, or even a pixel of an image, the method of the invention comprises a step of comparing a sharpness criterion of each pixel or each group of pixels of a image with those of another image comprising an overlap of this or these pixel(s). When a pixel with sufficient sharpness is detected, for example when the value of the criterion is greater than a predefined threshold, then the pixel is retained in the composite image. When no pixel has sufficient sharpness, i.e. whose sharpness criterion exceeds a certain threshold, then the sharpest pixel is selected.

In order to compare the sharpness or the blurring of pixels of different images covering the same area, an image alignment algorithm can be used to calibrate the images to each other and more particularly the portions of images covering the same areas of the surface. from the body.

The construction of a sharp image includes a clipping of acquired images according to their sharpest portions while checking that the unretained clipped areas are included in other acquired images covering the removed part of an image. This control can be carried out automatically from the coverage rate of the acquired images and a marking of the covered areas.

The invention makes it possible to generate a sharpness map of each image of the stack of images in order to select the sharpest pixels of the stack of acquired images covering a given area. It is therefore possible to generate an image from the sharpness map(s) in which the sharpest pixels are selected.

According to one embodiment, a sharpness estimation step is performed to generate sharp areas and remove blurred areas. In order to achieve this step, a sharpness estimation algorithm can be implemented. Among the algorithms that can be used within the invention, different solutions can be implemented such as a calculation of the mean or the variance of a gradient. Another method can be implemented by means of automatic learning from a learning function. The learning function can be deep learning type learning. According to one example, a CNN convolutional neural network may be implemented. Such an algorithm makes it possible to generate sharpness maps of all the images in order to identify the sharpest pixels in a second step to generate a completely sharp 3D image or at least the sharpest possible. Such a deep learning algorithm, i.e. deep learning, can also be implemented to process the alignment of successively acquired images. When a neural network is used in this way, it is said to be functional in the present application insofar as it makes it possible to perform an intermediate function such as the generation of a sharpness map or a depth map or even to realign pictures.

The capture of successive images having a rate of overlap between them makes it possible to improve the estimation of the sharpness. Finally, the a priori knowledge on the one hand of the distance between the target point P ₁ and the variable focal length optics 20 and on the other hand of the focal length also makes it possible to carry out a good estimation of the sharpness since there are reference areas that can be defined as net.

When the sharpest zones are identified and the images cut out according to these zones, the method of the invention comprises a step of assembling the sharp zones to construct a sharp image of the surface of all or part of the body.

The stitching step can be seen as a step of aggregating image portions from different shots. Another way of carrying out an assembly is to carry out a fusion of the zones of covering by addressing each point by the sharpest pixels and by removing the pixels of these zones of covering more fuzzy.

According to one embodiment, the assembled images can be assembled in a two-dimensional space or in a three-dimensional space. Indeed, knowing the depth information either from a body model or from an estimate during image acquisition, a 3D image can be automatically reconstructed.

The 3D image can be reconstructed following assembly during processing applied to the final aggregated image.

According to another example, an end-to-end learning algorithm can be used to generate from a stack of acquired images, also called in English terminology “stack” of images, corresponding to the plurality of images collected, outputting a sharp image on the one hand and a depth map on the other. The learning of such a learning function, for example of the CNN type, can be carried out on the basis of images acquired by optics or images generated by a computer of a body or of a model of body and on the other hand a totally clear 3D image obtained from all the images. We then speak here of an end-to-end neural network to designate a neural network capable of directly generating a clear 3D image at any point.

The passage through the sharpness map of each of the images of the acquired image stack is then implicit in the implementation of such an end-to-end algorithm. According to one embodiment, this last algorithm can be trained on synthetic data.

There are at input the images of the stack of the zone in the vicinity close to the acquired point P ₁ and at the output a composite image of this same zone and the depth map. A network of the CNN convolutional neural network type can for example be implemented.

The set of learning data makes it possible to train the neural network to obtain a learning function configured or learned with weighting coefficients. In this case, the completely clean 3D image can be calculated from another algorithm in order to supervise the training of the neural network. Such learning can also lead to the generation of a depth map.

The neural network used, whether functional or end-to-end, can be configured in different ways.

According to one embodiment, the configuration of the CNN neural network may include:

• Convolutions or neural network layers comprising a plurality of multiplications of matrices comprising weighting coefficients obtained from a learning process;
• nonlinear operations.

According to one embodiment, the configuration of the CNN neural network includes as inputs images acquired by the optics.

The CNN neural network can include in its first layers convolutions then layers of fully connected neurons, called “fully connected layers” at the end of the model. In the latter case, they are neurons connected to all the neurons of the previous layer and connected to all those of the following layer.

The convolution layers can include a scan of an input matrix producing a series of matrix calculations. The other layers of the neural network generally include matrix calculations on the size of the input matrix.

According to one example, each convolution includes a matrix product between an input matrix and a weight matrix and the consideration of an additional bias.

Applying layered processing within the CNN neural network includes applying a series of matrix multiplications which are followed by a nonlinear function to produce an output of said layer. The succession of these operations defines the depth of the neural network.

According to an exemplary embodiment, the neural network is a multilayer perceptron, known by the acronym MLP and in English terminology by “multi-layers perceptron”. According to one example, the neural network can be a network equivalent to the MLP.

An advantage of the invention is also to reconstruct a relative depth map of all or part of the body. By “relative depth map”, we mean a map where each point of the image, for example each pixel, can be positioned vis-à-vis another point of the reconstructed image. One of the advantages of this characteristic is to make it possible to eliminate the effects of perspective or deformation of the images linked to the viewing angle or to the optics themselves. Another advantage of this method is to obtain an accurate estimation of the local geometry of the imaged surface. This allows to augment and refine the 3D body model when this latest body model is available as input to the system. This estimated depth information can have a resolution between 500um and 1cm. The input body model can then be refined to define a high resolution 3D body model.

Platforms

According to an exemplary embodiment, the image pickup device 20 is mounted on a mobile platform. According to a first embodiment, the mobile platform can be arranged at the end of a robotic arm mobile in space and comprising a plurality of degrees of freedom. According to a second embodiment, the platform is a drone controlled by a wireless remote control.

According to another embodiment, the platform is a manual gun comprising a distal end piece on which is arranged the image pickup device 20 and in particular the variable focal length optics.

According to another mode, the device is fixed and the patient is turned by means of a rotating platform or vice versa. The invention is intended for any type of dynamic mobile platform comprising a distal end piece on which the variable focal length optics are arranged.

Claims (21)

Method for acquiring images of a first surface (12) of a human body using an image pickup device (20) comprising variable focal length optics (40), said method comprising the steps of:

• acquisitions (ACQ) of a plurality of images using the image pickup device (20), said acquisitions being carried out by varying the focal distance (d _f ) of the variable focal length optics (40) according to a control law;
• construction of a sharp image of a portion of the surface (12) of the human body (10) from the plurality of acquired images of said surface (12),
• estimation of the depth map of the local geometry of the first surface (12).

Image acquisition method according to Claim 1, characterized in that the construction of an image of a portion of the surface (12) of the body (10) comprises:

• cropping the images of the plurality of images from estimated sharpness information to generate sharp areas and blurred areas;
• assembling the sharp areas of said images to construct a sharp image of the surface of the portion of the body.

Acquisition method according to any one of the preceding claims, characterized in that the method comprises a measurement or an estimation (MES) of a distance between the variable focal length optics and a first point (P ₁ ) of the surface of the body, the control law taking into account said measured or estimated distance between the variable focal length optics (40) and the first point (P ₁ ) to modify the focal length of the optics between at least two images acquired at the vicinity of the point (P ₁ ) of the surface of the body. Acquisition method according to the preceding claim, characterized in that the measurement or estimation (MES) of the distance between the variable focal length optics (40) and a first point (P₁) of the surface of the body is carried out:

• by a distance measurement per point and/or;
• by a distance measurement combined with known topological information of the body model of the body surface and/or;
• by known topological information of the body model of the surface of the body and position information of the optics with respect to the 3D model and/or;
• by a distance measurement combined with a measurement of the local geometry of the imaged area of the body surface and/or,
• by a measurement of the local geometry of the imaged zone of the surface of the body and information on the position of the optics with respect to the 3D model.

Acquisition method according to the preceding claim, characterized in that the calculation of the distance between the variable focal length optics (40) and a first point (P₁) of the surface of the body is realized either by a measurement or by an estimation:

• using a distance measuring means (42);
• from sharpness information of all the images acquired and from an algorithm for estimating the blur and/or the sharpness.

Acquisition method according to the preceding claim, said method further comprising the steps of:

• receiving (REC) a 3D model of at least a part of a body (10), said 3D model modeling at least a first surface (12) of said body (10);
• Recovery (RECUP) of local topological data from the neighborhood of at least the first point (P ₁ ), said local topological data being extracted from the 3D model or calculated from the 3D model;
• The acquisitions (ACQ) of a plurality of images being carried out by varying the focal distance (d _f ) of the variable focal length optics (40) according to a control law also taking into account the local topological data (D _TOPO ) of the neighborhood of the first point (P1) of the first surface (12).

Image acquisition method according to the preceding claim, said method further comprising the steps of:

• Generation (GEN ₁ ) of a treatment trajectory (30) on the surface (12) of the 3D model by means of a computer, said treatment trajectory (30) comprising at least two points on the surface (12) of the model 3D;
• Construction of an image of a portion of the surface of the body in the vicinity of the treatment trajectory (30).

Image acquisition method according to any one of the preceding claims, characterized in that the range of values of the focal length of the variable focal length optics (40) is defined as a function of the local topological data of the 3D model in the vicinity of the first point (P ₁ ) of the processing trajectory (30). Image acquisition method according to any one of the preceding claims, characterized in that the acquired images are processed so as to generate sharpness gradient information between the pixels of the same image, the said gradients being exploited so as to generating alignment instructions between the successive acquired images, said realigned images to produce a set of image portions covering the same areas of pixels, the sharpness or blur information or of these portions being compared to select the most sharp, said selected pixels being merged to construct a composite 2D or 3D image. Image acquisition method according to any one of the preceding claims, characterized in that:

• the construction of the image of a portion of the surface (12) of the body (10) comprising the cutting and the assembly of the images is carried out using a learning algorithm, of the neural network type, trained so as to produce a sharp 3D image as output from a plurality of images acquired as input and/or,
• the learning algorithm estimating the sharpness or the blur of each point of the images of the plurality of images, said learning algorithm being implemented by a first learning function (CNN ₁ ), for example an automatic learning method.

Image acquisition method according to Claim 10, characterized in that the image processing algorithm comprises a selection and a division of each group of pixels of each image having a sharpness greater than a predefined threshold or a sharpness greater than the pixel sharpness of at least one other image of the same area. Image acquisition method according to any one of the preceding claims, characterized in that it comprises a step of estimating the value of the distance between the variable focal length optics (40) and at least one first point (P ₁ ) of the processing path (30) from an estimate of the sharpness of points of the images of the plurality of images, said estimate being made from an image processing algorithm. Image acquisition method according to the preceding claim, characterized in that it comprises an estimation of the relative depth between the pixels of the same image and a calibration of the measurement of at least one pixel to generate depth information absolute of at least one pixel. Image acquisition method according to the preceding claim, characterized in that it comprises, between the generation steps (GEN₁) of a processing trajectory (30) and distance measurement (MES), the steps of:

• Calculation (CALC) of a guide path (32) of the variable focal length optics (40);
• Generation (GEN ₂ ) of a kinematics of the guide element (50) so that a point of the variable focal length optics (40) is slaved on the guide path (32);
• Activation (ACT) of the kinematics of the guide element (50) to traverse the guide trajectory (32), said guide trajectory (32) being calculated in real time from the 3D surface model.

Image pickup device (20) characterized in that it includes variable focal length optics (40) for acquiring a plurality of images of the surface of an individual's body with the aid of the image (20), said acquisitions being made by varying the focal distance (d _f ) of the variable focal length optics (40) according to a control law, said device (20) comprising at least one computer configured to put implements an image processing algorithm to generate a sharp 3D image. Image pick-up device (20) according to Claim 15, characterized in that it comprises means for measuring a distance between the variable focal length optics and a first point (P ₁ ) on the surface of the body, said acquisitions being carried out by varying the focal distance (d _f ) of the variable focal length optics (40) according to a control law taking into account the distance between the variable focal length optics (40) and the first point (P ₁ ). Image pickup device (20) according to claim 15, characterized in that it further comprises an interface for receiving a 3D model of at least one part of a human body (10), said model modeling at least one first surface (12) of the body and a mobile platform, said one or more computers being configured to:

• Retrieving local topological data from the neighborhood of at least the first point (P ₁ ), said local topological data being extracted from the 3D model or calculated from the 3D model;
• Control the acquisition of a plurality of images by the image pickup device (20), said acquisitions being carried out by varying the focal distance (d _f ) of the variable focal length optics (40) according to a law command taking into account the distance between the variable focal length optics (40) and the first point (P ₁ ), and local topological data (D _TOPO ) of the vicinity of the first point (P ₁ ) of the first surface (12) .

Image pickup device (20) according to any one of Claims 15 to 17, characterized in that it comprises a guide element (50) comprising the variable focal length optics (40), and in that the the computer(s) is/are configured for:

• generating a processing path (30) on the surface of the 3D model, said processing path (30) comprising at least two points on the surface (12) of the 3D model;
• Calculate a guide trajectory (32) of the variable focal length optics (40);
• Generate a kinematics of the guide element (50) so that a point of the variable focal length optics (40) is slaved to the guide path (32);
• Activate a kinematics of the guide element (50) to traverse the guide trajectory (32), said guide trajectory (32) being calculated in real time from the 3D surface model.

Image pickup device (20) according to claim 18 characterized in that the variable focal length optics (40) is a liquid lens, said liquid lens being arranged on a guide element (50) configured to move said lens. Mobile platform comprising an image pickup device (20) according to any one of claims 15 to 19, said platform comprising a command interface for receiving guidance instructions calculated in particular from the body model calculated in real time. Mobile platform according to Claim 20, characterized in that it is:

• a robot arm having a plurality of degrees of freedom;
• A drone comprising optics forming an image pickup device (20);
• A guide rail on which the imaging device (20) is movable or:
• A reception booth for a human subject comprising means for guiding an image pickup device (20),
• A platform comprising a fixed part accommodating the image pickup device (20) a mobile part causing an individual to move vis-à-vis the fixed part,
• A gun having a tip holding the imaging device (20).