Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/50—Depth or shape recovery
  • G06T7/55—Depth or shape recovery from multiple images
  • G06T7/571—Depth or shape recovery from multiple images from focus
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/70—Determining position or orientation of objects or cameras
  • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
  • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10016—Video; Image sequence
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10028—Range image; Depth image; 3D point clouds
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10141—Special mode during image acquisition
  • G06T2207/10148—Varying focus
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20021—Dividing image into blocks, subimages or windows
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20081—Training; Learning
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20084—Artificial neural networks [ANN]
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20212—Image combination
  • G06T2207/20221—Image fusion; Image merging
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30004—Biomedical image processing

Definitions

• the field of the invention relates to that of optical devices with variable focal length making it possible to acquire images of a body.
• the field of the invention relates to the field of liquid lens optical devices making it possible to acquire high-resolution images of a human body and to exploit these images.
• the field of the invention finds particular application in guiding a robot arm at the end of which an optical device is configured to acquire images of the skin of an individual.
• optics with a shallow depth of field are generally used.
• the quality of the image is essential for the a posteriori analysis of the surface of the skin.
• reduced-field optics are favored to highlight the definition of the colors and roughness of a surface of the skin.
• the invention relates to a method for acquiring images of a first surface of a human body using an image pickup device comprising variable focal length optics, said method comprising the steps of:
• One advantage is to make it possible to reconstruct a clear image of a human body in order to make it possible, for example, to categorize dermatological patterns such as moles.
• the construction of an image of a portion of the surface of the body comprises: cutting out the images of the plurality of images from estimated sharpness information to generate sharp areas and areas blurry;
• One advantage is to allow reconstruction of a complex surface of the human body by making the most of the optics and the images acquired over the entire area considered.
• the method comprises a measurement or an estimation of a distance between the variable focal length optics and a first point on the surface of the body, the control law taking into account said distance measured or estimated between the optics with variable focal length and the first point to modify the focal length of the optics between at least two images acquired in the vicinity of the point on the surface of the body.
• One advantage is to increase the performance of the control law by reducing the range of values of the variation of the focal length. Furthermore, a distance measurement also allows a better estimation of the sharpness of the images.
• the measurement or estimation of the distance between the variable focal length optics and a first point on the surface of the body is carried out:
• ⁇ by a distance measurement combined with known topological information of the body model of the body surface and/or; ⁇ by known topological information of the body model of the surface of the body and position information of the optics with respect to the 3D model and/or;
• ⁇ by a measurement of distance combined with a measurement of the local geometry of the imaged zone of the surface of the body and/or, ⁇ by a measurement of the local geometry of the imaged zone of the surface of the body and position information of the optics compared to the 3D model.
• An advantage is to obtain a better estimate by combining different estimates or measurements of the distance between the varifocal optics and a first point on the surface of the body so as to increase the performance and/or the quality of the result.
• the measurement, calculation or estimation of the distance between the variable focal length optics and a first point on the surface of the body is carried out either by a measurement or by an estimation: "at the using a means of measuring the distance;
• the method comprises the steps of: ⁇ Reception of a 3D model of at least a part of a body, said 3D model modeling at least a first surface of said body;
• the acquisitions of a plurality of images being carried out by varying the focal length of the variable focal length optics according to a control law also taking into account the local topological data of the vicinity of the first point of the first surface.
• An advantage is to know a topological information of the body which allows on the one hand to optimize the control law in calculation and in time and on the other hand to effectively guide a mobile device if necessary.
• the first surface of the 3D model is a surface of a human body.
• the first surface of the 3D model is a cloud of points, in particular a connected graph oriented in space.
• An advantage is to have easily exploitable metrics to address and superimpose the sharp 3D images reconstructed on the body. Another advantage is to allow easy planning of guidance trajectories on the surface of the body. Another advantage is to make it possible to know the local depths of field of a region of the surface with precision to control the control law of the focal length of the optics.
• the method comprises the steps of:
• An advantage is to automatically generate a complete image of a human body by programming a mobile device such as a robot arm in advance.
• the range of values of the focal length of the variable focal length optics is defined according to the local topological data of the 3D model in the vicinity of the first point of the processing trajectory.
• the acquired images are processed so as to generate sharpness gradient information between the pixels of the same image, said gradients being exploited so as to generate alignment instructions between the successive acquired images, said realigned images to produce a set of image portions covering the same pixel areas, the sharpness or blur information or of these portions being compared to select the sharpest pixels, said selected pixels being merged to construct a 2D image or Composite 3D.
• An advantage is to obtain a sharpness map of each image in order to collect and select the set of sharpest pixels of an image.
• the construction of the image of a portion of the surface of the body comprising the cutting and the assembly of the images is carried out using a learning algorithm, of the neural network type, trained so as to producing a sharp 3D output image from a plurality of input acquired images.
• One advantage is to generate sharp images quickly thanks to an artificial intelligence algorithm.
• the image processing algorithm estimating the sharpness or the blur of each point of the images of the plurality of images is implemented by a first learning function, for example an automatic learning method.
• the image processing algorithm comprises a selection and a cutting of each group of pixels of each image having a sharpness greater than a predefined threshold or a sharpness greater than the sharpness of pixels by at least one another image of the same area.
• the method comprises a step of estimating the value of the distance between the variable focal length optics and at least a first point of the processing trajectory from an estimate the sharpness of points of the images of the plurality of images, said estimation being carried out from an image processing algorithm.
• An advantage is to make it possible to generate a depth map by software means that do not require additional equipment.
• the estimation step carried out by software means makes it possible, for example, to refine the measurement of a device.
• the method comprises an estimation of the relative depth between the pixels of the same image and a calibration of the measurement of at least one pixel to generate absolute depth information of at least one pixel.
• variable focal length optics are arranged on a guide element configured to move a lens.
• the method comprises, between the steps of generating a processing trajectory and measuring the distance, the steps of:
• the method includes a step of correcting the guidance trajectory from the real-time reconstruction of the 3D model.
• the invention relates to an image pickup device characterized in that it comprises variable focal length optics for acquiring a plurality of images of the surface of an individual's body using of the image pickup device, said acquisitions being made by varying the focal distance of the variable focal length optics according to a control law, said device comprising at least one computer configured to implement an image processing algorithm to generate a sharp 3D image.
• the image pickup device comprises means for measuring a distance between the variable focal length optics and a first point on the surface of the body, said acquisitions being carried out by varying the focal length of the variable-focal lens according to a control law taking into account the distance between the variable-focal lens and the first point.
• the image pickup device further comprises an interface for receiving a 3D model of at least one part of a human body, said model modeling at least a first surface of the body and a mobile platform, said computer(s) being configured for:
• the image pickup device comprises a device for measuring the distance between the variable focal length optics and the first point configured to measure in real time a distance between the variable focal length optics and a first point on the surface of the body and means for measuring the orientation of the optics to deduce viewing angle information, said orientation and said distance being used by a computer of the device to correct or transform the acquired image.
• the image pickup device comprises a guide element comprising variable focal length optics, the computer(s) being configured for:
• ⁇ Generate a processing trajectory on the surface of the 3D model, said processing trajectory comprising at least two points on the surface of the 3D model; ⁇ Calculate a guide trajectory for the varifocal optics
• ⁇ Activate a kinematics of the guiding element to traverse the guiding trajectory, said guiding trajectory being calculated in real time from the 3D surface model.
• variable focal length optic is a liquid lens.
• the invention relates to a mobile platform comprising an image capture device of the invention, said platform comprising a command interface for receiving guidance instructions calculated in particular from the body model calculated in real time .
• the mobile platform is:
• - a drone comprising optics forming an image capture device; - a guide rail on which the imaging device is movable or:
• a reception booth for a human subject comprising means for guiding an image-taking device
• a platform comprising a fixed part accommodating the image capture device a mobile part driving an individual in motion vis-à-vis the fixed part.
• the invention relates to a gun-type device comprising a tip holding the image pickup device according to the invention.
• Figure 1 an example of a guidance device configured to acquire images of the surface of a body and to reconstruct a sharp image of at least a portion of the surface of the body;
• Figure 2 an example of an image pickup device of the invention guided along a guide trajectory to acquire images in the vicinity of a trajectory on the surface of a body;
• Figure 3 an example of a guiding device configured to acquire images of the surface of a human body such as a patient;
• Figure 4 an example of a flowchart representing the steps of the image acquisition process for their processing in order to reconstruct a clear image of all or part of a surface of a body.
• FIG. 1 illustrates an image acquisition device 20 of the invention comprising variable focal length optics 40.
• An image acquisition device 20 also called an image pickup device, is configured to acquire a plurality of images in the vicinity of at least one point located on the surface of a body.
• the invention finds particular application in dermatology and the analysis of images of an individual's skin. However, the invention is not limited to this application and can be implemented in other fields. Finally, the invention is particularly advantageous when image acquisition devices 20 comprise optics suitable for macrophotography, or even microphotography, such as optics having a large aperture.
• One objective of the invention is to reconstruct the photograph of the surface of a body so that each portion of the reconstructed three-dimensional image is as sharp as possible.
• body will mean the body of a human.
• the invention is not limited to this embodiment, the invention can be implemented for any type of body having a three-dimensional surface of which it is sought to represent at least a portion as faithfully as possible.
• the acquisition parameters including the number of images, the dimensions of the images and the resolution of the images can be predefined or defined according to a displacement configuration of the acquisition device 20, for example when the latter is arranged on a mobile platform moving around the body.
• the invention advantageously makes it possible to acquire a set of images in the vicinity of a point by varying the focal distance of the acquisition optics according to a control law taking into account the distance between the variable focal length optics and the target point.
• the operation can thus be repeated on a plurality of points of a surface of a body in order to acquire several sequences each comprising a plurality of images in the vicinity of a plurality of points.
• This method can be carried out continuously on a set of points on a surface to cover all or part of the surface of a body or this method can be carried out along a trajectory on the surface of the body in order to acquire images in the vicinity of points defining the trajectory.
• step by step the entire surface of a body can be treated.
• the expression "image in the vicinity of a point” must be understood as an image of a body surface including said point.
• variable focal length optics 40 is advantageously a variable focal length lens, also called an adaptive lens.
• the lens used is a lens deformable by piezoelectric actuation.
• the lens is a liquid crystal lens. The latter use a change in the optical index of the liquid crystals in order to modify the focal length of the lens.
• another embodiment can be implemented from a liquid lens whose geometry is controllable by micro-fluidic parameters or mechanical parameters such as parameters defining the geometry of a membrane retaining a liquid or even a combination of the two.
• a liquid lens comprises a circular membrane made of electroactive polymer. The central part of the membrane forms the lens and the peripheral part is surmounted by an annular electrode. By applying an electric voltage between this electrode and the substrate, an electrostatic pressure is created, which tends to move the fluid towards the central part, inflating the membrane, and thus modifying the curvature of the lens.
• These parameters can be adjusted, configured or controlled from a digital setpoint from a sensor or a computer or any other electronic device.
• variable focal length optic comprises a magnetic actuator making it possible to move an optical element, such as a lens, to modify the focal length.
• the optical element moves for example in a tube, the position in the tube defining a focal distance specific to the optical configuration of the lens.
• the magnetic actuator can be controlled by an electromagnetic field generated from a dedicated component.
• the field can be adjusted, configured or controlled from a digital control setpoint. This digital setpoint can be generated by a control law from a computer or a sensor or any other electronic equipment.
• variable focal length optic comprises a mechanical actuator making it possible to move an optical element, such as a lens, to modify the focal length.
• the actuator can be controlled by an electronic component making it possible to receive a digital instruction and to generate a mechanical instruction.
• this digital setpoint can also be generated by a control law from a computer or a sensor or any other electronic equipment.
• variable focal length optics is to overcome the shortcomings of macrophotography which acquires images with a relatively shallow depth of field which remains incompatible with the acquisition of images in a portion of curved surface for example on the calf or the arm.
• Another advantage is to have very low response times, for example a response time of less than 25ms.
• variable focal length optics for example in addition to optics with a larger aperture, makes it possible to acquire sufficient precision of the images of a surface of a body so as to reconstitute a clear global image by an image reconstruction algorithm.
• the invention makes it possible to automatically vary the focal length of an optic according to a control law in order to reconstitute a clear image of the vicinity of the targeted point.
• the controller controls the numerical value of the focal length of the lens.
• the numerical value makes it possible to vary the shape of the lens.
• the control of the liquid lens is carried out by means of a continuous signal such as a trigonometric function, for example a sine function or a linear function, for example a function whose shape is in -saw.
• a continuous signal such as a trigonometric function, for example a sine function or a linear function, for example a function whose shape is in -saw.
• Such control makes it possible to optimize the stabilization of the medium constituting the liquid lens.
• Such a control signal makes it possible to minimize the nonlinear effects and to reduce the relaxation times or the transient phases of modification of the medium of the liquid lens.
• Such a variation of the focal length of the lens is moreover consistent with the evolution of the curvature of the zone in the vicinity of the target point Pi.
• the image acquisition by the liquid lens 20 is carried out according to a predefined sampling over the entire range of focal lengths addressable by the lens.
• the number of images acquired is constant according to a speed of movement of the imaging device.
• the acquired image is a color image in the visible spectrum.
• the frequency range of the acquired image can in this case be the visible, that is to say for wavelengths between 380 nm and 700 nm. According to modes where ranges extend on either side of the visible spectrum, it is possible to acquire images in a broader spectrum from dedicated equipment.
• the acquired images are acquired in an infrared range, that is to say for wavelengths comprised between 700 nm and 1000 nm.
• the acquired images are acquired in an ultraviolet range, that is to say for wavelengths comprised between 10 nm and 380 nm.
• the image is a multispectral image.
• spectral imaging equipment can be used.
• a configuration of such equipment with a controllable variable focal length can be used in the context of the invention.
• the equipment used for the acquisition can be configured to acquire images whose spectrum is notably distributed over several spaced frequency bands. An interest is to obtain an image over a wider band of frequencies including for example infrared, near infrared and ultraviolet.
• the acquired image is a hyperspectral image.
• an imaging spectrometer also called an imaging spectrometer or an imager associated with a spectrometer, makes it possible to generate a hyperspectral image.
• the equipment used for the acquisition can be configured to acquire images whose spectrum is notably distributed over several contiguous frequency bands.
• the equipment used can be configured with a controllable variable focal length. An interest is to obtain an image over a wider band of frequencies with a high degree of resolution including for example infrared, near infrared and ultraviolet.
• the equipment is a chemical imager or a chemical imaging device configured to create an image from an acquired frequency spectrum such as a spectrometer and spatial and temporal information.
• the image pickup device 20 includes means for measuring the distance between the variable focal length optical device 40 and a first point Pi on the surface of the body.
• This means for measuring the distance can be an active physical device such as an emitted laser placed close to the variable focal length optics 40.
• the emission of the laser beam makes it possible to measure the distance to the center of the acquired image.
• Other devices for measuring the distance between a point on the surface of the body and the variable focal length optics 20 can be used, for example a radioelectric rangefinder, an optical rangefinder or even an acoustic rangefinder.
• Other examples of distance measuring devices can be used in the context of the invention, such as a Lidar, a Radar, a Sonar or even a stadimetric rangefinder.
• the invention can implement a single-point measurement device such as a laser or a multipoint measurement system such as a laser, a so-called “matrix Time-of-Flight” device. , a lidar, an active/passive stereoscopic device, or even a device comprising a projection of structured light.
• a single-point measurement device such as a laser or a multipoint measurement system such as a laser, a so-called “matrix Time-of-Flight” device.
• a lidar an active/passive stereoscopic device, or even a device comprising a projection of structured light.
• the means for measuring the distance between the variable focal length optics 40 and a first point Pi of the surface of the body is a software means implementing a computer processing the images acquired in order to extract sharpness or blur information to reconstruct from this data information characterizing the distance between a point Pi of the surface and the optics.
• An exemplary embodiment may be an algorithm detecting the sharpest pixel(s) of an image to deduce therefrom information on the distance between a point Pi of the surface and the optics, knowing moreover the focal length of the optics used .
• the gradients between the sharpest pixels and the other pixels can also be exploited in order to consolidate the measurement of the distance between a point Pi on the surface and the optics.
• the focal length information is then used to process the evaluation of the distance between the variable focal length optics and a point Pi on the surface of the body.
• the set of shades of sharpness or blur between the pixels can then be used to consolidate the calculation of the distance between a point Pi on the surface and the optics or to define a local depth map in the vicinity of the aiming point Pi or a point whose sharpness is considered the highest.
• an algorithm can be implemented by a learning function, of the function type driven by automatic learning, such as a deep learning algorithm.
• a CNN-like convolutional neural network can be used.
• the learning function makes it possible in particular to detect the pixel(s) assumed to be the sharpest in the image and to determine a distance between a point Pi on the surface and the optics.
• a sharpness map also called “defocus map” in English terminology, can be established for each of the images in the image stack by varying the focal length of the lens.
• Such a sharpness map makes it possible to encode blur or sharpness information, for example on a standardized scale of 0 to 1.
• the sharpness map of an area is constructed by considering the sharpest pixels of each image.
• the distance measured by a sensor can be combined with another measurement of the distance obtained either by another sensor or by another method making it possible to evaluate the distance.
• a first method consists in evaluating the distance between the variable focal length optics 40 and a point Pi of the surface of the body from the acquired images and by an algorithm for estimating the sharpness of different points of the image in order to reconstruct distance information between at least one point on the surface and the optics.
• this method the differences in sharpness of different points of different images having been acquired with different focal lengths make it possible to obtain an estimate of the distance of each of these points with respect to the optics.
• This estimate may or may not be combined with another method or data from a sensor. This estimate can be used to define an input of a control law Le of the focal length of the optics 40.
• the invention comprises a component making it possible to generate a body model, in particular of the three-dimensional surface of this body.
• This surface can be a surface defined in space or it can be defined by a cloud of points or a mesh.
• the point cloud can be advantageously connected and oriented in space, we speak of spatial orientation in space. When the cloud of points is oriented, it is possible to identify the surface of the body model in space, for example within a predetermined reference frame Ro.
• this control law can be calculated in real time without having to plan a trajectory of the optics beforehand.
• This embodiment is particularly advantageous in the case of optics embedded in a mobile device of the pistol type, that is to say removable according to the movements of an operator's hand for example.
• this control law can be calculated for example by planning a preliminary trajectory of the optics.
• This embodiment is particularly interesting in the case of an optics embedded in a robotic device of the type an articulated robot arm and more generally a device embedded in any removable platform.
• the invention When the invention implements an automatic generation of the body model of an individual, the latter can be generated from an optical device capturing partial views or complete views of the body of the individual.
• the partial views can then be used to generate a body model for example from an algorithm implemented by a learning function, such as a function that has been trained by automatic learning.
• a learning function such as a function that has been trained by automatic learning.
• An example is a convolutional neural network.
• Such a function makes it possible to generate in real time 3D body models of an individual from acquired partial views.
• Such a possibility makes it possible to calculate topological data in the vicinity of a target point Pi of the surface of the body during the scanning of a trajectory by the imaging device 20.
• the knowledge of the topology can come from different devices or methods of calculation of at least one local topological descriptor at the targeted point Pi.
• the descriptor comes from a calculation carried out in real time from an acquired image.
• This calculation can be performed with a second optical device configured with an aperture, a depth of field or a focal length allowing local analysis of the depth map in the vicinity of the point Pi.
• the second optical device can comprise a optics in an infrared range.
• a device comprising a projection of images whose deformation is calculated makes it possible to calculate a depth map.
• Another example is the use of structured light.
• the descriptor comes from a real-time calculation from an image extracted from a surface model of the human body which is oriented and positioned in space.
• a body model can for example be generated in real time from partial or total knowledge of a patient's body.
• a priori knowledge of the trajectory that has been planned and of the patient's body model for which local images of the skin are to be obtained and finally knowledge of the imaging device 20 and its arrangement vis-à-vis the bodies make it possible to generate a set of local descriptors a priori before the image acquisition process in the vicinity of the targeted points Pi.
• Such a technique makes it possible to configure the control law Le of the variable focal length optics without depending on a constraint real time. Moreover, this technique saves calculation time in real time and makes it possible to optimize the best ratio of images necessary for the reconstruction of a clear image of all or part of the human body for which a clear image is desired.
• At least one local shape descriptor and/or one global shape descriptor is calculated.
• a descriptor can be of the type: “Wave Kernel Signature” (WKS) or “Heat Kernel Signature” (HKS) or “Gaussian Curvature”.
• WKS Wave Kernel Signature
• HKS Heat Kernel Signature
• Gaussian Curvature a descriptor that is characterized by the Laplace-Beltrami operator.
• An advantage is to define descriptors shape from a spectral analysis of the region in the vicinity of a point.
• An advantage is the simplified use of standardized equations or operators whose numerical processing can be optimized.
• the calculation of the vector normal to the surface at the targeted point Pi also makes it possible to define a local shape descriptor.
• the control law is calculated in real time on the body in such a way that the focal length of the optic scans the entire depth range of the points of the mesh of a local zone considered, it that is to say between the most distant points of the optics and the closest points of the optics.
• the a priori knowledge of the local topology due to the knowledge of the body model makes it possible to control the control law in an optimal manner according to the inspected zone.
• the invention makes it possible to extract a range of focal length values making it possible to control the control law of the optics.
• the value of the latter makes it possible to generate a range of values of the focal length and to define a sampling of images taken from the variable focal length optics within this range.
• This sampling can be predefined or calculated in real time depending on the specific case.
• a shape coefficient is calculated in order to establish a simple correspondence rule between the shape coefficient and a range of variation of the focal length.
• the invention comprises guiding an image pickup device 20 along a trajectory or a surface to be traversed on the surface of an individual's body.
• the method therefore makes it possible to process a plurality of target points Pi along a trajectory.
• the trajectory can be planned initially on the surface of a calculated body model of an individual and then this trajectory can be recalculated or deformed during the guiding of the imaging device 20 according to the movements or movements of the individual. .
• the position in space of at least one point of the surface can be recalculated in order to control the path of movement of the imaging device.
• the topological descriptors can be recalculated at the points of the servo path, for example if the movement involves a deformation of the surface of the body.
• the values of the descriptors in first approximation can be kept at the points located on the trajectory having changed position.
• a shape detection algorithm can be implemented. This algorithm can be, for example, implemented to recognize postures ⁇ sitting, standing, arms raised, etc. ⁇ or even typical movements. According to an alternative, an operator can initiate a change of imaging modes imposing a recalculation of the body model and its position in space.
• the optical device is moved at a plurality of points forming a trajectory of the optical device.
• the optical device can be moved along a predefined trajectory in an automated way, for example by a robotic arm.
• the optical device is guided directly by a user.
• the optical device can comprise a handle to be grabbed by the hand of the user.
• the trajectory should be understood as a relative trajectory with respect to the body surface.
• the displacement of the optical device along its trajectory is continuous, that is to say that the displacement relative to the human body does not mark a pause.
• the relative displacement of the optical device with respect to the human body is never zero along said trajectory.
• two successive images are taken from two different positions. The two acquired images sharing an overlap rate overlap so as to share a common image portion as detailed later.
• the movement of the optical device along a predetermined trajectory comprises a step of acquiring a point cloud representing the surface of the subject's skin, a step of continuously updating said point cloud in according to changes in the three-dimensional geometry of said surface due to the action of the muscles or a movement of the body and optionally, the continuous updating of said trajectory according to the updated point cloud.
• the device of the invention comprises an electronic controller making it possible to control the device with variable focal length 40.
• One advantage of the implementation of a control law Le is to vary the focal distance at each acquisition of images in the vicinity of a point Pi of a trajectory 30 of the surface of the body while guiding the imaging device 20 along a trajectory to be traveled 32.
• the control law Le can be planned in advance, that is to say before the images are taken.
• This embodiment is possible when the body model of an individual is known and when the guiding trajectory of the imaging device 20 is planned.
• all the variables of the control law can be precomputed so that all the images to be acquired can be in order to form the final composite image.
• This embodiment can be combined with local modifications of the position of the body and therefore of the trajectory of the targeted points and therefore of the guiding trajectory of the imaging device 20. In this case, the control law does not is not modified since the configuration of the system ⁇ body - imaging device ⁇ remains unchanged.
• the control law Le is calculated in real time from the data acquired in real time.
• a trajectory can be planned on the surface of the body, but the distance between the imaging device 20 and the surface of the body is not known a priori or even the local topology of the vicinity of the targeted points is not known. unknown and must be discovered in real time.
• a real-time calculation is carried out to control the variable focal length optics and the position of the image pickup device 20 so that all of the images necessary for the reconstruction of a clear composite image is achieved.
• the control law Le can be configured so that the image pickup device 20 scans a set of points to acquire sharp images of a complete zone of all or part of a surface of the body. In order to optimize the number of images acquired and the range of values of the focal length of the varifocal optics, the control law Le can take into counts a topology datum in the vicinity of a point P1 of which it is desired to acquire images.
• the points in the vicinity of the point P1 are in a reduced range of values of the focal length.
• the focal length of the optics is controlled to acquire sharp images near the point P1.
• the range of values of the focal length makes it possible to acquire sharp high definition images in a depth of field of 1 mm
• the number of images to be acquired to obtain a set of sharp images in the vicinity of the point P1 can be reduced.
• the capture of images is limited to the variation of the focal length over a range of reduced focal length values. Such an area can correspond to that of a flat stomach of a fit and not overweight person.
• the points in the vicinity of the point Pi are in a wide range of focal length values.
• the focal length of the optics is controlled to acquire sharp images near point P1.
• the range of values of the focal length is configured so as to be widened and the number of images acquired in this zone is greater than in the previous case.
• Such an area can correspond to that of a calf or an arm.
• Some areas may have even greater curvature such as the bent elbow, nose, ears or toes.
• the control law Le can also have an influence on the speed of movement of the imaging device 20 depending on the number of images to be acquired locally in the vicinity of a point Pi and more generally for all the points Pi d a trajectory.
• this control law Le can be configured a priori or in real time according to the desired recovery rate of the images acquired from the surface of the human body.
• the coverage rate can be a percentage value defining the proportion of the image surface covered, for example on the edges of the images in order to reconstruct an overall image.
• the recovery rate can also be calculated so that it defines a statistic of proportion of sharp images at a given precision factor with a given error factor.
• the recovery rate can also be defined step by step between two successive acquisitions.
• the overlap rate can be a percentage of the proportion of a repeated image within an image as a function of a previously acquired image.
• An advantage of this last configuration is to obtain an area covered with two acquired images obtained with different focal lengths.
• overlap rate between two successive images is thus meant that two successive images partially overlap in a common portion, that is to say that the common portions of two successive images can be superimposed.
• the purpose of this overlap rate will thus be to use two different images to form a single continuous image comprising the common portions and the non-common portions of said acquired images.
• Two successive images sharing an overlap rate can be acquired with different focal lengths of the optical device.
• the optical device is moved along the trajectory, to a first position and then to a second position.
• the control law triggers the acquisition of at least one image of the point P1 from the first position then of at least one image from the second position.
• the first position and the second position are sufficiently close to each other to allow partial overlap of the successive images acquired at these two positions.
• the control law generates the acquisition of a first group of images from the position then of a second group of images from the second position.
• the images of the first group and the images of the second group partially overlap in a common portion as previously described.
• the first group of images and the second group of images each include a plurality of images acquired with different focal lengths.
• An advantage is to obtain a first multifocal image and a second multifocal image from respectively the first group of images and the second group of images.
• a first image is acquired from the first position and a second image is acquired from the second position.
• the first and the second image are then acquired with different focal lengths.
• the first image and the second image share an overlap rate.
• the movement of the relative optical device is continuous. Continuous motion is understood to mean a motion that does not reach zero speed. In this case, each image is acquired from a different position.
• the recovery rate is strictly less than 100%. In other words, the two successive images only partially overlap and each includes a common portion, but also a portion not included in the other image. Preferably, the recovery rate is strictly less than 99%.
• the recovery rate is greater than 75% and strictly less than 100% or 99%. At least 75% of the image area will be shared with at least one other image.
• the advantage of such an overlap rate is, for each point of the body surface, to acquire a plurality of images.
• the advantage is to be able to acquire a plurality of images acquired with different focal lengths, increasing the chances of having at least a sharp portion of said point for the construction of a sharp image from the plurality of images acquired from said surface 12 as described below.
• a point on the surface of the subject's body is captured by a plurality of acquired images, even when the optical device was in motion between the taking of successive acquired images.
• the acquisition focal lengths of two successive images having an overlap rate are different from each other.
• control law Le can be configured with the following variables:
• the movement speed V d is selected from a speed range [Vdmin; Vmax ];
• the image overlap rate TR is selected from a range of image overlap rates [Tmin; Tmax];
• the focal length of the varifocal optics Fv is selected from a range of focal lengths [Fmin; F ma x];
• the topological descriptor or the local shape coefficient calculated from several local topological descriptors Cf Fv is selected from a range of shape coefficients [Cmin Cmax]
• the distance df between the variable focal length optics and the aiming point Pi of the surface 12 of the body is selected from a distance range [dmin; d my x].
• the acquisition speed or delay between two successive TA acquisitions is selected from a range of durations
• the viewing angle and image size are considered predefined.
• the dimensions of images and the viewing angles may vary as parameters.
• control law Le allowing the images to be acquired.
• the control law Le can itself be a complex control law depending on several control laws Lc1, Lc2, etc. depending on how input variables are taken into account.
• the platform transporting the optical device is moved along a trajectory at a speed of movement Vd comprised between 4 cm/s and 30 cm/s.
• the optical device acquires images of the structure of the subject's skin at a frequency of between 5 images per second and 200 images per second (i.e. a delay between two acquisitions of between 0.2 s and 5 ms) while moving the optics.
• the control law varies, between two successive acquisitions, the focal length of the variable focal length optics.
• the displacement of the optics coupled with the acquisition speed and the variation of focal length between two successive acquisitions advantageously allow the acquisition of successive images sharing a common portion at different focal lengths.
• the common portion can thus be composed from the pixels of different images acquired according to the sharpness index of each pixel for each image acquired.
• the invention comprises a step of constructing a sharp image of all or part of the surface 12 of the body 10 from the plurality of images acquired of said surface 12.
• This image is a composite image formed from a plurality of portions of images acquired by the image pickup device 20.
• the method of the invention comprises a step of comparing a sharpness criterion of each pixel or each group of pixels of a image with those of another image comprising an overlap of this or these pixel(s).
• a pixel with sufficient sharpness for example when the value of the criterion is greater than a predefined threshold, then the pixel is retained in the composite image.
• no pixel has sufficient sharpness, i.e. whose sharpness criterion exceeds a certain threshold, then the sharpest pixel is selected.
• an image alignment algorithm can be used to calibrate the images to each other and more particularly the portions of images covering the same areas of the surface. from the body.
• the construction of a sharp image comprises a cutting of acquired images according to their sharpest portions while checking that the cut-out areas not retained are included in other acquired images covering the removed part of an image. This control can be carried out automatically from the rate of coverage of the images acquired and from a marking of the areas covered.
• the invention makes it possible to generate a sharpness map of each image of the stack of images in order to select the sharpest pixels of the stack of acquired images covering a given area. It is therefore possible to generate an image from the sharpness map(s) in which the sharpest pixels are selected.
• a sharpness estimation step is performed to generate sharp areas and remove blurred areas.
• a sharpness estimation algorithm can be implemented.
• different solutions can be implemented such as a calculation of the mean or the variance of a gradient.
• Another method can be implemented by means of automatic learning from a learning function.
• the learning function can be deep learning type learning.
• a CNN convolutional neural network may be implemented.
• Such an algorithm makes it possible to generate sharpness maps of all the images in order to identify the sharpest pixels in a second step to generate a 3D image that is entirely sharp or at least as sharp as possible.
• Such a deep learning algorithm i.e. deep learning, can also be implemented to process the alignment of successively acquired images.
• the method of the invention comprises a step of assembling the sharp zones to construct a sharp image of the surface of all or part of the body.
• the stitching step can be seen as a step of aggregating image portions from different shots.
• Another way of carrying out an assembly is to carry out a fusion of the zones of covering by addressing each point by the sharpest pixels and by removing the pixels of these zones of covering more fuzzy.
• the surface area of the body surface represented by the sharp image generated is greater than the surface area of the body surface represented by one of the acquired images that made it possible to generate said sharp image.
• the assembled images can be assembled in a two-dimensional space or in a three-dimensional space. Indeed, knowing the depth information either from a body model or from an estimate during image acquisition, a 3D image can be automatically reconstructed.
• the 3D image can be reconstructed following assembly during processing applied to the final aggregated image.
• an end-to-end learning algorithm can be used to generate from a stack of acquired images, also called in English terminology “stack” of images, corresponding to the plurality of images collected, outputting a sharp image on the one hand and a depth map on the other.
• the learning of such a learning function for example of the CNN type, can be carried out on the basis of images acquired by optics or images generated by a computer of a body or of a model of body and on the other hand a totally clear 3D image obtained from all the images.
• We then speak here of an end-to-end neural network to designate a neural network capable of directly generating a sharp 3D image at any point.
• the passage through the sharpness map of each of the images of the stack of acquired images is then implicit in the implementation of such an end-to-end algorithm.
• this latter algorithm can be trained on synthetic data.
• a network of the CNN convolutional neural network type can for example be implemented.
• the set of learning data makes it possible to train the neural network to obtain a learning function configured or learned with weighting coefficients.
• the completely clean 3D image can be calculated from another algorithm in order to supervise the training of the neural network.
• Such learning can also result in the generation of a depth map.
• the neural network used whether functional or end-to-end, can be configured in different ways.
• the configuration of the CNN neural network may include:
• the configuration of the CNN neural network includes as inputs images acquired by the optics.
• the CNN neural network can include in its first layers convolutions then layers of fully connected neurons, called “fully connected layers” at the end of the model. In the latter case, they are neurons connected to all the neurons of the previous layer and connected to all those of the following layer.
• the convolution layers may include a scan of an input matrix producing a series of matrix calculations.
• the other layers of the neural network generally include matrix calculations on the size of the input matrix.
• each convolution comprises a matrix product between an input matrix and a weight matrix and the consideration of an additional bias.
• Applying layered processing within the CNN neural network includes applying a series of matrix multiplications which are followed by a nonlinear function to produce an output of said layer.
• the succession of these operations defines the depth of the neural network.
• the neural network is a multilayer perceptron, known by the acronym MLP and in English terminology by “multi-layers perceptron”.
• the neural network can be a network equivalent to the MLP.
• An advantage of the invention is also to reconstruct a relative depth map of all or part of the body.
• relative depth map we mean a map where each point of the image, for example each pixel, can be positioned vis-à-vis another point of the reconstructed image.
• One of the advantages of this characteristic is to make it possible to eliminate the effects of perspective or deformation of the images linked to the viewing angle or to the optics themselves.
• Another advantage of this method is to obtain an accurate estimation of the local geometry of the imaged surface. This allows to augment and refine the 3D body model when this latest body model is available as input to the system. This estimated depth information can have a resolution between 500um and 1cm. The input body model can then be refined to define a high resolution 3D body model.
• the image pickup device 20 is mounted on a mobile platform.
• the mobile platform can be arranged at the end of a robotic arm mobile in space and comprising a plurality of degrees of freedom.
• the platform is a drone controlled by a wireless remote control.
• the platform is a manual gun comprising a distal end piece on which is arranged the image pickup device 20 and in particular the variable focal length optics.
• the gun includes further a handle intended to be taken in hand by the examiner, for example at the proximal end of the gun.
• the device is fixed and the patient is turned by means of a rotating platform or vice versa.
• the invention is intended for any type of dynamic mobile platform comprising a distal end piece on which the variable focal length optics are arranged.
• the mobile platform comprises a rail or a guide means along a predefined trajectory and the optical device is movably mounted on said rail or guide means between two positions.
• the invention further comprises a cabin comprising a location intended to receive a subject, in particular a human subject and said mobile platform.
• the cabin may include at least one rail to give the mobile platform a curved trajectory around said subject placed in said location.
• the rail is movable in translation.
• the cabin comprises a plurality of rails and a plurality of platforms for imaging the skin of said subject.
• the varifocal optics include a photographic sensor.
• This last photographic sensor comprises a photosensitive electronic component and is used to convert electromagnetic radiation passing through the varifocal optics to convert it into an analog or digital signal.
• the photographic sensor can integrate, according to the embodiments, various components making it possible to filter or amplify the converted signal.
• CMOS sensor whose acronym in English terminology stands for “Complementary Metal-Oxide-Semiconductor”
• a CCD sensor designating “charge coupled device” in English terminology, that is to say a charge transfer device
• An important advantage of the invention is that it does not require a specific hardware sensor having to implement a hardware configuration that is structurally difficult to implement or costly from a hardware architecture point of view. In other words, a CMOS photographic sensor or CCD is sufficient to obtain clear images in a very short time, the operations being able to be carried out in real time.
• a single calculation unit such as a single processor, addressing all the pixels of the sensor is implemented.
• the processor is thus coupled to a physical memory.
• the implementation of a single couple of ⁇ processor, memory ⁇ allows a simplified implementation of the software processing processing the acquired images.
• An advantage of the invention is not having to process subsets of pixels that must be addressed and physically associated with electronic hardware dedicated to the subset, such as a processor or a memory.
• the invention makes it possible to obtain a sharp image at any point of a volume, such as a human body, and makes it possible to obtain real-time depth information at a high rate with market components such as a CMOS or CCD sensor that does not require the implementation of a complex electronic architecture.
• the invention can be implemented with a simple photographic sensor due to the following different possibilities making it possible to obtain fast calculation times:
• the range of values of the variation of the focal distance can be reduced due to a priori knowledge of a pre-generated body model making it possible to know locally the curvature of the volume from which the images are acquired and/or;
• the distance between the volume and the optics can be measured by means such as a laser to estimate the distance between the optics on the one hand and the acquired volume on the other, which makes it possible to perform a variation of the focal length circumscribed to a reduced range of values.
• the invention makes it possible to obtain depth data and of sharpness in a reduced range of values. This possibility makes it possible to reconstitute the three-dimensional envelope and the sharp images of each point of this envelope with great real-time acquisition performance.
• the invention also includes the implementation of a neural network classifying the sharp images of singularity of the skin of an individual.
• the invention comprises a step of detection or a step of segmentation of the image or a step of regression or even a step of generation of a new image.
• a step of generating a new image can be carried out by modifying the resolution of said image, by improving all or part of the image by a given processing, a transformation of the image for example by changing its dimensions or its colorimetry, its sharpness, or any other color parameter, or else a step of increasing, for example, the scale of the image or of only part of the image.
• Different analysis tasks can thus be performed from the sharp images produced by the method of the invention.
• all or part of the volume of an individual's body is scanned by varifocal optics.
• This operation can be carried out for example from a robot arm automatically evolving around a human body in order to follow a guide trajectory.
• the images are acquired and processed according to the method of the invention in order to collect a set of sharp images of the body of a patient.
• Machine learning can be done from photos of skin peculiarities, such as moles, scars, melanomas, carcinomas, freckles, etc.
• An advantage of the invention is to make it possible to train a neural network with sharp images at all points in order to obtain very good classification performance due to the maximum resolution obtained by the sharp images.
• the invention therefore relates to a method for training a neural network from sharp images obtained by the acquisition method of the invention.
• the training method is characterized by the input data of the neural network which are images obtained by the method of the invention.
• the method applies for example to a convolutional neural network of the CNN type and to any other type of neural network, in particular those previously described in this description.
• a “transformer” type network used alone or in combination with one or more convolutional neural network(s) of the CNN type.
• Such a type of network makes it possible in particular to differentially weight the importance of each part of the input data.
• the training can conventionally use image labels to classify the different singularities visible on the images or to perform any other analysis task previously listed. Furthermore, the invention relates to a method for classifying sharp images according to labels. Such a classification method can be applied to all of the images acquired by the method of the invention of an individual's body.
• the images can be represented according to a grouping of images of the same label with a mention of the member of the human body to which the image relates or an indication of a position on the human body or a body model.
• the invention can represent different zoom scales of a region of interest of a singularity so that a user can appreciate the high resolution of the image of a given singularity alongside a larger image. of the part of the body on which the photo is extracted.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Length Measuring Devices By Optical Means (AREA)
• Lenses (AREA)
• Telescopes (AREA)
• Instruments For Viewing The Inside Of Hollow Bodies (AREA)

Applications Claiming Priority (3)

Publications (1)

Family

ID=82115791

Family Applications (1)

Country Status (4)

Family Cites Families (3)

• 2022
  • 2022-06-01 EP EP22731608.0A patent/EP4377880A1/de active Pending
  • 2022-06-01 US US18/565,409 patent/US20240265562A1/en active Pending
  • 2022-06-01 AU AU2022286641A patent/AU2022286641A1/en active Pending
  • 2022-06-01 WO PCT/EP2022/064972 patent/WO2022253925A1/fr active Application Filing

Also Published As

Similar Documents

Legal Events