EP3274961A1 - Imaging of a macroscopic plant object - Google Patents

Abstract

Imaging of a macroscopic plant object. The invention relates to a method for imaging a macroscopic plant object, comprising the steps of: reception (201) of two-dimensional image data of said object, these image data coming from at least one two-dimensional capture device and comprising data acquired beyond the visible spectrum, and three-dimensional reconstruction (202, 203, 204, 205, 206, 207, 208) of said object from at least these data acquired beyond the visible spectrum.

Description

 Imaging a macroscopic plant object

The invention relates to macroscopic plant imaging.

 By "plant" is meant in the present application both plants, including algae, and macroscopic organisms commonly referred to as fungi, even if these organisms are not always recognized as being plants themselves.

 By "macroscopic" is meant that the object to be represented is seen with the naked eye. This object therefore has dimensions of the order of at least 1 / 10th of a millimeter, advantageously at least one millimeter, for example at least one centimeter. The dimensions of the object may however be less than ten meters, for example less than one meter.

 The invention can for example find an application in phenotyping, field counting, laboratory measurements of dimensions, etc.

 Phenotyping is a technique that determines how a given plant variety reacts to a given environment. More precisely, a large number of plants of the same variety or not are subjected to various manipulations, for example of the hydrous state of the soil, of the temperature (lighting with

Sodium for example), the CO2 content of the ambient air, the inoculation of diseases, etc. Measurements of plant growth, architecture and transpiration are then carried out in order to better correlate environmental conditions and plant conditions.

 Imaging boxes comprising a set of sensors make it possible to estimate various parameters of the plants having thus grown in these various environments. For example, an infra-red camera can be used to estimate nitrogen stress, a thermal sensor can be used to estimate water stress, etc.

 The current imaging boxes can make it possible to characterize up to 1800 plants per day.

A varietal selection can then be made based on this characterization, which will have made it possible to know that in a given environment, one variety prospers better than another. Field counting is a technique for estimating a number of plant elements, for example a number of ears of wheat or bunches of grapes, produced in a given plot, from acquisitions made on this parcel. , allowing to estimate for example a yield. Field counting conventionally involves pattern recognition software for the identification and counting of plant elements from images.

 The acquisition platforms used in macroscopic plant imaging may include several devices, for acquisition in the visible, in the infrared, etc.

 We are looking for 3D images, in order to have more information. For example, if a 2D image on which two ears of wheat are superimposed, the software may count only one ear. A three-dimensional image can thus limit this risk of error. In the context of a phenotyping application, a 3D reconstruction can make it possible to estimate a leaf volume, rather than a leaf area, and more generally to obtain a depth information. This can be particularly useful when studying species competition, since the foliar cover of each plant can be evaluated more precisely.

 It has been envisaged to add to a platform of the type known from the prior art a scanner, but there is a need for a simpler method.

 We therefore seek to perform a 3D reconstruction.

Document CA 2 764 135 describes an example of a 3D reconstruction method, in which a TOF camera (of the English "Time of Flight") and a color camera are used. The data from these two cameras are confronted with each other to reconstruct a 3D image of the plant.

 There is a need for a 3D reconstruction of macroscopic plant objects to reconcile simplicity and high image quality.

 There is provided a method of imaging a macroscopic plant object, wherein:

receiving two-dimensional image data from said object, said two-dimensional image data being derived from at least one a two-dimensional capture device, comprising for example an optical sensor, and these two-dimensional image data comprising data acquired beyond the visible spectrum, and

 a three-dimensional reconstruction of said object is performed from at least this acquired data beyond the visible spectrum.

 Thus, information corresponding to wavelengths outside the visible band for the 3D reconstruction of a macroscopic plant object is taken into account. By using this information outside the visible band, the accuracy of the 3D reconstruction can be improved.

 The invention may furthermore make it possible to avoid registration between acquisitions made outside the visible spectrum and the reconstructed 3D image, since the 3D image is reconstructed from these acquisitions outside the visible spectrum.

 In addition, this method has the advantage of implementation with little or no modifications of an existing platform, since most of these platforms already incorporate sensors outside the visible band.

 The macroscopic plant object may for example comprise: a macroscopic portion of a plant, for example the roots or a bud,

 - all of this plant, or even

 - a macroscopic set of several plants, in a state of competition or not. It is thus possible to carry out a reconstruction of several different plants in the same scene, or on the contrary of the same plants.

 Very advantageously, the received two-dimensional image data can also include data acquired in the visible spectrum, and these acquired data can also be used in the visible spectrum for the 3D reconstruction.

 Advantageously, the reconstruction process may be an SFF (Shape From Focus) process. This type of method has proved to be well suited to macroscopic plant imaging.

Thus, during the reception step, N> 1 sets of two-dimensional image data can be received, each set being acquired at a corresponding depth of field and including values of a plurality of pixels. Sets may include data acquired outside the visible spectrum, and possibly data acquired in the visible spectrum.

 These N sets can come from the same two-dimensional acquisition device. The method can thus be implemented with relatively simple optical equipment.

 A resetting step can be provided to match pixels of these N sets.

 It is then possible to estimate for at least one, and preferably each pixel, a depth of field value corresponding to a maximum of sharpness on these N acquisitions. The method may thus comprise, for each pixel matched, a calculation step for estimating from N sets of received image data, a depth of field value corresponding to a maximum of sharpness.

 For example, one can estimate for at least one, and preferably each, pixel, and for at least one, and preferably each, together, a sharpness value from the data of this set. The depth of field value corresponding to the maximum sharpness value is then chosen, or a depth of field value corresponding to a maximum of sharpness is calculated by interpolation.

 The invention is in no way limited to SFF reconstruction.

 For example, one could choose a technique of active stereovision, also called structured light. One then projects on the object a known pattern, then one captures for example a single image. The received two-dimensional image data can then comprise a single image, from which a 3D reconstruction is performed by analyzing the shape on this image of this prior known pattern.

 Alternatively, the reconstruction method may be a stereovision process. The received image data then comprises a left image and a right image.

 Advantageously, the data acquired in the non-visible spectrum may comprise data acquired in the infrared spectrum, for example data corresponding to wavelengths between 700 nm and 5 mm, advantageously between 780 nm and

50 μπι, advantageously between 800 nm and 20 μπι. These data may be relatively relevant when the object is a plant. Alternatively, or in addition, the data acquired in the non-visible spectrum may comprise data acquired in the ultraviolet spectrum, or the like.

 In an advantageous embodiment, the two-dimensional image data received may comprise, for each pixel, a set of P> 1 pixel values, each value being derived from an acquisition made in a corresponding frequency band. P frequency bands are thus provided, for example P frequency bands disjoined and covering a wide range of the spectrum, for example the visible and infra-red.

 In the case of an SFF reconstruction in particular, the method may then comprise, for at least one, and preferably each, pixel, and for at least one, and preferably each, N sets, a step of estimating a additional pixel value from the P> 1 pixel values corresponding to this pixel.

 The calculation step leading to estimating a depth of field for a pixel can then be carried out starting from at least one, and preferably N, additional values thus estimated for this pixel, and possibly with values of neighboring pixels, for example, additional values estimated for the neighboring pixels.

 When estimating this additional value, it is possible, for example, to choose one of the values P corresponding to this pixel, or else to proceed by interpolation.

 This additional pixel value may correspond for example to a maximum of reflected energy.

 The additional pixel value may be estimated to correspond to one or more frequency bands identified as particularly information-rich.

 Alternatively, this additional pixel value can be estimated so as to correspond to a maximum of sharpness. The additional pixel value can then be obtained by further using the values corresponding to the neighboring pixels.

The invention is therefore in no way limited by the way in which the additional pixel value is determined, provided that this estimate is a function of at least some of the P pixel values corresponding to this pixel. In another embodiment, the method may comprise, for at least one, and preferably each, pixel, and for at least one, and preferably each, N sets, a step of combining at least some of the P values pixel corresponding to this pixel so as to obtain an enriched pixel value, for example a P-couplet of values. By repeating this step for each pixel of a set, one can thus obtain an enriched image.

 For each pixel, it will be possible to estimate from the enriched image or images, for example from N enriched images, a depth of field value corresponding to a maximum of sharpness.

 There is further provided a device for imaging a macroscopic plant object, comprising

 reception means, for receiving from at least one two-dimensional image capture device two-dimensional image data of said object, said two-dimensional image data comprising data acquired beyond the visible spectrum and

 processing means arranged to carry out a three-dimensional reconstruction of said object from at least this acquired data beyond the visible spectrum.

 This imaging device can for example include or be integrated in one or more processors, for example in a microprocessor, a DSP (the "Digital Signal Processor"), or other.

 The receiving means may for example comprise an input pin, an input port, or the like.

 The processing means may for example comprise a processor core, or CPU (of the "Central Processing Unit"), or other.

 The imaging device may further comprise transmission means for transmitting three-dimensional reconstruction data from the processing means to a display device, for example a screen, a 3D monitor and / or other, with a view to a 3D rendering of the reconstructed object.

In addition, an imaging system for a macroscopic plant object is proposed, comprising an imaging device as described above and at least one capture device able to acquire two-dimensional data out of the visible spectrum and at a plurality of depths of field.

 For this purpose, displacement means may be provided for moving all or part of the capturing device, for example simply a lens, in order to adapt the depth of field.

 These moving means can be controlled by control means, for example all or part of a processor. These control means can be integrated in the same processor as the processing means of the management device, or not.

 Advantageously, provision can be made for a connection between the control means and these displacement means, on which are sent by the message control means in order to control the movement of all or part of the recording device, and another connection between the device. capture device and the imaging device, on which are emitted by the message capturing device carrying the data acquired.

 The use of these two separate links can reduce the congestion of the data link between the capturing devices and the imaging device. It will be possible to use a unidirectional bus.

 There is further provided a computer program product comprising instructions for performing the steps of the method described above when these instructions are executed by a processor. This program can for example be downloaded, saved on a hard disk type support, or other.

 Embodiments of the invention are now described with reference to the accompanying drawings in which:

 Figure 1 shows schematically an example of an imaging system according to one embodiment of the invention;

FIG. 2 is a flowchart corresponding to an exemplary method according to one embodiment of the invention. Similar references from one figure to another may be used to designate similar or similar objects.

With reference to FIG. 1, an imaging system 100 of a macroscopic plant object 2 comprises a two-dimensional capture device 3 and a terminal 5. The object 2 may for example comprise a portion of a plant plant, as illustrated in the figure.

 The capture device 3 may for example comprise a camera, a camera, or other.

 The terminal 5 may for example include a computer, a smartphone, or other.

 The capture device comprises a lens 32, drivable in movement along an optical axis by displacement means 31, 33. These displacement means may for example comprise a stepping motor 33 and a processor 31 driving this motor 33 The processor 31 can thus impose a displacement of the lens 32, and thus an adaptation of the depth of field.

 A sensor 34 makes it possible to generate electrical signals carrying information of energy reflected by the object 2. These signals are received by the processor 31 and sent by a unidirectional data bus 42 to the terminal 5.

 The displacement of the lens 32 is controlled by the terminal 5, via another bus 41, separate from the bus 42 in order to avoid cluttering the bus 41.

 The terminal comprises an imaging device 1, here a microprocessor, comprising:

 - Receiving means 1 1 of measured data from the capture device 3, for example one or more input pins,

 transmission means 13 for positioning control message of the lens 32, for example one or more output pins,

 processing means 12, for example a processor core or CPU, for developing the messages intended for the stepper motor 33, and for performing a 3D reconstruction of the object 2 from the 2D image data coming from the device capture 3,

transmission means (not shown) of reconstructed 3D data from the CPU 12, for transmitting this data for example to a screen for display.

The devices 1, 3 may be separate and possibly distant from each other, but it can be expected to embark these devices 1, 3 in the same device, for example a smartphone or a dedicated terminal.

 FIG. 2 is a flowchart of a method implemented in the imaging device 1.

 During a step 201, two-dimensional image data (p (x, y, i, j)) is received. More precisely, the lens referenced 32 in FIG. 1 is regularly moved from one position to the other. other along the optical axis, so that N> 1 acquisitions are made, each acquisition corresponding to a depth of field.

 In FIG. 2, the parameter j indexes the acquisitions.

 For each acquisition, we receive a set of pixel values, x and y indicating the position of the pixels on a 2D image. Each measured value p is a value of reflected energy.

 In addition, for a given acquisition, for each pixel corresponding to a pair (x, y), P values of reflected energy are received, each reflected energy value corresponding to a spectral band.

 The P frequency bands are disjoint and form a partition including visible and infra-red.

 In FIG. 2, the parameter i indexes the frequency bands. A registration is then made to match the pixels of the N acquisitions made. For example, you can enlarge some images and find fixed points from one image to another. As the depth of field varies from one acquisition to another, certain pixel values, especially on the edges of the images, can not be matched with the pixel values of the image acquired at the shortest distance from the image. object.

 In FIG. 2, the parameters x ', y' indicate the positions of the pixels after registration.

 Thus, the registration 202 is implemented by integrating the data acquired outside the visible spectrum, which is relatively advantageous compared to a method in which the reconstruction would be based on data acquired in the visible only and which would then impose a registration. between reconstructed 3D data and 2D data outside the visible.

A loop is then set up in order to traverse each of the N acquisitions, with initialization steps 203 of the index j, incrementing 206 of this index j, and with further a loop output test 207.

 For the sake of simplicity, the loops for traversing all the pixels of the same acquisition, which would relate to the pairs (x ', y'), have not been represented, but the person skilled in the art will understand that the Steps 204 and 205 are performed for each of the pixels after registration.

 When executing a loop, and for each of the set pixels, it is estimated during a step 204 an additional pixel value PE (X ', y', j). This estimate is made from the P values corresponding to this pixel.

 In this example, among these P values, the maximum value is chosen, that is to say that PE (X ', y', j) = max ({p (x ', y', i, j) i = i, ..., p).

 If this maximum value is reached in the infrared, it is thus information acquired outside the visible spectrum that will be retained for the rest of the process, thus allowing a better 3D reconstruction quality.

 This step 204 is performed for each of the pixels. Then, it is estimated a sharpness value for each of the pixels, during a step 205.

 In FIG. 2, step 205 succeeds step 204 within the same loop, but one skilled in the art will understand that in practice two other loops, subscripted x ', y', may be implemented because the estimation of the sharpness value H (x ', y', j) of a pixel is a function of the additional values obtained for the neighboring pixels.

The set of pixels is then scanned to estimate an additional value for each of the pixels before making a sharpness estimation.

 During this step 205, it is possible to use a sharpness operator of the type known from the prior art, for example a gradient-based operator, as described in the article by S. Pertuz et al., "Analysis of focus fit operators for shape-from-focus ", Pattern Recognition (2013), pages 1415- 1432, 2013, or another operator.

When the sharpness of each pixel of each of the N acquisitions is thus estimated, the loop output test 207 is positive and new loops, still unrepresented, are put in place in order to traverse all the set pixels. For each of the pixels, corresponding to a pair (x ', y'), it is estimated during a step 208 a depth of field value from the N additional values obtained for this pixel. We are actually looking for the depth of field corresponding to a maximum of sharpness.

For example, we choose the index J ^' MAX for which the relation

PE (X ', y', J ^' MAX) = max ({p (x', y ', is checked.

 Then, during a step 209, reconstructed image data is transmitted to a 3D engine. This data may comprise, for each pixel, the estimated depth of field value at step 208 for this pixel, as well as an energy value p (x ', y').

 This value p (x ', y') may be the additional value corresponding to the depth of field estimated at step 208, for example.

 The reconstructed image can then be analyzed in an automated manner, for example using shape recognition tools, particularly in the context of a field counting application.

 The step of transmitting the reconstructed image data to a 3D engine is optional. For example, it would be possible to provide transmission to processing means arranged to derive from these data a height value, particularly in the case of a growth monitoring type application. These means can, for example, determine a maximum (or minimum) depth value from the received depth of field values and assign the height value this maximum or minimum depth of field of view value.

 In another embodiment of the invention, not illustrated, rather than reducing in step 204 the P values of pixels to a single additional value, a spectrum is developed for each pixel. In other words, when all the pixels have been traveled, an enriched image has been obtained, and the sharpness of each pixel is estimated from the spectrum of this pixel and the spectra of the neighboring pixels.

The method described above can be implemented in the context of phenotyping, field counting, growth monitoring in the laboratory, or the like. This method can be implemented with a monocular tool and allows to obtain 3D information, which can participate in the quality of the variety selection in the case of phenotyping, or even counting in the case of field counting, or height measurements in the case of growth monitoring.

 The invention can thus find applications in agriculture, especially in viticulture, for example to characterize a culture in real time with an onboard system, for example to evaluate damage in a field, detect diseases early, characterize a number of plants per square meter, early yield, leaf volume, nitrogen content, number of plants seeded / sown, or other.

Claims

A method of imaging a macroscopic plant object, comprising the steps of:  receiving (201) two-dimensional image data of said object, said two-dimensional image data being derived from at least one two-dimensional capture device and including data acquired beyond the visible spectrum, and  performing a three-dimensional reconstruction (202, 203, 204, 205, 206, 207, 208) of said object from at least this acquired data beyond the visible spectrum. The method of claim 1, wherein  the received two-dimensional image data further comprises data acquired in the visible spectrum, and  these acquired data are used in the visible spectrum for three-dimensional reconstruction. The method of any one of claims 1 or 2, wherein the data acquired in the non-visible spectrum comprises data acquired in the infrared spectrum. 4. Process according to any one of claims 1 to 3, comprising  a step of receiving (201) N> 1 sets of two-dimensional image data, each set being acquired at a corresponding depth of field and including values of a plurality of pixels,  a resetting step (202) for matching the pixels of the N sets received, for each pixel, a calculation step (205, 208) for estimating from N sets of received image data, a depth of field value corresponding to a maximum of sharpness. The method of any one of claims 1 to 4, wherein  the received two-dimensional image data comprises, for each pixel, a set of P> 1 pixel values, each value being derived from an acquisition performed in a corresponding frequency band. 6. The method of claim 5 when dependent on 4, comprising  for each set of N sets, for each pixel of said set, an estimation step (204) from P values corresponding to this pixel of an additional value corresponding to a maximum of reflected energy,  and wherein the calculating step (205, 208) is performed from at least additional estimated pixel values. The method of claim 5 when dependent on 4, comprising a step of  for each set of N sets, for each pixel of said set, combining at least some of the P pixel values corresponding to a pixel to an enriched pixel value, so as to obtain N enriched images, and  wherein the computing step is performed from the N enriched images. 8. Imaging device (1) of a macroscopic plant object (2), comprising  receiving means (1 1), for receiving from at least one two-dimensional capturing device (3) two-dimensional image data of said object, said image data comprising data acquired beyond the visible spectrum and  processing means (12) arranged to perform a three-dimensional reconstruction of said object from at least this acquired data beyond the visible spectrum. 9. An imaging system (100) of a macroscopic plant object (2), comprising an imaging device (1) according to claim 8 and the two-dimensional capture device (3), said capture device (3) being able to acquire two-dimensional data out of the visible spectrum and at a plurality of depths of field. 10. System (100) according to claim 9, further comprising displacement means (31, 33), for moving all or part (32) of the two-dimensional capture device (3) so as to adapt the depth of field ,  control means (12) for controlling the moving means,  wherein the link (41) between the control means and the moving means is separate from the link (42) between the two-dimensional pick-up device and the receiving means (1 1) of the imaging device (1).