EP3638978A1 - Method and device for imaging a plant, and method for determining a characteristic of a plant - Google Patents
Method and device for imaging a plant, and method for determining a characteristic of a plantInfo
- Publication number
- EP3638978A1 EP3638978A1 EP18731889.4A EP18731889A EP3638978A1 EP 3638978 A1 EP3638978 A1 EP 3638978A1 EP 18731889 A EP18731889 A EP 18731889A EP 3638978 A1 EP3638978 A1 EP 3638978A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- image
- images
- plant
- window
- sensor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims description 51
- 238000003384 imaging method Methods 0.000 title claims description 13
- 230000033001 locomotion Effects 0.000 claims abstract description 15
- 238000013519 translation Methods 0.000 claims abstract description 6
- 241000196324 Embryophyta Species 0.000 claims description 92
- 238000012545 processing Methods 0.000 claims description 29
- 235000002017 Zea mays subsp mays Nutrition 0.000 claims description 21
- 238000006073 displacement reaction Methods 0.000 claims description 20
- 230000003287 optical effect Effects 0.000 claims description 20
- 230000011218 segmentation Effects 0.000 claims description 18
- 235000005824 Zea mays ssp. parviglumis Nutrition 0.000 claims description 16
- 235000005822 corn Nutrition 0.000 claims description 16
- 230000002093 peripheral effect Effects 0.000 claims description 10
- 230000004907 flux Effects 0.000 claims description 9
- 230000005484 gravity Effects 0.000 claims description 6
- 230000001902 propagating effect Effects 0.000 claims description 6
- 230000000877 morphologic effect Effects 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 238000002156 mixing Methods 0.000 claims description 3
- HHXYJYBYNZMZKX-UHFFFAOYSA-N 3,4:15,16-diepoxy-7-oxo-13(16),14-clerodadien-20,12-olide-(3alpha,4alpha)-form Natural products C12CCC3C4(C)CCCC(C)(C)C4CCC3(C)C1(C)CCC1C2(C)CCC1C(=C)C HHXYJYBYNZMZKX-UHFFFAOYSA-N 0.000 claims description 2
- 239000000203 mixture Substances 0.000 claims description 2
- 241000209149 Zea Species 0.000 claims 2
- 230000000694 effects Effects 0.000 claims 1
- 235000013339 cereals Nutrition 0.000 description 44
- 240000008042 Zea mays Species 0.000 description 20
- 210000005069 ears Anatomy 0.000 description 15
- 238000005259 measurement Methods 0.000 description 14
- 238000003306 harvesting Methods 0.000 description 6
- 238000004519 manufacturing process Methods 0.000 description 6
- 235000016383 Zea mays subsp huehuetenangensis Nutrition 0.000 description 5
- 235000009973 maize Nutrition 0.000 description 5
- 201000002200 Congenital disorder of glycosylation Diseases 0.000 description 4
- 235000013921 calcium diglutamate Nutrition 0.000 description 4
- 238000005286 illumination Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000011065 in-situ storage Methods 0.000 description 3
- UHPMCKVQTMMPCG-UHFFFAOYSA-N 5,8-dihydroxy-2-methoxy-6-methyl-7-(2-oxopropyl)naphthalene-1,4-dione Chemical compound CC1=C(CC(C)=O)C(O)=C2C(=O)C(OC)=CC(=O)C2=C1O UHPMCKVQTMMPCG-UHFFFAOYSA-N 0.000 description 2
- 241000223218 Fusarium Species 0.000 description 2
- 230000036579 abiotic stress Effects 0.000 description 2
- 230000001154 acute effect Effects 0.000 description 2
- 230000000712 assembly Effects 0.000 description 2
- 238000000429 assembly Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000012512 characterization method Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 201000010099 disease Diseases 0.000 description 2
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 2
- 230000009261 transgenic effect Effects 0.000 description 2
- 206010000210 abortion Diseases 0.000 description 1
- 231100000176 abortion Toxicity 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000009418 agronomic effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000001143 conditioned effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000001035 drying Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000004720 fertilization Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 239000004459 forage Substances 0.000 description 1
- 230000002068 genetic effect Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 239000010903 husk Substances 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 230000006698 induction Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 235000016709 nutrition Nutrition 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000003976 plant breeding Methods 0.000 description 1
- 238000012794 pre-harvesting Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000011002 quantification Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 230000035882 stress Effects 0.000 description 1
- 208000024891 symptom Diseases 0.000 description 1
- 239000012780 transparent material Substances 0.000 description 1
- 235000013311 vegetables Nutrition 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01D—HARVESTING; MOWING
- A01D45/00—Harvesting of standing crops
- A01D45/02—Harvesting of standing crops of maize, i.e. kernel harvesting
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01D—HARVESTING; MOWING
- A01D75/00—Accessories for harvesters or mowers
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
- G01B11/245—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures using a plurality of fixed, simultaneously operating transducers
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B5/00—Measuring arrangements characterised by the use of mechanical techniques
- G01B5/0035—Measuring of dimensions of trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/12—Edge-based segmentation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/50—Constructional details
- H04N23/54—Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/50—Constructional details
- H04N23/55—Optical parts specially adapted for electronic image sensors; Mounting thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/56—Cameras or camera modules comprising electronic image sensors; Control thereof provided with illuminating means
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/695—Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30128—Food products
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30181—Earth observation
- G06T2207/30188—Vegetation; Agriculture
Definitions
- the present invention relates to a device for taking images of a plant, in particular a device for taking peripheral images of a part of an elongated plant, such as a cluster or an ear, and to methods of taking images of a plant using such a device, and processing these images.
- the quantification of characteristics is important for characterizing the yield potential of a line or a hybrid, transgenic or not.
- Pre-harvest estimates of maize yields are generally made, notably by measuring the number of ears per acre, the number of rows and the number of grains per row, as well as the specific weight of the grain. Values such as grain size, length, width, volume can then be taken into consideration. The measurement of these characteristics by an operator is, however, laborious and subject to error.
- the patent application US2009046890A1 describes a method and a system for analyzing digital images of maize ears, which make it possible to evaluate at least one property of the ear and maize grains, in particular the number and size of the maize ears. grains.
- the system includes an image sensor such as a CCD camera, which delivers images to image processing means that apply to the images various processing algorithms such as filtering or contour search, for example.
- image sensor such as a CCD camera
- image processing means that apply to the images various processing algorithms such as filtering or contour search, for example.
- the taking of images of ears placed on a support such as a conveyor does not allow to have a complete view of the ear and allows only an estimate of the properties of the ear and its grains.
- WO 2013/101806 describes a variation of this device for the measurement of characters on immature ears; in this device also a picture of the ear is interpreted; it is stated that the ear may be on the plant, but no technical solution is provided to describe this embodiment.
- WO2017021285A1 discloses a method and device for imaging the outer surface of a corncob during the fall of the spike in the field of view of an imaging module that has an odd number of sensors image attached to a frame.
- the contour of the portion of the spike seen by the sensor is extracted, and these contours are then combined with respective rotations adapted to construct a model of the three-spike spike. dimensions.
- This device can be used in the laboratory or combined with a harvester but does not allow measurement on ears in situ (standing).
- the volume of this equipment is particularly limiting to allow such use
- Such methods and systems for taking images of a plant are not suitable for determining properties of a growing plant from images of the plant.
- the application WO2015086988 describes a portable device for taking images of objects. having an elongated transparent window adapted to surround the plant, a recessed reflective optic extending around the window and whose reflective surface (s) is (are) directed towards the axis longitudinal of the window, and one (or more) image sensor (s) of parts of the plant that are reflected by the (or) surface (s) reflective (s) substantially parallel to the longitudinal axis.
- This application WO2015086988 proposes to construct a 3D model by stereoscopy.
- the image-taking device is moved along the plant, successive images are taken during this displacement, and then these two images are connected in pairs. adjacent longitudinal portions (i.e. "slices") of the plant.
- this displacement In order to obtain a faithful image of the outer surface of the plant, it is necessary that this displacement be, as perfectly as possible, rectilinear, coaxial with the longitudinal axis of the plant, and regular (ie at a constant speed). .
- the use of this device in the field for the characterization of an ear of corn is therefore conditioned by the regularity of the gesture which is made difficult by the weight of the device, and the potential movement of the standing ear.
- the irregularity of the movement can be corrected by the use of an accelerometer which can measure the speed of introduction and correct the image assembly accordingly, however this solution is not very compatible with a fast signal processing, allowing the user to quickly know the characteristics of the spike studied.
- the ear Due to the irregularity of the ears and the introduction movement, the ear can come into contact with the wall of the device and the processing of the images thus obtained is made all the more complex.
- An object of the invention is to propose a method and a device for taking object images, this object being in particular a plant, in particular a method and a device for taking peripheral images of a plant of elongated shape, which are improved and / or which remedy, at least in part, the shortcomings or disadvantages of known methods and devices for taking images of an object.
- Another object of the invention is to propose a method for determining a characteristic of a plant from images of the external surface of the plant, which is improved and / or which at least partly remedies the deficiencies. or disadvantages of known methods.
- plant is used hereinafter to designate a part to be imaged of a standing plant.
- the invention is in no way limited to standing plants.
- At least one reflecting surface directed towards the longitudinal axis is arranged around the window, i.e. a normal vector of the reflecting surface is directed towards the longitudinal axis, and at least one image sensor of the plant reflected by the (or) surface (s) reflective (s);
- At least one first image is captured from at least a first peripheral portion of the plant which is reflected by the reflective surface (s) substantially parallel to the longitudinal axis;
- the reflective surface (s) and the image sensor (s) are moved relative to the window along the longitudinal axis;
- an image of a peripheral part of the plant is formed from the first image and the second image.
- the invention thus makes it possible to obtain images of the external surface of the plant imaged through the window, without moving the overall device, which facilitates the combination of the images successively taken to form an image of the complete external surface of the plant which be faithful and allow a precise determination of the properties of the plant from the images.
- the number of images taken during a complete movement of the reflective surface (s) and (or) image sensor (s) along the window may for example be of the order of three up to one or more ten (s).
- the images taken correspond to the same number of portions or adjacent zones (substantially adjacent) two by two of the external surface of the plant, these areas or neighboring portions overlapping preferably partially two by two, as well as the corresponding images.
- the images of the plant are captured during the displacement of the reflective surface (s) and the image sensor (s), ie without interrupting this movement, which helps to accelerate the taking of images;
- the plant is illuminated by a light pulse produced by at least one lighting source (which can be mobile), which contributes to limiting the influence of the surrounding light;
- the ratio of the time separating two successive light pulses to the duration of each light pulse is greater than 1, in particular situated in a range of from approximately 10 to approximately 1000, so that the displacement of the sensor (s) during each light pulse is very low, which helps to improve the quality of the images obtained;
- the acquisition of the images is controlled in synchronism with the displacement of the reflective surface (s) and the image sensor (s) along the window;
- the reflective surface (s), the image sensor (s) and, if appropriate, the source (s) of illumination are integral with the least one movable support common to the reflective surface (s) and the image sensor (s), which is moved (by an actuator) with respect to the window, along the longitudinal axis, so as not to vary the relative position of the (or) surface (s) reflective (s) and (or) sensor (s) images.
- a device for acquiring images of a plant which comprises:
- An observation window (substantially radial) of the plant which has a longitudinal axis and is arranged to receive, cover, and / or surround a portion to be imaged of the plant;
- At least one image sensor disposed outside (and / or around) the window, and whose field of vision includes the reflecting surface, so as to capture images of the plant part surrounded by the window which are reflected by the reflective surface (s) substantially parallel to the longitudinal axis, when the reflective optics is disposed around the plant part;
- an actuator arranged to drive the support in motion (relative to the window) along the longitudinal axis.
- a device for acquiring images of a plant which comprises:
- a first structure delimiting a cavity for receiving a part to be imaged by the plant, the cavity being elongate along a longitudinal axis and having an opening at (at least) one of its two longitudinal ends, for the introduction of the portion of plant to be imaged in the cavity, the cavity being delimited by at least one transparent wall (of the structure) forming a viewing window (substantially radial) of the plant part to be imaged;
- a second structure extending around the window and mounted to move in translation along the longitudinal axis of the cavity relative to the first structure, the second structure comprising (and supporting):
- each sensor including at least a portion of the reflective optics so as to capture images (substantially radial) of the vegetal portion surrounded by the window that are reflected by the reflective optics;
- an actuator arranged to drive the second structure in motion relative to the first structure, along the longitudinal axis.
- the reflective optics comprises a reflective surface associated with each image sensor
- each reflecting surface crosses (ie extends across) the optical axis of (an) image sensor; in particular, the intersection of the optical axis and the reflecting surface may be close to (substantially coincidental with) the center of this surface;
- the device comprises at least one light source associated with each image sensor, which is integral with the second structure and is arranged to produce a luminous flux directed towards the reflecting surface associated with the sensor, in particular a light flux propagating substantially parallel to the optical axis of the image sensor, and / or substantially parallel to the longitudinal axis of the window and the cavity;
- the device comprises several moving sets of imaging (substantially radial) each comprising said reflective surface, said image sensor whose field of vision includes the reflecting surface, and optionally said source of illumination;
- the number of image sensors and / or imaging assemblies is preferably in a range from about four to about twelve or sixteen;
- the lighting source (s) is (are) integral with the mobile support (and / or the second structure);
- the optical axis of the (each) image sensor is substantially parallel to the longitudinal axis of the window
- the device comprises guide members integral with the first structure and arranged to guide the second structure during its movement;
- the device comprises a processing unit of the signals and / or image data delivered by the image sensor (s), which is integral with (immobile with respect to) the window and the first structure, which contributes to limiting the mass of the moving equipment; for this purpose also, we can supply the (s) sensor (s) and where appropriate the (or) source (s) of illumination by a power unit secured to (immobile with respect to) the window and the first structure;
- the transmission of the signals or data from the sensors to the processing unit can be carried out by a wired easily deformable link such as a sheet of wires connecting the image sensor (s) to the a processing unit, or a wireless link such as a radio link, or an optical link;
- a wired easily deformable link such as a sheet of wires connecting the image sensor (s) to the a processing unit, or a wireless link such as a radio link, or an optical link;
- the power supply (s) of the (s) sensor (s) and (or) source (s) of lighting can also be performed by a deformable wire link connecting the (or) sensor (s) and / or source (s) to the power unit, or through a (wireless) induction link;
- the (each) reflecting surface is inclined with respect to the longitudinal axis (and generally with respect to the optical axis of the image sensor associated with the reflecting surface), at an acute angle of inclination which is preferably distinct and close to 45 degrees, in particular an angle of inclination in a range of about 30 or 35 degrees up to about 40 or 44 degrees, or in a range of about 46 or 50 degrees up to about 55 degrees. or about 60 degrees, which helps reduce stray light propagating to the image sensor (s);
- the (each) reflecting surface is curved (convex), in particular cylindrical, which makes it possible to obtain an image of a longitudinal portion
- the device comprises a (first) screen (opaque) extending around the reflective optics, so as to form a background (for the (s) sensor (s) of images) contrasting with the plant and contributing to limit the influence of the surrounding light;
- the first screen may be integral with the mobile support (and / or the second structure);
- the device comprises a (second) screen extending around and along the window, from the image sensor (s), over a length less than the distance separating the sensor from the associated reflecting surface; to the sensor, this second screen being integral with the movable support (and / or the second structure), so as to reduce the stray light propagating towards the image sensor (s), in particular the stray light resulting from reflection (s) of the flux produced by the source (s) of lighting.
- a data processing unit of the device may be programmed to implement a method of determining a characteristic of a plant as described hereinafter.
- a method for determining a characteristic of a plant in which initial images are acquired of areas adjacent to the outer surface of the plant respectively acquired from several points of view by a plurality of sensors of an imaging device comprising an optical device (having at least one objective) associated with each sensor, the neighboring areas partially overlapping in pairs, which method comprises the following operations:
- a pixel quality indicator is furthermore calculated for each pixel of the segmented image, the indicator varying as a function of the angle under which the point of the model of the plant which is the image of the pixel by the optical device, and wherein the quality indicator is used for mixing the developed images;
- the quality indicator corresponds substantially to the angle of inclination with respect to the normal to the surface of the model, to the image point of the pixel in question, to the radius coming from the image point of the pixel considered and resulting in the pixel in question;
- the determination of the characteristic comprises a segmentation of the assembled image or developed images, so as to obtain closed contours; we can then calculate the center of gravity of each closed contour whose elongation is less than a given value;
- the segmentation of the assembled image or developed images includes a conversion to black and white
- the segmentation of the assembled image or of the developed images may comprise a global thresholding operation, or preferably a local thresholding operation;
- one selects, among the closed contours obtained by segmentation of the assembled image or the developed images, those whose at least one morphological parameter such as the lengthening of the contour, the convexity of the contour, or the surface delimited by the contour, satisfies a determined selection criterion, in particular those whose elongation is less than a given value; when the plant is an ear, the number of closed outlines selected is a reliable estimate of the number of grains of the ear;
- the center of gravity of each of the closed contours thus selected is calculated; this allows for example to determine the number of rows of an ear, or to determine if the ear has a regular grain structure and in deduce the areas correctly fertilized (presence of grains or grains in formation) and the zones or grains are unfertilized or aborted;
- the developable surfaces of the model are determined from the segmented images resulting from initial images acquired by several sensors, in particular by all the sensors, in particular by intersecting contour information contained therein in segmented images.
- a data processing system comprising means for implementing the method for determining a characteristic of a plant described in the present application
- a program comprising instructions which, when the program is executed by a data processing unit such as a computer, cause the processing unit to implement the method for determining a characteristic of a plant described in the present application;
- FIG. 1 illustrates schematically an image taking device, in longitudinal sectional view in a plane containing the axis of symmetry of the reflective optics of the device and of the receiving cavity of a plant to be imaged, in a first configuration in which the reflective optics is located near the inlet opening in the cavity.
- FIG. 2 schematically illustrates the device illustrated in FIG. 1, in the same longitudinal sectional view, in a second configuration in which the reflective optics is situated at a distance from the entrance opening in the cavity.
- FIG. 3 schematically illustrates a reflective optic such as that of the device illustrated in FIGS. 1 and 2, in longitudinal section, and its support.
- FIG. 4 schematically illustrates a part of the image taking device illustrated in FIGS. 1 and 2, in cross-sectional view, and is a view along line IV-IV of FIG.
- FIG. 5 is a timing diagram schematically illustrating the variations in the speed of displacement of the second structure of a device such as that illustrated in FIGS. 1, 2 and 4, with respect to the first structure of the device, during a cycle displacement, and illustrating the imaging and lighting operations during this cycle.
- Figures 6 and 7 are flowcharts of methods for determining a characteristic of a plant using a device such as that illustrated in Figures 1 to 3.
- FIG. 8 illustrates a point determination operation of a three-dimensional model which is the image of a pixel of an image by the optical device associated with an image sensor.
- FIGS. 9 to 14 are images obtained by a device as described in the present application:
- FIG. 9 is an image of an angular sector of a "slice" of a corn cob acquired by an image sensor
- FIG. 10 is an image obtained by segmentation of the image of FIG. 9;
- FIG. 11 is an image (flattened) of a longitudinal band of an ear of corn that can be obtained by the assembly of images such as that illustrated in FIG. 10 which have been laid flat;
- FIG. 12 is an image obtained by segmentation of the image of FIG. 11, so as to include grain boundaries identified by segmentation and their respective centers of gravity;
- FIG. 13 is an image (flattening) of the entire outer surface of a corn cob that can be obtained by assembling images such as that illustrated in Figure 11;
- FIG. 14 is an image obtained by segmentation of the image of FIG. 13, so as to include grain boundaries identified by segmentation and their respective centers of gravity;
- FIGS. 15 and 16 are graphs illustrating the correlations between the number of corn cob grains obtained by image processing methods and the number of grains obtained by manual counting;
- FIG. 15 corresponds to the processing of an image of the reconstituted whole spike, and
- FIG. 16 corresponds to the processing of several images of vertical portions of the spike;
- FIGS. 17 and 18 are graphs illustrating the correlations between the width (FIG. 17) and the length (FIG. 18) of ears of corn obtained by the methods and devices described in the present application, and by manual measurements.
- phenotypic characteristics related to the ear or grain must be both high throughput to address a large number of plants compatible with these programs, but with high data quality.
- yield components In varietal selection, complex traits such as yield can be broken down into yield components. These components are the number of grains per ear, the size of this grain, length, width and volume, the total volume of the ear, the number of rows and the number of grains per row. The analysis of these components on a corn ear can be used to select the ears during the varietal selection steps.
- Imaging analysis of the spike also makes it possible to visualize the fertilized and unfertilized portions of the spike, this characteristic being of particular importance if one wants, for example, to evaluate the tolerance of a plant to abiotic stress such as drought. Grain abortion is one of the important markers for these stresses.
- Field image acquisition can also be a tool for monitoring of diseases and characterization of equipment tolerance to abiotic stresses (Fusarium verticiloides, Fusarium monoliforme, etc.)
- the information generated can be used in many applications. These applications can be the identification of character of interest in genetic resources, the use of these characteristics in a selection program, whether on a line or on a hybrid, the visualization of the efficiency of a multiplication program seeds, etc.
- grain corn production the early evaluation of grain value such as grain number, color, grain size, will allow to evaluate the quality of the crop, this value is not only important in production corn grain, in corn production forage this value will also be an indicator of the value of the crop, and allow to evaluate a nutritional value, and for example to set the harvest price.
- the measurement of maize ear characteristics is of agronomic interest, particularly if this measurement is made in situ before harvesting.
- the device described by the invention allows these measurements on foot after removal or folding spathes covering the spike, and can be advantageously coupled to a system of geo-referencing to allow the operator to have data on ears by having the exact knowledge of their position, especially when measurements are made on plants selected randomly on a plot, but also to bring more directly the results of the plant evaluated when one is in a specific experimental plan.
- the device can also be used on cob without folding spathes, for the research of particular characteristics such as the presence of symptoms related to a disease, especially on husks.
- the image taking apparatus 10 comprises a reflective optic 11 which is pierced by a central recess 21 and extends around an internal window 12.
- the optic 11 is surrounded by an external window 15.
- the windows 12, 15 may be made of a transparent material such as glass or a lighter material. Alternatively, the outer window 15 may be opaque.
- the windows 12, 15 have a tubular shape, in particular a cylindrical shape of circular section, and whose axis 13 forms the longitudinal axis and of general symmetry of the device 10.
- the windows 12, 15 are part of the first structure 60 of the device, the structure 60 further comprising a wall 62 secured to the windows, which extends to - and closes - the upper end of the windows and supports an actuator 64 such that a stepper motor.
- the window 12 and the wall 62 thus delimit a cylindrical cavity
- tubular 33 open at its lower end 65, having an elongated shape along a longitudinal axis 13, a height (measured along the axis 13) and a cross section which are adapted to the portion of plant to be imaged which must be introduced into this cavity to do this.
- the walls 12, 62 of the device 10 are intended to "cap” and surround the whole of an ear of a standing plant, the stem supporting the spike extending through the opening 65 and under the device.
- the optic 11 is formed of eight identical mirrors whose respective reflective surfaces 19 are inclined relative to the axis 13 of an acute angle 20 which is common to the surfaces 19 and for example substantially equal to 50 degrees, and are directed towards the top (see Figures 1 to 3) and the axis 13 of the device 10.
- the optic 11 is part of the second structure 61 of the device, which extends around the window 12 and is mounted to move in translation along the longitudinal axis 13 with respect to the first structure 60.
- the structure 61 further comprising eight identical imagers each comprising an image sensor 18 and an objective (reference 14 FIG. 8) with an optical axis 66 parallel to the axis 13.
- the image sensors 18, which may be matrix sensors of CMOS technology, are attached to an annular wall 72 and are arranged, as are the surfaces 19, evenly distributed around the window 12 and the axis 13.
- the reflective optics thus comprises a reflective surface 19 associated with each image sensor.
- each surface 19 is curved (convex), these surfaces extending along cylinders 120 of circular section and whose respective longitudinal axes 121 intersect at the same point 122 of the axis 13.
- the intersection of the optical axis 66 of a sensor 18 and the associated reflecting surface 19 is substantially merged with the center of this surface.
- the field of vision 77 of each sensor includes at least a portion of the reflective optics so as to capture images of an angular portion (or sector) of a horizontal section of the plant part surrounded by the window 12 which are reflected by the reflective optics.
- the positioning of the device 10 around the spike to be imaged is performed so that the spike extends substantially along the axis 13 of symmetry of the optics 11 and the cavity 33, so that the different portions of the peripheral surface of the spike are situated substantially at the same distance from the window 12, and consequently from the surfaces 19 of the optics 11.
- a guide 69 for the "free" end of the spike i.e. its upper end may be provided inside the cavity 33, as shown in FIG.
- the guide 69 is slidably mounted in the cavity 33 along the axis 13, and rests, by its upper portion, on a spring 70 integral with the wall 62.
- the lower face of the guide 69, against which the end of the spur can rest, comprises a central depression 71 serving to center the upper end of the spike in the cavity 33.
- the spike is furthermore positioned in the lower part of the cavity 33, to extend above the recess 21 of the optic 11, when the optic 11 and the second structure 61 are in the end position.
- "Bass" substantially as illustrated in FIG. 1, and is positioned to extend below the recess 21 of the optic 11, when the optics 11 and the second structure 61 are in the "high" end-of-travel position, substantially as illustrated in FIG. 2, so that the optic 11 sweeps the entire external surface of the spike during a cycle of displacement of the second structure 61 with respect to the first structure 60.
- the first structure also comprises guides 68 for guiding the second structure during its movement, which are fixed to the wall 62 and to a transverse annular wall 75 connecting the respective lower ends of the walls 12, 15.
- the guides 68 extend parallel to the axis 13 and pass through openings 74 respectively provided in two walls 72, 73 forming part of the second structure and each having an annular shape.
- the actuator 64 fixed to the first structure 60 is arranged to drive the second structure 61 in translation (alternative) along the axis 13, with respect to the first structure, via a member 76 for transmitting motion such as than a toothed belt.
- the first structure comprises an image data processing unit 67 delivered by the image sensors 18, and the device comprises ribbon cables 78 connecting the sensors to the unit 67 and serving to transmit the data from the sensors to the sensor unit. to the treatment unit.
- the data processing unit can be connected to a display interface that can be integrated with the portable device, so that the measurement and image processing results can be viewed by an operator carrying the device, via this interface. . If a geolocation system is associated with the device, the geographical position of the measurement will also be available.
- the mobile structure 61 furthermore comprises two coaxial cylindrical screens (of axis 13): a first screen 79 extends around the reflective optics and forms a background for the sensors images; and a second screen 80 extends around and along the window 12, from the image sensors 18 which extend around this screen, along a length 81 (FIG. 1) less than the distance separating each sensor from the reflecting surface associated with it, so as to reduce stray light.
- the mobile structure 61 includes a light source 29 associated with each sensor 18, which is fixed to the support 72 common to the sensors 18, and is arranged to produce a light flux directed towards the reflecting surface associated with the sensor, propagating parallel to the light. optical axis of the image sensor, and parallel to the longitudinal axis 13, for illuminating a portion to be imaged of a portion of the surface of the spike, by reflection of the luminous flux by the surface 19.
- the luminous flux produced by the source (s) of illumination may have a "white” or “yellow” light spectrum.
- the portable device 10 may furthermore comprise a data storage unit connected to the processing unit 67 and arranged to record data delivered by this processing unit, and a battery arranged to supply the actuator 64, the sensors 18, sources 29, unit 67 and the data storage unit.
- the timing diagram illustrated by a thick and continuous line represents the speed of the structure 61; the timing diagram illustrated by a short dashed fine line represents the (simultaneous) activation of the lighting sources 29; and the chronogram illustrated by a fine dotted line represents the (simultaneous) capture of an image by each of the sensors 18.
- the unit 67 can control the operation of the actuator 64 for moving the structure 61 , lighting sources 29, and sensors 18, so as to take pictures of the spike, as shown in FIG. 5:
- the actuator is controlled to move the structure 61 upwards, at a constant speed 92, which may for example be close to one to one hundred centimeters per second, so as to perform a "go" race 93, until the mobile structure 61 reaches the upper end of travel illustrated in Figure 2; this results in a displacement of the reflecting surfaces 19, the sources 29, and the sensors 18, with respect to the window 12, along the axis 13;
- the unit 67 controls the operation of the sources 29 and the taking of images of the spike reflected by the surfaces 19, during this operation, three times in the example illustrated in FIG. 5; the duration of a taken of an image, which is marked 91, may for example be close to one or more milliseconds, and the time between two successive image acquisitions, which is marked 90, may for example be close to one hundred milliseconds to one second ; each image can be taken entirely during a light pulse produced by the sources 29, the duration 95 may also be close to one or more milliseconds, this time then being slightly greater than that 91 image, as illustrated in Figure 5; according to an alternative (not shown), each image can be started during a light pulse and continue after the end of this pulse; in this case, it is important for the light pulse to be sufficiently intense so that the flux received by each sensor during the pulse is much greater (for example a hundred times greater) than that received by the sensor after the end of the pulse. impulse ; the acquisition of the images can be controlled by the control and data processing unit 67,
- the speed 92 is chosen so that, for each sensor 18, the three images successively taken are respectively images of three zones (of slices or sections) adjacent to the external surface of the spike, partially overlapping two to two;
- the acquisitions of images can be done at the rise and / or the descent of the mobile structure.
- the determination of the number of grains Ng of the spike by the data processing unit 67 can comprise the following operations:
- this operation may include the removal of the background image including for example the contour of the reflecting surface, to retain only the portion of the spike seen by each sensor, and the determination of a contour of this part; this operation may include, for each initial image taken by each sensor, the detection of the left and right edges of the spike or the edge of the spike, and where appropriate the modeling of these edges by segments of line;
- the coordinates Co of the point m of the model Mv which is the image of the pixel by the optical device, are calculated; to do this, as illustrated in FIG. 8, the path of the ray 114 coming from the pixel 110, passing through the lens 14 of the sensor, reflected by the surface 19, and ending on one of the cylindrical surfaces 115 of the Mv model, at 111, is calculated.
- these coordinates can be the Cartesian coordinates (X, Y, Z) of the point 111 in the Cartesian coordinate system (: * ⁇ , y, z) linked to the model Mv and which one of the axes, for example the z axis, can be confused with axis 13;
- the segmented image can then be flattened according to the coordinates (X, Y, Z) calculated for each pixel of the segmented image, so as to obtain a developed image Id, for example that illustrated in FIG. 11 ;
- the determination of the number of grains Ng of the spike can then be carried out from the assembled image or the developed images Id.
- the lateral assembly of the developed images of the spike or slice of the spike can be done after the choice of a projection plane and an angular origin common to the different images; for the flat projection, the Cartesian coordinates associated with the pixels can be transformed into cylindrical coordinates.
- a quality indicator of the pixel which varies according to the angle at which the point of the model of the plant is seen which is the image of the pixel by the optical device formed by the lens 14 and the mirror 19, and this quality indicator can be used to perform a mixture of the overlapping portions of the developed images, during their assembly: the pixel values of the image portions corresponding to the overlap areas can be weighted by a factor proportional to this indicator.
- This quality indicator may correspond to the angle of inclination 112 with respect to the normal 113 at the surface 115 of the Mv model, at the image point 111 of the pixel 110 considered, the radius 114 connecting the image point 111 of the pixel in question to the pixel considered.
- the determination of the number of seeds Ng of the spike from the assembled image la can be carried out for each image of a lateral band of the spike obtained by assembling images taken by one of the sensors, in particular during the displacement of the mobile structure of the device.
- the determination of the number of grains Ng can be carried out for the image resulting from all the images of the spike.
- the determination of the number of grains Ng can be performed by segmentation of the developed images corresponding to the images acquired by the images. different sensors.
- Segmentation of the assembled image or developed images is performed to obtain closed contours, which are the contours of "candidate" grains.
- the segmentation of the image may include a black-and-white conversion of the assembled image, and a local thresholding operation.
- the number of grains it is possible to select, among the closed contours of candidate grains obtained by segmentation, those whose at least one morphological parameter such as the lengthening of the contour, the The convexity of the contour, or the surface delimited by the contour, satisfies a particular selection criterion, in particular those whose elongation is less than a given value.
- a particular selection criterion in particular those whose elongation is less than a given value.
- CDG center of gravity
- the selected closed contours and the corresponding CDGs may be included in the images Id, the projected and, where appropriate, the assemblies illustrated in FIGS. 13 and 13, to form final segmented images such as those respectively illustrated in FIGS. 12 and 14.
- the zones "not fertilized or aborted" of the external surface of the spike which are those not including closed contour (grain) and whose at least one geometric or morphological parameter satisfies a given criterion, and calculate the area of these zones.
- CDG also makes it possible to determine a number of ranks of grains of the imaged spike.
- the developable surfaces of the model Mv are determined from the segmented images resulting from initial images acquired by several sensors, in particular by all sensors.
- contour information contained in segmented images resulting from initial images acquired by all sensors can be intersected.
- This model can be used in particular to calculate the volume of the ear and the area of the outer surface of the ear.
- FIG. 15 corresponds to the number of grains counted on the image of the reconstituted whole cob as illustrated in FIGS. 13 and 14;
- the number Ng2 carried on the ordinate axis Figure 16 corresponds to the number of grains counted from 8 images corresponding to vertical sections of the spike as illustrated in Figure 11, developed images;
- the ellipses illustrated in FIGS. 15 and 16 correspond to a density (correlation) of 0.95.
- the correlation between the number of ranks counted by the apparatus and the method described, and the number of ranks counted manually, is also significant.
- the ellipses illustrated in FIGS. 17 and 18 correspond to a density (correlation) of 0.95.
- Figure 17 shows that the correlation between the width of the ears obtained by the claimed device, which is plotted on the abscissa, and the manual measurements of diameter (carried on the ordinate axis) is approximately 0.95. Manual diameter measurements were made at one third of the head height from the base of the ear.
- FIG. 18 shows that the correlation between the length of the spike obtained by the invention, which is plotted on the abscissa, and the manual measurements taken on the ordinate axis, is about 0.9.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Environmental Sciences (AREA)
- Quality & Reliability (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Botany (AREA)
- Investigating Or Analysing Materials By Optical Means (AREA)
- Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)
Abstract
Description
Claims
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
FR1755391A FR3067824B1 (en) | 2017-06-14 | 2017-06-14 | METHOD AND DEVICE FOR IMAGING A PLANT |
PCT/FR2018/000153 WO2018229360A1 (en) | 2017-06-14 | 2018-06-01 | Method and device for imaging a plant, and method for determining a characteristic of a plant |
Publications (1)
Publication Number | Publication Date |
---|---|
EP3638978A1 true EP3638978A1 (en) | 2020-04-22 |
Family
ID=60765682
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP18731889.4A Withdrawn EP3638978A1 (en) | 2017-06-14 | 2018-06-01 | Method and device for imaging a plant, and method for determining a characteristic of a plant |
Country Status (4)
Country | Link |
---|---|
US (1) | US10873699B2 (en) |
EP (1) | EP3638978A1 (en) |
FR (1) | FR3067824B1 (en) |
WO (1) | WO2018229360A1 (en) |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3657156A4 (en) * | 2017-07-18 | 2020-06-17 | Sony Corporation | Information processing device, information processing method, program, and information processing system |
US11823408B2 (en) * | 2020-03-13 | 2023-11-21 | Oregon State University | Apparatus and method to quantify maize seed phenotypes |
CN113935963B (en) * | 2021-10-08 | 2022-05-13 | 广东省农业科学院果树研究所 | Image recognition detection method and system for litchi embryo development degree |
Family Cites Families (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US3437022A (en) * | 1965-10-04 | 1969-04-08 | Nathan Hamonds Jr | Cross-sectional determinant |
US8073235B2 (en) * | 2007-08-13 | 2011-12-06 | Pioneer Hi-Bred International, Inc. | Method and system for digital image analysis of ear traits |
DE102009055626A1 (en) * | 2009-11-25 | 2011-05-26 | Ralph Klose | Optical measuring device for optical measurement of maize plant, has evaluation device evaluating distance signal of camera pixels to determine three-dimensional enveloped surface of measuring object to be detected in measuring region |
US8563934B2 (en) * | 2010-09-10 | 2013-10-22 | Mississippi State University | Method and detection system for detection of aflatoxin in corn with fluorescence spectra |
CA2861591A1 (en) | 2011-12-30 | 2013-07-04 | Pioneer Hi-Bred International, Inc. | Immature ear photometry in maize |
HUE040105T2 (en) * | 2013-12-10 | 2019-02-28 | Shakti | Vorrichtung und verfahren zur abbildung eines objekts |
US9878842B2 (en) * | 2013-12-23 | 2018-01-30 | Dow Agrosciences Llc | Plant imaging and spectral scanning system and method |
WO2016123525A1 (en) * | 2015-01-30 | 2016-08-04 | Raytheon Company | Apparatus and methods for classifying and counting corn kernels |
WO2017021285A1 (en) | 2015-07-31 | 2017-02-09 | Biogemma | Method for imaging a corn ear and apparatus for implementing such method |
-
2017
- 2017-06-14 FR FR1755391A patent/FR3067824B1/en active Active
-
2018
- 2018-06-01 WO PCT/FR2018/000153 patent/WO2018229360A1/en unknown
- 2018-06-01 EP EP18731889.4A patent/EP3638978A1/en not_active Withdrawn
- 2018-06-01 US US16/622,762 patent/US10873699B2/en active Active
Also Published As
Publication number | Publication date |
---|---|
FR3067824A1 (en) | 2018-12-21 |
FR3067824B1 (en) | 2019-07-19 |
US20200162668A1 (en) | 2020-05-21 |
US10873699B2 (en) | 2020-12-22 |
WO2018229360A1 (en) | 2018-12-20 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
BE1022488B1 (en) | TIME-OF-FLIGHT VIEWING APPARATUS SYSTEM | |
FR3067824B1 (en) | METHOD AND DEVICE FOR IMAGING A PLANT | |
EP0279730A1 (en) | Method and device for tridimensional measurement | |
FR2975205A1 (en) | METHOD AND SYSTEM FOR ULTRASONIC IMAGING WITH CUTTING PLAN IMAGES | |
EP1633243B1 (en) | Method and device for recording and processing images of an object such as a tooth | |
JP2018004645A (en) | Determination device of degree of ripeness, and method for determining degree of ripeness | |
CA2315357A1 (en) | Analysing device non-destructive of plants and vehicle comprising such device on board | |
EP3698554B1 (en) | Imaging system for environmental acoustic sources | |
FR2587191A1 (en) | METHOD AND DEVICE FOR MEASURING DIRECTION OF LOOK | |
EP3627210A1 (en) | System for viewing in virtual or augmented reality with eye image sensor, and associated method | |
WO2019158839A1 (en) | System for indicating the crossing of a loudness threshold | |
CA3111361C (en) | Method for ultrasound detection and characterisation of defects in a heterogeneous material | |
FR3031035B1 (en) | IMAGING DEVICE AND IMAGING METHOD | |
FR2710564A1 (en) | Device for recognizing and / or sorting fruits or vegetables, method and use thereof. | |
WO2019197297A1 (en) | Method for generating a 3d model of a dental arch | |
JP2020073862A (en) | Water quality inspection system and water quality inspection method | |
EP0604302B1 (en) | Method and apparatus for analysis of moving objects by radiography | |
WO2021048511A1 (en) | Device and method for detecting tear film breakup | |
EP3304038B1 (en) | Device with a single camera and measuring method for characterising rain drops | |
KR101771234B1 (en) | Non-destructive Measurement System for Sugar Content of Fruit Using Laser Speckle Photometry and its method therefor | |
FR3034234A1 (en) | IMAGING A MACROSCOPIC VEGETABLE OBJECT | |
WO2018109044A1 (en) | Device for detecting an object of interest and method implementing this device | |
FR2807163A1 (en) | Monitoring system for goods on conveyor, uses two cameras at inclined axes taking images for analysis to produce three-dimensional model | |
EP3468451B1 (en) | Monitoring of the quality of the sleep of a baby by combination of multimodal data including photoplethysmography | |
FR3051341A1 (en) | MEDICAL WIRELESS DEVICE FOR ACQUIRING SKIN VIDEOS BIMODALITY WITH LIGHT CONTROL |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: UNKNOWN |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20200113 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20210614 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN |
|
18D | Application deemed to be withdrawn |
Effective date: 20211026 |