EP3226672A1 - Système et procédé d'ajustement automatique de la hauteur d'un outil agricole au moyen d'une reconstruction 3d - Google Patents
Système et procédé d'ajustement automatique de la hauteur d'un outil agricole au moyen d'une reconstruction 3dInfo
- Publication number
- EP3226672A1 EP3226672A1 EP15804457.8A EP15804457A EP3226672A1 EP 3226672 A1 EP3226672 A1 EP 3226672A1 EP 15804457 A EP15804457 A EP 15804457A EP 3226672 A1 EP3226672 A1 EP 3226672A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- height
- arm
- camera
- reconstruction
- scene
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims description 7
- 238000005259 measurement Methods 0.000 claims abstract description 52
- 238000003384 imaging method Methods 0.000 claims abstract description 28
- 238000012545 processing Methods 0.000 claims abstract description 21
- 240000008042 Zea mays Species 0.000 claims description 38
- 235000002017 Zea mays subsp mays Nutrition 0.000 claims description 37
- 241000196324 Embryophyta Species 0.000 claims description 31
- 235000005824 Zea mays ssp. parviglumis Nutrition 0.000 claims description 21
- 235000005822 corn Nutrition 0.000 claims description 21
- 239000011159 matrix material Substances 0.000 claims description 6
- 235000016383 Zea mays subsp huehuetenangensis Nutrition 0.000 description 16
- 235000009973 maize Nutrition 0.000 description 16
- 238000011144 upstream manufacturing Methods 0.000 description 9
- 238000004364 calculation method Methods 0.000 description 5
- 230000008030 elimination Effects 0.000 description 5
- 238000003379 elimination reaction Methods 0.000 description 5
- 238000004519 manufacturing process Methods 0.000 description 5
- 230000033001 locomotion Effects 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 230000018109 developmental process Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000003306 harvesting Methods 0.000 description 3
- 239000002184 metal Substances 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000000630 rising effect Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 230000010165 autogamy Effects 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000017260 vegetative to reproductive phase transition of meristem Effects 0.000 description 2
- 241001057636 Dracaena deremensis Species 0.000 description 1
- 230000001594 aberrant effect Effects 0.000 description 1
- 230000009418 agronomic effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000009395 breeding Methods 0.000 description 1
- 230000001488 breeding effect Effects 0.000 description 1
- 235000013339 cereals Nutrition 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000001010 compromised effect Effects 0.000 description 1
- 238000011109 contamination Methods 0.000 description 1
- 210000005069 ears Anatomy 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000004438 eyesight Effects 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 210000004392 genitalia Anatomy 0.000 description 1
- 230000004313 glare Effects 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 230000001939 inductive effect Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000001681 protective effect Effects 0.000 description 1
- 230000005855 radiation Effects 0.000 description 1
- 230000009257 reactivity Effects 0.000 description 1
- 230000001932 seasonal effect Effects 0.000 description 1
- 238000010396 two-hybrid screening Methods 0.000 description 1
- 238000002604 ultrasonography Methods 0.000 description 1
- 239000011800 void material Substances 0.000 description 1
Classifications
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01D—HARVESTING; MOWING
- A01D75/00—Accessories for harvesters or mowers
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01B—SOIL WORKING IN AGRICULTURE OR FORESTRY; PARTS, DETAILS, OR ACCESSORIES OF AGRICULTURAL MACHINES OR IMPLEMENTS, IN GENERAL
- A01B63/00—Lifting or adjusting devices or arrangements for agricultural machines or implements
- A01B63/02—Lifting or adjusting devices or arrangements for agricultural machines or implements for implements mounted on tractors
- A01B63/10—Lifting or adjusting devices or arrangements for agricultural machines or implements for implements mounted on tractors operated by hydraulic or pneumatic means
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01B—SOIL WORKING IN AGRICULTURE OR FORESTRY; PARTS, DETAILS, OR ACCESSORIES OF AGRICULTURAL MACHINES OR IMPLEMENTS, IN GENERAL
- A01B63/00—Lifting or adjusting devices or arrangements for agricultural machines or implements
- A01B63/002—Devices for adjusting or regulating the position of tools or wheels
- A01B63/008—Vertical adjustment of tools
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01B—SOIL WORKING IN AGRICULTURE OR FORESTRY; PARTS, DETAILS, OR ACCESSORIES OF AGRICULTURAL MACHINES OR IMPLEMENTS, IN GENERAL
- A01B63/00—Lifting or adjusting devices or arrangements for agricultural machines or implements
- A01B63/02—Lifting or adjusting devices or arrangements for agricultural machines or implements for implements mounted on tractors
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01D—HARVESTING; MOWING
- A01D47/00—Headers for topping of plants, e.g. stalks with ears
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/593—Depth or shape recovery from multiple images from stereo images
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/10—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used
- B60R2300/107—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used using stereoscopic cameras
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
- B60R2300/303—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing using joined images, e.g. multiple camera images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
Definitions
- the field of the invention is that of agricultural robotic systems.
- the invention relates more particularly to a system for automatically adjusting the position of an agricultural implement, such as a tool for cutting or tearing plants. It finds particular application in the automation of mechanical castration operations of seed corn production fields.
- Corn is an autogamous plant strongly subject to the effect of heterosis, that is to say to the improvement of the capacities and the vigor of the plant in hybrid or heterozygous form with respect to the plant in homozygous form. Breeders are therefore seeking to cross two homozygous spawning lines in order to obtain a more productive and vigorous hybrid combination with interesting agronomic characteristics.
- the corn plant is monoecious, consisting of a male inflorescence, called the panicle which is at the top of the plant and is responsible for the emission of pollen, and a female inflorescence located at mid-height of the plant, at the base of a leaf, this female inflorescence being composed of receptor stigmas or silks which are fertilized by the pollen, and lead to the formation of the ear.
- Hybrid seed production is carried out by growing in the same plot, usually on interspersed rows, plants from both parent lines of the hybrid.
- the panicle of the female genital line is removed before flowering for these plants to be fertilized by the pollen of the male breeding line.
- the ears harvested on this female spawning line thus carry the hybrid seeds that are marketed.
- Hybrid production is not limited to obtaining simple hybrids from two maize lineages, and may consist of a cross between a hybrid and a line (three-way hybrid) or at the crossroads of two hybrids (double hybrid).
- the topping of the male panicle on female broodstock before flowering is thus an essential step in the process of producing hybrid maize seed. Indeed, the presence of pollen even in small quantity leads to self-fertilization on these female spawners and contamination in the production of hybrids. This topping was traditionally done manually. The labor cost generated by this step being important, we sought to mechanize this step with various tools, including knives for cutting the crowns of plants and rollers for the panicle or the horn.
- the process of mechanical castration can be performed with one or more knife passages only, or in two stages.
- a knife crossing is performed on the female breeder ranks: rotating blades cut the top of the male plant.
- the ideal cutting height is three-quarters of the horn, that is, the set of leaves surrounding the panicle, from its top or three-quarters of the panicle if it exceeds the horn.
- the objective is to catch the panicle with the rollers or the tires by pulling as little as possible the cornet or the sheets which would be still around.
- the purpose of the first cut is to clear the row so that the panicle that grows faster than the rest of the plant will grow out of the foliage.
- the objective of the second cut is to tear off the rest of the panicle and thus eliminate any risk of self-fertilization.
- the farmer visually assesses the optimum cutting height of the tassel, from the nacelle of his machine and controls the height of the cutting or ripping tools.
- two rows are cut simultaneously at the same height which saves time but may penalize the cutting of one of the two rows if the plants are not of uniform height.
- the quality of the cut is therefore limited by the reactivity of the driver but also by the accuracy that allows the current architecture of the machine.
- many panicles are not cut to the correct height. At the end of the two stages of mechanical castration, only 70 to 80% of the panicles are totally eliminated.
- a system proposed by SARL Duissard is also known, based on two batches of transmitting / receiving photocells placed upstream of the cutting tools and located 13 cm one below the other, on either side. two rows. These cells operate in direct detection: an infrared beam is emitted by the photocell while a reflector, located opposite, reflects the beam to the cell that receives it. If corn is on the path of the beam, then it will not be received by the photocell. The maximum height is detected when the lower cell sees corn (beam obstructed by the corn and therefore not perceived by the lower cell) and that the upper cell does not see (beam perceived by the upper cell), with precision the distance between the two photocells is 13 cm.
- This system sends an instruction "all or nothing" to the actuator (up or down) because of the presence of only two photocells.
- the system therefore constantly tries to adjust the cutting height with repeated instructions and sometimes contrary which solicits strongly.
- the entire arm will descend to have to go back immediately to the level of the next foot.
- the entire system will descend and probably miss the first few feet of the next row.
- the degree of precision is also limited by the vertical spacing between the two lots of photocells, 13 cm. Cells also process information for two rows simultaneously so the lowest rank cornstalk will be cut too high. Finally, the congestion of the overall system remains a problem for farmers.
- the aim of the invention is to improve the existing systems for automating plant cutting or tearing operations, and to this end proposes a system for adjusting the height of an agricultural implement, comprising an arm capable of being controlled for allow to raise and lower the agricultural tool, a height measuring sensor, and a computer configured to provide an instruction for controlling the height of the arm from the measurements made by the height measuring sensor, characterized in that the height measuring sensor is an imaging system comprising at least one camera mounted so as to be able to image a scene located in front of the agricultural tool in a traveling direction of a vehicle equipped with the arm, and a computer processing unit configured to produce, from the images delivered by the at least one camera, a 3D reconstruction representative of the relief of the imaged scene.
- the imaging system is a stereoscopic imaging system comprising two cameras mounted so as to be able to image the same scene from two distant points of view, and the computer processing unit is configured to develop the 3D reconstruction from a couple of stereoscopic images delivered by the cameras; the at least one camera of the imaging system is a flight time camera, and the computer processing unit is configured to perform a measurement of flight time between the imaged scene and the camera.
- the 3D reconstruction provides information representative of the relief of the image scene in matrix form in rows and columns, and the computer processing unit is further configured to identify for each line or line portion of the 3D reconstruction the highest point ;
- the computer processing unit is configured to identify one or more rows of plants in the 3D reconstruction
- the computer furthermore comprises a memory in which the measurements made by the height measurement sensor are recorded, the computer being configured to supply the instruction for controlling the height of the arm on the basis of measurements previously recorded in the memory and corresponding to several 3D reconstructions of pictorial scenes of which at least one point is at a distance from the current position of the lower cutting tool at a predetermined distance. It further comprises a sensor for measuring the speed of travel in the direction of travel of the vehicle, and the computer is configured to determine the distance traveled along the direction of travel between two 3D reconstructions successively developed by the unit of travel. data processing ;
- the computer is configured to provide the control command of the height of the arm from the average of said measurements previously stored in the memory;
- the computer is configured to provide the control command of the height of the arm from the maximum of said measurements previously stored in the memory;
- the calculator is configured to calculate a leaf density indicating, for each pitch interval of a set of successive intervals, the percentage of measurements among said previously stored measurements in the memory which indicate a height greater than the lower limit of the interval height, and to provide the control command of the arm height from the lower bound of a height interval corresponding to a threshold percentage.
- FIG. 1 is a schematic perspective view of a system according to a possible embodiment of the invention.
- FIG. 2 is a diagram illustrating the geometric model of a stereoscopic imaging system
- FIG. 5 is a diagram showing a leaf density that can be determined in a possible embodiment of the invention.
- the invention relates to a system for adjusting the height of an agricultural implement. It is generally applicable to any area where height measurement and adjustment is required, especially for tools for cutting or removing plants such as those used for corn castration or for cereal harvesting.
- the invention relates to a system for adjusting the height of an agricultural implement 0.
- This system is formed of an arm 1 that can be controlled in order to raise and lower the agricultural tool, a height measurement sensor 2, and a computer configured to provide an instruction for controlling the height of the arm from the measurements made by the height measurement sensor.
- the agricultural tool is typically a cutting tool (for example a knife cutting tool) or tearing (for example tires or rollers) of plants.
- a cutting or tearing instrument for example tires or rollers
- Such a tool generally comprises a cutting or tearing instrument (knives, tires or rollers), an instrument drive motor, a vegetation guide and a protective cover.
- the vehicle used is a straddle which allows to circulate between the rows of maize and to cut or tear several rows female spawners contiguous at a time.
- a conventional straddle generally has two or four arms upstream of the vehicle, each arm carrying two cutting or tearing tools to castrate simultaneously two adjacent rows at the same height, which saves time but can penalize the cut of one or two rows, the two rows being cut at the same height.
- only one cutting or arm picking tool is used to individualize the cut per crop row.
- the arm 1 comprises a rear vertical upright 11 forming the main structural connection with the agricultural vehicle, and a vertical upright 12 which carries at its lower end a horizontal bar of agricultural tool support 13 which extends perpendicularly to the direction of travel and on which is mounted the agricultural tool 0.
- the arm can carry several agricultural tools, typically two, for carrying out simultaneous operations on several rows of culture.
- the uprights 11 and 12 are interconnected by connecting uprights 14 and 15 pivotally mounted relative to the rear vertical upright 11.
- a jack 16 is mounted between the rear upright 11 and the upright 14, and provides means for adjusting the height of the connecting post 14, and thereby the height of the vertical upright 12 before and the cutting tool O, by pivoting the connecting post 14 relative to the vertical upright 11.
- the height measurement sensor 2 is an imaging system comprising at least one camera mounted so as to be able to image a scene located in front of the agricultural tool in the direction of travel of the vehicle equipped with the arm.
- the imaging system further comprises a computer processing unit configured to develop, from the images delivered by the at least one camera, a 3D reconstruction representative of the relief of the pictorial scene.
- the computer processing unit can be deported from the at least one camera.
- the imaging system is a time-of-flight system including a flight time camera and a computer processing unit configured to perform a flight time measurement between the imaged scene and the camera and allow 3D reconstruction.
- Such a measurement system delivers in real time and at a rate of several tens of Hz a 3D map of the observed environment. It is an active system that illuminates the scene from an infrared flash imperceptible to man but captured by the camera.
- the sent light source is reflected by the surfaces present in the scene and is captured by the pixel matrix of the camera.
- the computer processing unit calculates the return time of the tap on each pixel, generally by a phase offset calculation.
- the return time corresponds to the travel time of return / return of the light wave between the camera and the surface.
- This technology is very similar to Lidar technologies (scanning laser scanner) with the essential advantage of having a 3D image whose pixels are acquired simultaneously, which is advantageous in the context of the invention where the system of measurement in motion.
- the imaging system is a stereoscopic system comprising two cameras mounted so as to be able to image, from two distant points of view, the same scene located in front of the agricultural tool in the direction of travel of the vehicle equipped with the arm. Both cameras are worn by a stereoscopic head.
- the computer processing unit is then configured to develop the 3D reconstruction.
- the calibration step is made prior to the reconstruction campaign and remains valid as long as the mechanical stability of the stereo head is guaranteed.
- the following steps are performed at each moment to produce a new 3D reconstruction of the environment to take into account a change in either the vehicle position or the observed surface.
- the calibration step can be performed by observing a known object (a pattern) from different angles and viewed simultaneously by the two cameras.
- a known object a pattern
- the mapping on each image pair of several characteristic points of the pattern makes it possible to estimate the projection matrix of each camera (intrinsic parameters) as well as the geometric transformation connecting the two cameras (extrinsic parameters).
- the parameters resulting from the calibration make it possible to apply a correction function which aligns the lines of the left camera with those of the right camera: a point P of the environment projected on the line i of the camera 1 will be projected on the line i of the camera 2.
- This operation makes it possible to accelerate and make more robust the pairing between the points of the two images corresponding to the same 3D point.
- the chosen transformation must take care to preserve the quality of the signal contained in the image (minimization of smoothing and interpolation).
- An example of this type of algorithm is presented in the article by Loop and Chang entitled Computing rectifying homographies for stereo vision Int. CVPR Conference, 1999.
- a mapping of the images 11 and 12 of the pair of stereoscopic images provided by the cameras is then performed, the purpose of which is to find the homologous points pl, p2 between the two images, that is, ie the projections of the same points P of the pictorial scene. It is thus identified that the point pl (ul, v1) in the left image 11 and the point p2 (u2, v2) in the right image 12 are the projection of the same point P of the image scene.
- the X, Y, Z coordinates of this point P can then be calculated.
- the first image 11 is scanned and for each pixel of this image, the second image 12 acquired at the same time is searched for the pixel corresponding to the same physical point. Thanks to the correction, this pixel is on the same line of the second image 12 as the original pixel of the first image 11.
- This constraint reduces the search times and the risks of error.
- To perform the search we consider a neighborhood of the original pixel and we search along the line for the pixel having the most similar neighborhood. Different correlation scores have been proposed in the literature, compromised between quality and speed. This search is iterated for all the pixels of the line of the original image, then to all the lines of this same image.
- the corresponding 3D point can be reconstructed by triangulation.
- the reconstructed 3D points correspond to points of the canopy. It is thus possible to reconstruct a surface based on these reconstructed 3D points and thus to model the canopy of the pictorial scene.
- this surface can be analyzed to determine the ridges connecting the vertices of the feet belonging to the same rank and the valleys corresponding to the inter-row space.
- the reconstructed 3D surface being expressed in the reference of the at least one camera and can therefore be expressed in the reference system of the vehicle, the desired ridge lines and valleys are in a direction parallel to the movement of the vehicle.
- the imaging system can thus be adapted to allow simultaneously imaging and measuring the height of several rows of culture.
- the 3D reconstruction thus provides an information representative of the relief of the scene imaged in matrix form in rows and columns where each point of the matrix provides information representative of the distance between the imaging system (more precisely the midpoint between the cameras on baseline 3 in the case of a stereoscopic system) and the top of the plants present in the pictorial scene.
- the computer processing unit can be configured to identify for each line or line portion of the 3D reconstruction the highest point, or calculate for each line or line portion of the 3D reconstruction the average of the points present on the line or line portion of the 3D reconstruction. line portion, so as to provide the height measurement information.
- the at least one camera of the imaging system is mounted so as to be stationary relative to the agricultural vehicle. As represented in FIGS. 1 and 3, it can notably be mounted on the upright vertical rear 11 of the arm 1 which serves as the main structural connection with the agricultural vehicle.
- the position of the at least one camera of the imaging system 2 is known in the absolute reference of the vehicle and more particularly its absolute height h absolute camera which is constant.
- the reconstitution of the 3D surface of the canopy at the level of the seedlings upstream of the tool 0 makes it possible to know the relative height h relative corn of the top of the plants with respect to the camera.
- the absolute height h absolute maize of the top of the plants with the relation:
- the at least one camera of the imaging system is mounted so as to be fixed with respect to the cutting tool 0. As shown in FIG. be mounted on the top of the vertical upright before 12 of the arm 1.
- the camera's absolute absolute height varies. Thanks to the reconstruction of the 3D surface of the canopy, we know the relative relative height h but the top of the corn compared to the camera. Thus, we can deduce the absolute height h absolute but the top of the plants with the relation:
- FIG. 2 A simplified model of the arm with tool (s) 0 and the height measurement sensor upstream in the direction of travel is proposed in FIG. 2.
- the acquisitions are carried out for example every millisecond.
- the setpoint is calculated for example every 100 ms, from a number of acquisitions preceding the calculation.
- An inverter coupled to a motor can be used to control voltage the extension of the cylinder 16, here electrical.
- the voltage sent to the drive is connected to the motor angular position by the relation:
- U max drive the maximum voltage that can be sent to the drive (10V for example), C cylinder the stroke of the electric cylinder (300 mm for example) and p the pitch of the motor screw (10mm / rev for example).
- the invention extends however to other types of cylinders, including hydraulic cylinders.
- the elongation of the jack 16 is recovered by means of an angular position sensor, for example a resolver, then in the case of FIG. 4, it is necessary to find the absolute height h absolute tools of the tool 0 by geometry to deduce absolute height absolute h but maize.
- the absolute heights of the maize can be treated according to different strategies, examples of which will be presented below, to obtain the cutting height setpoint, that is to say the setpoint of absolute height of the tools. The latter is then converted into cylinder extension and voltage which is sent to the drive.
- the objective is first of all to know the relation between the extension of the cylinder and the absolute height of the tools as well as that between the absolute height of the tools and the absolute height of the maize, in order to be able to apply the different strategies to the absolute heights corn.
- the motor position sensor makes it possible to know the number of increments n increments of the motor driving the jack 16 which is to be converted into the engine position ⁇ engine and then to the elongation of cylinder x elongation thanks to the formulas:
- the objective of the cut with the knives is to clear the view on the row by cutting just enough leaves and panicles so that they can still push back later.
- the cut must therefore be relatively homogeneous, without creating too many irregularities.
- the ideal cutting height is three-quarters of the panicle. This involves identifying the height of the base and top of the panicle to locate three quarters, which is difficult to reproduce with sensors, especially since the panicle can be hidden in the cornet.
- imposing a cutting height to a certain number of centimeters below the top of the foot is not suitable because the panicle may as well be close to this vertex as much lower. It is therefore necessary to obtain a relative measure, adapted to each foot of corn and to different architectures of plants (upright growth, drooping leaves, ).
- Each strategy can be based on the fact that the elapsed time between two 3D reconstructions developed by the height measurement imaging system is converted into distance traveled thanks to the speed calculated at each moment.
- the imaging system developing 3D reconstructions of scenes located upstream of the tool or tools, the instruction to be sent to the actuator at a time t is not calculated thanks to the reconstruction developed at the same time t but thanks to the reconstructions earlier.
- the reconstructions are memorized progressively and then the setpoint is calculated from a small sample of reconstructions of pictorial scenes of which at least one point is at a distance from the current position of the cutting tool, in the vehicle travel direction, less than a predetermined distance from Ech .
- the setpoint sent is thus calculated from the previous reconstructions carried out while the imaging system imaged the canopy over a distance of Ech in the direction of travel of the vehicle upstream of the current position of the tool or tools.
- the system according to the invention may furthermore comprise a memory in which the measurements made by the height measuring sensor are recorded, and the computer may be configured to supply the control instruction of the height of the arm on the basis of measurements previously recorded in the memory and corresponding to several 3D reconstructions of pictorial scenes of which at least one point is at a distance from the current position of the cutting tool less than the predetermined distance of Ech .
- the system may comprise a sensor for measuring the speed of travel in the direction of travel of the vehicle, and the computer is then configured to determine the distance traveled along the direction of travel between two measurements successively made by the height measurement sensor.
- the speed measuring sensor may consist of an inductive proximity sensor associated with a plastic wheel fixed in the rim of a wheel of the vehicle and on the perimeter of which metal pins are arranged. At each passage in front of a metal pin, the signal at the output of the sensor is modified. Thus, the speed of advance can be deduced by calculating the time elapsed between two rising edges of the output signal, that is to say between two metal pins, knowing the length of the rope between these two points.
- the maximum absolute height of the absolute maize corn is memorized . Then we calculate the setpoint absolute height tools h to which the tools must come cut the plant.
- This calculation can be done in different ways depending on the chosen strategy.
- the computer is configured to provide the control command of the height of the arm from the average of said measurements previously made by the height measurement sensor on said predetermined distance Ech .
- the computer is configured to provide the control command of the height of the arm from the maximum of said measurements previously made by the height measurement sensor over said predetermined distance Ech .
- This embodiment corresponds to a so-called "maximum” strategy according to which for the acquisitions of Ech , one identifies the maximum height among the heights of corn on this sample.
- the setpoint sent absolute tools is then equal to this maximum height at which we can subtract a height h 0ffset that depends on the variety cut.
- This strategy is more for pulling up the rollers. It has indeed been verified that the panicles measuring about 10 mm in diameter are not detectable by the light curtain, only the leaves are detected by drawing a relatively straight profile after cutting the knives. The objective is here to come and place the rollers just at the top of this profile, knowing that everything that exceeds can only be the tassel and will be torn off.
- the computer is configured to calculate a leaf density indicating, for each height interval of a set of successive intervals, the percentage of measurements among said measurements previously made by the height measurement sensor. said predetermined distance of Ech indicating a height greater than the lower limit of the height interval, and for providing the control command of the arm height from the lower limit of the height interval corresponding to a threshold percentage.
- This embodiment corresponds to a so-called "leaf density" strategy according to which for Ech acquisitions, the minimum height among the corn heights on this sample is identified. Then a loop is created that defines a current height equal to the minimum height incremented by 5 mm for example at each loop turn and the percentage of acquisitions that correspond to a height greater than this new current height is calculated. Thus, for the minimum height of the sample, 100% of the sample acquisitions have a height greater than the minimum height. Then as the current height increases, fewer and fewer acquisitions have a height greater than the current height. This percentage corresponds to what is called a leaf density. FIG. 5 shows an example of leaf density that can be determined in this embodiment.
- the ordinate is the height of the plants h relative corn and the abscissa the percentage plants higher than this height h relative maize .
- a threshold S from which it is estimated that the leaf density is sufficiently low (30% in the example of Figure 5) to correspond to the optimal cutting height: in fact, the closer we get to the top of the plant the higher the leaf density is supposed to decrease. It will be noted that conversely the leaf density can be estimated starting from the maximum height and coming to verify the percentage of acquisitions having a lower height.
- the setpoint sent to absolute tools is then equal to the current height corresponding to this threshold S (around + 175mm in the example of Figure 5), to which we can subtract a height h 0ffset that depends on the cut variety.
- This strategy is intended to be used for knife cutting. It is closer to the ideal measure of three quarters of panicles: indeed, we imagine that at this ideal height there is only a certain percentage of leaves. In addition, it takes into account the architecture of the plant: for a variety with drooping leaves, there will be more leaves around the panicle than for an upright variety.
- the invention is not limited to the system as previously described but also extends to a vehicle, including a straddle, equipped with one or more systems according to the invention. It also extends to a method of adjusting the height of an agricultural implement by means of a system comprising an operable arm for raising and lowering the agricultural implement, comprising the steps of:
- the height measurement sensor being an imaging system comprising at least one camera mounted so as to be able to image the same scene located in front of the agricultural tool in a travel direction of a vehicle equipped with the arm, and a computer processing unit configured to produce, from the images delivered by the at least one camera, a 3D reconstruction representative of the relief of the imaged scene;
- the invention provides a gain in precision by coming to determine a height set quantified, accurate and adapted to each foot of corn through the development of calculation algorithms adapted to the stakes of cutting and tearing.
- the invention also provides a gain in productivity since the system can be reproduced on several arms to cut or tear several rows simultaneously, independently of each other.
- this system makes it possible to reduce the hardness of the work of the farmer, who no longer has to constantly watch for corn heights and adjust the height of the tools.
- this system improves the profitability of the mechanical castration operations and reduces the number of remaining panicles to tear by hand.
Landscapes
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Environmental Sciences (AREA)
- Mechanical Engineering (AREA)
- Soil Sciences (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Signal Processing (AREA)
- Length Measuring Devices By Optical Means (AREA)
- Harvester Elements (AREA)
- Image Processing (AREA)
- Guiding Agricultural Machines (AREA)
Abstract
Description
Claims
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
FR1462003A FR3029389B1 (fr) | 2014-12-05 | 2014-12-05 | Systeme et procede d'ajustement automatique de la hauteur d'un outil agricole au moyen d'une reconstruction 3d |
PCT/EP2015/078386 WO2016087529A1 (fr) | 2014-12-05 | 2015-12-02 | Système et procédé d'ajustement automatique de la hauteur d'un outil agricole au moyen d'une reconstruction 3d |
Publications (3)
Publication Number | Publication Date |
---|---|
EP3226672A1 true EP3226672A1 (fr) | 2017-10-11 |
EP3226672B1 EP3226672B1 (fr) | 2018-11-28 |
EP3226672B2 EP3226672B2 (fr) | 2022-07-27 |
Family
ID=52423997
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP15804457.8A Active EP3226672B2 (fr) | 2014-12-05 | 2015-12-02 | Système et procédé d'ajustement automatique de la hauteur d'un outil agricole au moyen d'une reconstruction 3d |
Country Status (4)
Country | Link |
---|---|
US (1) | US10172289B2 (fr) |
EP (1) | EP3226672B2 (fr) |
FR (1) | FR3029389B1 (fr) |
WO (1) | WO2016087529A1 (fr) |
Families Citing this family (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10219449B2 (en) * | 2013-03-07 | 2019-03-05 | Blue River Technology Inc. | System and method for plant dislodgement |
FR3029388B1 (fr) * | 2014-12-05 | 2017-01-13 | Commissariat Energie Atomique | Systeme et procede d'ajustement automatique de la hauteur d'un outil agricole au moyen d'un rideau lumineux de mesure |
US10342176B2 (en) * | 2016-07-13 | 2019-07-09 | Monsanto Technology Llc | Angled sensor bar for detecting plants |
WO2019025827A1 (fr) | 2017-08-03 | 2019-02-07 | Vilmorin & Cie | Outil agricole et véhicule agricole comprenant un tel outil agricole |
EP3836783A4 (fr) | 2018-08-13 | 2022-05-04 | Farmwise Labs, Inc. | Procédé de détection autonome d'emplacement de récolte sur la base de la profondeur et de l'emplacement d'un outil |
CN110892819B (zh) * | 2018-12-27 | 2021-09-10 | 江苏大学 | 一种头季再生稻的多层分段-留茬割刀装置及控制方法和头季再生稻的联合收获机 |
US11622494B2 (en) | 2019-05-10 | 2023-04-11 | Great Plains Manufacturing, Inc. | Tillage implement with vision sensors |
CN110542933B (zh) * | 2019-09-24 | 2021-03-09 | 吉林大学 | 一种高秆作物收获盲区人畜热敏探测装置 |
US11503756B2 (en) | 2019-09-25 | 2022-11-22 | Cnh Industrial America Llc | System and method for determining soil levelness using spectral analysis |
US11930726B2 (en) | 2020-10-26 | 2024-03-19 | Deere & Company | Machine-vision system for tracking and quantifying missed tassel during a detasseling operation |
US11980130B2 (en) * | 2020-10-26 | 2024-05-14 | Deere & Company | Machine-vision system for automated adjustment of a detasseler machine |
Family Cites Families (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
AU5387473A (en) * | 1973-03-15 | 1974-10-03 | Nicholls John | Harvester improvement |
US4197691A (en) | 1978-09-25 | 1980-04-15 | Hagie Manufacturing Co. | Detasseling device depth adjusting control system and method |
US4507910A (en) | 1983-11-21 | 1985-04-02 | Ezra C. Lundahl, Inc. | Automatic sonar activated height control for a header |
DE4411646A1 (de) † | 1994-04-02 | 1995-11-02 | Bernhardt Gerd Prof Dr Ing Hab | Verfahren und Anordnung zur Bestimmung der geometrischen Eigenschaften von Objekten vor einem Fahrzeug |
US6615570B2 (en) † | 2001-06-28 | 2003-09-09 | Deere & Company | Header position control with forward contour prediction |
DE10212722B4 (de) | 2002-03-21 | 2008-07-10 | Gebr. Pöttinger GmbH | Erntemaschine |
US7765780B2 (en) | 2003-12-12 | 2010-08-03 | Vision Robotics Corporation | Agricultural robot system and method |
WO2009149473A2 (fr) * | 2008-06-06 | 2009-12-10 | Martin R Steven | Nettoyeur de rang de plantation réglable |
US8666550B2 (en) † | 2010-01-05 | 2014-03-04 | Deere & Company | Autonomous cutting element for sculpting grass |
US8381502B2 (en) * | 2011-06-06 | 2013-02-26 | Walter Dunn | Cut sight gauge |
US8452501B1 (en) | 2011-11-09 | 2013-05-28 | Trimble Navigation Limited | Sugar cane harvester automatic cutter height control |
US9064173B2 (en) * | 2012-03-07 | 2015-06-23 | Blue River Technology, Inc. | Method and apparatus for automated plant necrosis |
EP2679085A1 (fr) * | 2012-06-26 | 2014-01-01 | Norac Systems International Inc. | Commande de hauteur |
BE1021123B1 (nl) † | 2013-01-14 | 2015-12-14 | Cnh Industrial Belgium Nv | Kalibreren van een afstandssensor op een landbouwvoertuig |
US9668420B2 (en) * | 2013-02-20 | 2017-06-06 | Deere & Company | Crop sensing display |
-
2014
- 2014-12-05 FR FR1462003A patent/FR3029389B1/fr active Active
-
2015
- 2015-12-02 US US15/531,617 patent/US10172289B2/en active Active
- 2015-12-02 WO PCT/EP2015/078386 patent/WO2016087529A1/fr active Application Filing
- 2015-12-02 EP EP15804457.8A patent/EP3226672B2/fr active Active
Also Published As
Publication number | Publication date |
---|---|
FR3029389B1 (fr) | 2017-01-13 |
US10172289B2 (en) | 2019-01-08 |
EP3226672B1 (fr) | 2018-11-28 |
FR3029389A1 (fr) | 2016-06-10 |
WO2016087529A1 (fr) | 2016-06-09 |
US20180279556A1 (en) | 2018-10-04 |
EP3226672B2 (fr) | 2022-07-27 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP3226672B1 (fr) | Système et procédé d'ajustement automatique de la hauteur d'un outil agricole au moyen d'une reconstruction 3d | |
EP3322277B1 (fr) | Système robotique à deux bras pour l'ajustement de la hauteur d'un outil agricole | |
EP3226671B1 (fr) | Système et procédé d'ajustement automatique de la hauteur d'un outil agricole au moyen d'un rideau lumineux de mesure | |
RU2747303C2 (ru) | Система для управления рабочим параметром уборочной жатки | |
EP1608216B2 (fr) | Procede et dispositif d'analyse de la structure et de la constitution de rangs de vigne | |
FR2994057A1 (fr) | Robot de taille de vignes comprenant des moyens de captation d'images mettant en oeuvre des moyens de projection d'un faisceau laser | |
WO2015140471A1 (fr) | Machine de récolte de racines comme des betteraves, comportant un moyen d'ajustement automatique de l'unité de récolte | |
FR3001102A1 (fr) | Procede de pilotage d'un dispositif agricole automatise autonome | |
FR3036830A1 (fr) | Systeme et methode pour l'estimation d'un volume de recolte au sein d'une exploitation viticole | |
Xiang et al. | Field‐based robotic leaf angle detection and characterization of maize plants using stereo vision and deep convolutional neural networks | |
FR3098083A1 (fr) | Système et procédé de culture. | |
FR3086502A1 (fr) | Machine a planter des pieux avec une haute precision | |
EP2986981B1 (fr) | Systeme de caracterisation de l'etat physiologique de vegetaux et procede correspondant | |
Wang et al. | Design of crop yield estimation system for apple orchards using computer vision | |
WO2022223630A1 (fr) | Procédé de réglage et/ou de calibrage d'une machine agricole | |
FR3097096A1 (fr) | Machine pour la récolte de racines, comprenant un moyen de réglage automatique de l’unité de récolte | |
McCarthy | Automatic non-destructive dimensional measurement of cotton plants in real-time by machine vision | |
FR3099683A1 (fr) | Système et procédé de culture. | |
Anatole | Design of Crop Yield Estimation System for Apple Orchards Using Computer Vision | |
FR3030720A1 (fr) | Dispositif de detection d'un cordon de plante palissee |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20170606 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20180615 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 1069121 Country of ref document: AT Kind code of ref document: T Effective date: 20181215 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602015020583 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D Free format text: LANGUAGE OF EP DOCUMENT: FRENCH |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20181128 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1069121 Country of ref document: AT Kind code of ref document: T Effective date: 20181128 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190228 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190228 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190328 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190301 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190328 Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R026 Ref document number: 602015020583 Country of ref document: DE |
|
PLBI | Opposition filed |
Free format text: ORIGINAL CODE: 0009260 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20181202 Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 |
|
PLAX | Notice of opposition and request to file observation + time limit sent |
Free format text: ORIGINAL CODE: EPIDOSNOBS2 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
26 | Opposition filed |
Opponent name: DEERE & COMPANY/JOHN DEERE GMBH & CO. KG Effective date: 20190819 |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20181231 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20181202 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20181231 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20181231 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20181231 |
|
PLBB | Reply of patent proprietor to notice(s) of opposition received |
Free format text: ORIGINAL CODE: EPIDOSNOBS3 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181128 Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20151202 Ref country code: MK Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20181128 |
|
GBPC | Gb: european patent ceased through non-payment of renewal fee |
Effective date: 20191202 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20191202 |
|
PLAY | Examination report in opposition despatched + time limit |
Free format text: ORIGINAL CODE: EPIDOSNORE2 |
|
PLBC | Reply to examination report in opposition received |
Free format text: ORIGINAL CODE: EPIDOSNORE3 |
|
PUAH | Patent maintained in amended form |
Free format text: ORIGINAL CODE: 0009272 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: PATENT MAINTAINED AS AMENDED |
|
27A | Patent maintained in amended form |
Effective date: 20220727 |
|
AK | Designated contracting states |
Kind code of ref document: B2 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R102 Ref document number: 602015020583 Country of ref document: DE |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20231221 Year of fee payment: 9 Ref country code: DE Payment date: 20231219 Year of fee payment: 9 |