EP3714688B1 - Camera based pest management sprayer - Google Patents

Camera based pest management sprayer Download PDF

Info

Publication number
EP3714688B1
EP3714688B1 EP19165176.9A EP19165176A EP3714688B1 EP 3714688 B1 EP3714688 B1 EP 3714688B1 EP 19165176 A EP19165176 A EP 19165176A EP 3714688 B1 EP3714688 B1 EP 3714688B1
Authority
EP
European Patent Office
Prior art keywords
treatment
image
insects
insect population
insect
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP19165176.9A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP3714688A1 (en
Inventor
Ole Janssen
Bjoern Kiepe
Mirwaes Wahabzada
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
BASF Agro Trademarks GmbH
Original Assignee
BASF Agro Trademarks GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to ES19165176T priority Critical patent/ES2929076T3/es
Application filed by BASF Agro Trademarks GmbH filed Critical BASF Agro Trademarks GmbH
Priority to EP19165176.9A priority patent/EP3714688B1/en
Priority to HUE19165176A priority patent/HUE060153T2/hu
Priority to PCT/EP2020/058438 priority patent/WO2020193666A1/en
Priority to US17/598,273 priority patent/US12075770B2/en
Priority to JP2021557195A priority patent/JP2022526157A/ja
Priority to CA3135084A priority patent/CA3135084A1/en
Priority to CN202080023887.XA priority patent/CN113631035B/zh
Priority to EP20714206.8A priority patent/EP3945806A1/en
Priority to BR112021017157A priority patent/BR112021017157A2/pt
Publication of EP3714688A1 publication Critical patent/EP3714688A1/en
Application granted granted Critical
Publication of EP3714688B1 publication Critical patent/EP3714688B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
    • A01M—CATCHING, TRAPPING OR SCARING OF ANIMALS; APPARATUS FOR THE DESTRUCTION OF NOXIOUS ANIMALS OR NOXIOUS PLANTS
    • A01M7/00—Special adaptations or arrangements of liquid-spraying apparatus for purposes covered by this subclass
    • A01M7/0089—Regulating or controlling systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G06T7/0014—Biomedical image inspection using an image reference approach
    • A—HUMAN NECESSITIES
    • A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
    • A01M—CATCHING, TRAPPING OR SCARING OF ANIMALS; APPARATUS FOR THE DESTRUCTION OF NOXIOUS ANIMALS OR NOXIOUS PLANTS
    • A01M2200/00—Kind of animal
    • A01M2200/01—Insects

Definitions

  • the present invention relates to a method and a device for pest treatment of a plantation field having an improved efficiency and a sustainability with respect to beneficial population.
  • a common approach for treatment of plantations with respect to pest population is a uniform application of a pesticide, which is for example uniformly sprayed onto a plantation over an entire field.
  • the farmer may have a view on the plantation of the field and may evaluate upon inspection of singular plantation spots which pesticide is to be applied and which dosage of pesticide may be considered as reasonable.
  • the pest management gets increasingly under pressure, as side effects of insecticides or pesticides may be heavy and may reduce biodiversity in a field to a degree which is not acceptable. If reducing the amount of insecticides or pesticides, in turn yield losses may be heavy, if pest management is not done properly, which may lead to a productivity loss. farmers see this effect in oilseed rape or potato plantation.
  • the present invention seeks to overcome this deficiency.
  • the present invention provides a method for pest treatment of a plantation field, a controlling device for pest treatment and a treatment device for pest treatment of a plantation field according to the independent claims, whereas further embodiments are incorporated in the dependent claims.
  • a method for pest treatment of a plantation field comprises taking an image of a predetermined range of a plantation of the plantation field, recognizing items on the taken image of the predetermined range, identifying one or a plurality of patterns of an insect population based on the recognized items and a database having stored samples of an insect population, determining a measure of a quantity of insect population based on the identified patterns, controlling a treatment arrangement of a treatment device based on the determined quantity of an insect population, wherein controlling of a treatment arrangement includes activating a treatment unit if a quantity of a first type and/or a second type of pest insects extends a predetermined threshold, and blocking a treatment unit from being activated if a quantity of a particular type of beneficial insects extends a predetermined threshold.
  • the method may provide an automated decision on whether or not a treatment unit is to be activated or not, based on the recognized and identified insect population, considering the population of pest insects as well as the population of beneficial insects.
  • the method includes taking an image from the predetermined range of a plantation of a plantation field, which image may be taken from an image capturing device like for example a camera.
  • image capturing not only includes taking a static picture allowing a spatial resolved detection, but may also include time resolved imaging, e.g. a video sequence or a laser scanning of the plantation, which may allow a time resolved imaging.
  • Image capturing may be sensitive to visible light as well as infrared light. Image capturing may also include optical detection of particular parameters, as motion pattern and frequencies.
  • Images can also be acquired by a for example, a UAV (unmanned aerial vehicle) which can fly around a plantation field or a robotic land vehicle that moves around the plantation field and acquires the imagery.
  • UAV unmanned aerial vehicle
  • This image is provided, so that the method can be carried out based on the provided image.
  • taking an image does not mandatorily include the actual operation of a camera but may also mean taking an already generated image via for example a data input interface for image data.
  • the method then recognizes items on the provided image and uses the recognized items to identify patterns of an insect population. The identification may take place based on the recognized items as well as stored samples of an insect population in a database.
  • the stored samples in a database may be for example patterns which can be compared to the recognized items so as to conclude from the database an insect population. Based thereon, the method includes determining a measure of quantity of insect population based on the identified patterns. It should be noted, that the determination may also consider additional information, like for example statistical data which provide a rate or percentage, how many insects statistically are on the top of a leaf or a bottom of a leaf, so that from this ratio it can be concluded how many insects in total are on the leaf, even if the image only depicts e.g. the top of the leaf.
  • a treatment arrangement of a treatment device can be controlled based on the determined quantity of insect population.
  • controlling a treatment arrangement does not mandatorily mean an immediate activation of for example a nozzle of a treatment device, but may also constitute providing a signal or an instruction to a remote treatment arrangement or treatment device, so that the treatment arrangement or treatment device on site can carry out the activation or deactivation of a e.g. nozzle of a sprayer based thereon, even if the recognizing, identification, determination and controlling is carried at a remote site.
  • the method as outlined above can be carried out on a remote controlling device, which does not have to be provided on the treatment device on the field.
  • the method can be carried out on a service provider side, which receives for example an image taken by a camera on the field and being submitted to the remote service provider, wherein the service provider carries out the method and provides a signal or instruction, which is sent to the remote treatment device on the field to carry out an activation or deactivation of a treatment arrangement like for example a nozzle arrangement.
  • the controlling of the treatment arrangement includes activating a treatment unit, for example a nozzle, if a quantity of a first type, e.g. a pollen beetle, and/or a second type, e.g. a potato beetle, of pest insects extends a predetermined threshold.
  • the method decides to activate a treatment arrangement of a treatment device, for example a nozzle.
  • a treatment device for example a nozzle.
  • this activation should only take place, if no beneficial insects are effected. Consequently, the activation as described above only takes place if no or only a low number of beneficial insects are detected, otherwise the method blocks the activation of a treatment arrangement.
  • the activation rate (percentage) can be related to a quantity of insects.
  • controlling of a treatment arrangement is conducted based on a look-up table having stored a relation of quantity of pest insect population and a quantity of beneficial insect population on the one hand and a respective quantitative activation grade for a treatment unit on the other hand the treatment unit being allocated to a respective relation.
  • the activation and deactivation is not only a digital switch on/switch off activation, but may also include a particular activation grade depending on a quantity of pest insects population and a quantity of beneficial insect population.
  • the quantitative activation grade of a treatment unit like for example a nozzle, is higher when a quantity of pest insects detected is high, and is decreased upon an increasing quantity of beneficial insect population.
  • the detected amount of pest insects may directly be related to the quantitative activation grade for a treatment unit.
  • the quantitative activation grade may be significantly reduced upon only a very low number of detected beneficial insects, as the yield of beneficial insects may be much higher than the deterioration by pest insects.
  • the look-up table may include a plurality of combinations of a quantity of pest insects and a quantity of beneficial insects and a corresponding quantitative activation grade of a treatment device or unit. It should be noted, that such a look-up table may be a numerical look-up table having discrete values and relations stored thereon, but may also be a particular algorithm, which may include the parameters of a pest insect population (of different types), a beneficial insect population (of different types) and an activation grade.
  • recognizing items on a taken image is conducted based on a spectral detection of at least one pixel, in particular a Red-Green-Near InfraRed RGNIR detection, wherein a pattern is based on a location of the at least one detected pixel in an RGNIR color space.
  • the particular properties of pest insects and beneficial insects with respect to a particular color spectrum may be used for identification of the particular type of insect. If for example in a particular color space like the RGNIR color space, a particular type of insect, be it pest insect or beneficial insect, has a unique spectral pattern, this makes it possible to clearly identify such an insect over for example the remaining plantation or soil which is also provided on the taken image.
  • the spectral analysis may be carried out based on camera based imaging, wherein the images may carry color and/or spectral information of the imaged items.
  • identifying one or a plurality of patterns of an insect population is based on an algorithm fed by or based on stored samples of an insect population (of a particular type) of the database.
  • the identification of the insect population may underlie a particular algorithm which may be based on the samples of the insect populations which are stored in the database. This may lead to a system which has a self-learning property, so that the identification may improve over the time after having applied the method for several seasons.
  • an identified pattern of an insect population is stored as a sample of an insect population in the database.
  • the pattern analysis may be carried out based on camera based imaging, wherein the images may carry shape information of the imaged items, in particular of the pest insects or pest insect population.
  • the pattern analysis may also be carried out based on a laser scan process, wherein the laser scanned patterns may carry shape information of the imaged items, in particular of the pest insects or pest insect population.
  • recognizing items is conducted on a plurality of predetermined ranges simultaneously, wherein identifying one or a plurality of patterns of an insect population is conducted on the plurality of predetermined ranges, respectively, wherein determining a measure of a quantity of insect population is conducted on the plurality of predetermined ranges, respectively, wherein controlling a treatment arrangement of a treatment device is conducted for one or a plurality of treatment units being allocated to a respective one of the plurality of predetermined ranges.
  • the resolution of a field is not limited to the track width of a treatment device, e.g. the length of a boom of a sprayer device, but can be improved by dividing the track width of a treatment device into sub-tracks.
  • a track of a treatment device is to be understood as a width of a line for which a treatment device can treat the plantation at the same time.
  • the size of the ranges can be reduced down to the size of a treatment spot which is treated by a single nozzle of a treatment device.
  • the method can be conducted on each of the predetermined ranges or treatment spots separately. It should be noted, that even if the method for more than one predetermined range bases on a single image, also a larger image can be divided into the respective ranges which serve as a base for recognizing and identifying.
  • recognizing items on the image includes at least one of a recognition of a shape, wing beat frequency and a spectral signature of a particular type of insect, in particular a spectral signature in an RGNIR color space.
  • the pattern of the insect population can be considered for recognition and identification, but also the shape of a single insect, a shape of a particular pattern of the insect, in particular the shape of a pattern on the back side of the insect, or a particular pattern of an insect population.
  • the wing beat frequency of an insect can be detected, determined and used for identification of a particular type of insect.
  • the wing beat frequency can be detected by e.g. laser scanning of the plantation and evaluation the frequency spectrum detected. Particular characteristics in the spectrum can be indicative of particular types of insects.
  • a particular optical spectral signature of an insect may be used for identification, which may be unique for a particular type of insect.
  • the optical spectral signature may allow to distinguish types of insects, which may have a similar shape or a similar population pattern, although the insects are different and may require different measures, e.g. different pesticides or insecticides.
  • This also applies for the wing beat spectrum analysis.
  • an optical spectral signature in the Red-Green-Near InfraRed RGNIR color space may represent a better distinction than a spectral signature in a Red-Green-Blue RGB color space, as insects and plantations often do not have a blue color contribution.
  • the motion of the pest insects as well as the beneficial insects may be carried out based on a laser scan process, wherein the laser scanned motion may carry a characteristic motion of the scanned items, in particular of the pest insects, the pest insect population, the beneficial insects and/or the beneficial insect population.
  • the stored samples represent a single pest insect or an agglomeration of a plurality of pest insects, in particular a single or an agglomeration of a plurality of potato beetles, a single or an agglomeration of a plurality of pollen beetles or a combination thereof, and/or the stored samples represent a single beneficial insect or an agglomeration of a plurality of beneficial insects, in particular a single or an agglomeration of a plurality of bees.
  • the stored samples in the database may represent a large number of different insects, be it pest insects or beneficial insects. It should be noted, that the insects are not limited to the potato beetles, pollen beetles or bees, but may also be additional type of insects or even variations of the above-mentioned types of insects which may require a different treatment.
  • the stored samples may include a shape information, including a shape of a body of that particular insect or a shape of a population of that particular insect, a characteristic wing beat frequency or frequency spectrum of that particular insect, and/or a characteristic optical spectrum of that particular insect or insect population.
  • the stored samples represent a combination of a single or an agglomeration of a plurality of pest insects, in particular a single or an agglomeration of a plurality of potato beetles and/or a single or an agglomeration of a plurality of pollen beetles on the one hand and a single or an agglomeration of a plurality of beneficial insects, in particular a single or an agglomeration of a plurality of bees on the other hand.
  • a population pattern or an optical or wing beat frequency spectral signature of a single type of insect may be stored as a pattern or sample in the database, but also a combination thereof, so that also an agglomeration of different types of insects may be recognized and identified by the method.
  • determining a measure of a quantity of insect population includes determining at least one of a quantity of a particular first type of pest insects, in particular potato beetles, a quantity of a particular second type of pest insects, in particular pollen beetles, and/or a quantity of a particular type of beneficial insects, in particular bees.
  • insects it pest insects or beneficial insects, a respective quantity of insect population.
  • different types of insects may have a different ratios of visible and invisible population, which means the relation between the insects on the top of the leaf and the bottom of the leaf, and that these different ratios may be considered for determining the true quantity of insect population from the provided image.
  • determining includes online determination of a measure of a quantity of insect population based on the identified patterns, wherein in particular during field passage images and/or scans are captured, information is collected and a localized application according to the imaged and/or scanned item is carried out; wherein image capture and/or scan capture is timed such that it correlates with the speed during passage.
  • the treatment of the plantation can be carried out in real time based on the in situ captured information of that respective plantation.
  • a controlling device for a treatment device in particular for a smart sprayer for pest treatment of a plantation field comprising an image interface, a treatment control interface, a database interface, an image recognition unit being adapted for recognizing items on an image received from the image interface, an identification unit being adapted for identifying one or a plurality of patterns of an insect population based on the output of the image recognition unit and samples of an insect population being received from a database having stored samples of a pest population and/or beneficial insect population via the database interface, a determination unit being adapted for determining a measure of a plurality of insect population based on the identified patterns, and a controlling unit being adapted for generating a controlling signal for a treatment arrangement of a treatment device based on the determined measure of a quantity of insect population and for outputting the controlling signal to the treatment control interface, wherein the controlling unit being adapted for generating an activating signal for a treatment unit if a quantity of a first type and/or a second type
  • the same explanations apply as for the method as outlined above.
  • the activating signal and a blocking signal do not prejudice a particular signaling pattern, but may be understood as a functional signaling.
  • the controlling unit may output either an activating signal or a blocking signal depending on whether the pest insect threshold is exceeded or a beneficial insect threshold is exceeded or not.
  • the activating signal and the blocking signal have to be understood as a logic signaling and the respective quantity of pest insects and beneficial insects as a base for an activation or deactivation of a treatment unit like for example a respective nozzle.
  • the controlling device may refer to a data processing element such as a microprocessor, microcontroller, field programmable gate array (FPGA), central processing unit (CPU), digital signal processor (DSP) capable of receiving field data, e.g. via a universal service bus (USB), a physical cable, Bluetooth, or another form of data connection.
  • the controlling device may be provided for each treatment device.
  • the controlling device may be a central controlling device, e.g. a personal computer (PC), for controlling multiple treatment devices in the field, and may also be implemented on a remotely located server.
  • PC personal computer
  • the image recognition unit is adapted for conducting an item recognition on a plurality of predetermined ranges simultaneously, wherein the identification unit is adapted for identifying one or a plurality of patterns of an insect population on the plurality of predetermined ranges, respectively, wherein the determination unit is adapted for determining a measure of a quantity of insect population on the plurality of predetermined ranges, respectively, wherein the controlling unit is adapted for controlling a treatment arrangement of a treatment device for one or a plurality of treatment units being allocated to a respective one of the plurality of predetermined ranges.
  • the controlling device is capable of controlling the application of an insecticide or pesticide based on a plurality of predetermined ranges. This allows to increase the resolution of areas of different treatment in a field being lower than the track width of a treatment device like a sprayer and its respective boom length.
  • a treatment device for pest treatment of a plantation field comprising an image capture device being adapted for taking an image from a predetermined range of a plantation of a plantation field, a treatment arrangement having one or a plurality of treatment units, an image interface being adapted for providing an image captured by the image capture device to a controlling device as described above, a treatment control interface being adapted for receiving a treatment control signal from a controlling device as described above, wherein the image interface of the treatment device is connectable to an image interface of a controlling device as described above, wherein the treatment control device of the treatment device is connectable to a treatment control interface of a controlling device as described above, wherein the treatment device is adapted to activate respective ones of treatment units of the treatment arrangement based on a signal received from a controlling device as described above via the treatment control interface of the treatment device.
  • a treatment device which allows to treat a field.
  • the method can be conducted remotely from a treatment device, and a controlling device as described above can be located remote from a treatment device, a controlling device can also be provided on site of a treatment device, so that the complete intelligence may be allocated to the treatment device on the field.
  • image capturing can be for example provided by a remote device, like for example an unmanned aerial vehicle or a satellite having a sufficient resolution, as well as the database may be provided remotely and may be centralized for a plurality of controlling devices, which may be reasonable for large amounts of data and a data updating procedure, which is much more easier if not carried out on a treatment device on the field.
  • an image capture device although being remote from the treatment device, may be allocated to the treatment device, which may be for example an unmanned aerial vehicle which is wirelessly connected to the treatment device. It should be noted, that even if one image is captured, this image can be divided into sub-images for treatment of predetermined sub-ranges as described above.
  • the image capture device comprises one or a plurality of cameras, in particular on a boom of a treatment device, wherein the image recognition unit is adapted for recognizing insects and/or a plantation using red-green-blue RGB data and/or near infrared NRI data and/or RGNIR data.
  • image recognition can be carried out on suitable color rooms which may be selected according to the objects to be recognized, in particular plantation or insects. It should be noted, that depending on the type of plantation or the type of expected insects, also different color rooms can be selected or can be adapted or even can be combined, if necessary.
  • the treatment device further comprises a controlling device as described above, wherein the treatment device is designed as a smart sprayer, wherein a particular treatment arrangement e.g. is a nozzle arrangement, wherein the treatment unit or units are e.g. a nozzle or nozzles.
  • a particular treatment arrangement e.g. is a nozzle arrangement
  • the treatment unit or units are e.g. a nozzle or nozzles.
  • Fig. 1 illustrates an exemplary embodiment of a method.
  • the method comprises the steps of taking an image S10, recognizing items on that image S20, identifying patterns based on the items S30, determining a measure of a quantity of insect population based thereon S40, determining a quantity of a particular type of pest insects or a type of beneficial insects S41, controlling a treatment arrangement S50 as well as activating or deactivating a treatment unit S60.
  • Taking an image which can be used by the method can be conducted by several imaging capturing devices, for example a satellite, an unmanned aerial vehicle, a manned aerial vehicle or a camera being mounted on a field driving device.
  • the image or even a plurality of images which may be merged to a single image or portions of an image may serve as basis for the method for pest treatment.
  • the image will be provided in form of an image, image data or the like and may serve for the method as basis for recognizing items on the image or in the image data.
  • the image can be evaluated with respect to a predetermined field range, wherein this portion of the image of the predetermined field range refers to a particular portion of a field and depicts a plantation, insects on the plantation, the soil, insects on the soil or other structures.
  • the recognized items being recognized in S20 will serve as a basis for identifying S30 one or a plurality of patterns of an insect population.
  • the insect population may be a pest insect population or a beneficial insect population.
  • determining a measure of a quantity of insect population may also consider particular statistic data, for example a ratio between an amount of insects on the top of a leaf and a statistical amount of a number of insects on the bottom of a leaf, so that from an image of a top of a leaf it can be concluded the total quantity of insect population.
  • statistic data for example a ratio between an amount of insects on the top of a leaf and a statistical amount of a number of insects on the bottom of a leaf, so that from an image of a top of a leaf it can be concluded the total quantity of insect population.
  • background data are not limited to such ratios and may include other empirical data which may serve for determining a measure of a quantity of insect population.
  • the treatment arrangement is controlled S50.
  • Controlling means that a particular signal is provided which may activate or deactivate a treatment unit. It should be understood, that in particular the actual activation or deactivation of a treatment unit as such is not mandatorily part of the method, as the method may be conducted remote from a field, for example by a service provider. If the method is conducted at the service provider's side, a signal may be provided as a signal for controlling a treatment arrangement, wherein this signal may be interpreted by a treatment arrangement so as to control individual treatment units, like for example nozzles of a sprayer device.
  • Fig. 2 illustrates the decision making based on which an activation of a treatment unit takes place or not.
  • a measure of a quantity of insect population in particular a quantity of pest insect population and a quantity of beneficial insect population
  • the method in this loop comes to an end.
  • activation of the treatment unit S60 takes place.
  • the threshold tp and the threshold tb may be thresholds which have a certain dependency on each other. This means, that a threshold of tp is higher in case the threshold of tb is higher.
  • a look-up table may be provided which takes into account the different levels of tp and tb and based on that a particular grade of activation.
  • the activation is not a digital activation of on/off, but may be a particular grade of activation, for example 10%, 30% or 70% depending on the determined quantity of pest insects and the quantity of beneficial insects.
  • the method is repeated for a subsequent range of plantation. The method can also applied a couple of times for one range of plantation, so as to eliminated statistical deviations.
  • Fig. 3 illustrates a further embodiment of a determination which considers two different types of pest insects. It should be considered, that the decision illustrated in Fig. 3 is not limited to only two types of insects but may be adapted accordingly to distinguish between more than two types of pest insects and even more than one type of beneficial insects and will be adapted accordingly by a skilled person.
  • After determining the measure of a quantity of insect population which includes a quantity of insect population of a first type of pest insects, a second type of pest insects and a quantity of beneficial insects for the purposes of Fig. 3 , it is determined whether a quantity of a first type of pest insects is above a certain threshold tp1 or a quantity of a second type of pest insects is above a certain threshold tp2.
  • Fig. 2 and Fig. 3 are illustrations of a decision based on a Yes/No decision, however, the several thresholds tp1, tp2 and tb may also be represented by an algorithm which expresses the grade of activation in step S50 or S60.
  • FIG. 3 the activation of a treatment unit S60 is illustrated, it should be understood, that Figs. 2 and 3 can also lead to an output of an instruction or a signal for controlling a treatment arrangement at a remote side, so that the instruction or activation/deactivation signal or an activation grade signal may be interpreted by a treatment arrangement to activate/deactivate a treatment unit like a nozzle.
  • Fig. 4 illustrates the procedure of recognizing items 20 on an image 10.
  • An image 10 which may be provided for applying the method and which may be used by a controlling device (which will be described in the following) includes particular items 20. Based on the recognized items 20, one or a plurality of patterns 31, 32, ..., 39 are identified whereas samples stored in a database 90 (not illustrated here) may serve as a basis for this identification.
  • the recognized items 20 on the image 10 may include a shape 21 of a particular type of insect or may be a spectral signature 22 of a particular type of insect. Both, the shape 21 as well as the spectral signature may be used for identification of a particular type of insect, be it a pest insect or a beneficial insect.
  • Spectral signature may include an optical spectrum as well as a frequency spectrum of a wing beat frequency.
  • a database may provide samples 931, 932, ..., 939, which provide a correlation between a particular pattern of a first type of pest insect 31, a particular shape 21, a spectral signature 22 and further additional information which are consolidated in for example sample 931.
  • the stored samples may include a shape information, including a shape of a body of that particular insect or a shape of a population of that particular insect, a characteristic wing beat frequency or frequency spectrum of that particular insect, and/or a characteristic optical spectrum of that particular insect or insect population. The same applies for the other samples stored in the database.
  • the patterns 31, 32, ..., 39 are identified and will be used for identifying a particular type of insect as well as a quantity of insect population based on the samples 931, 932, ..., 939 of the database.
  • a quantity of a particular type of insect may be determined, so that based on the determined quantity of insect population controlling of a treatment arrangement may take place.
  • a sample may represent a single insect of a particular type, but may also represent a population, i.e. a plurality of insects of a particular type or an agglomeration of different types of insects.
  • FIG. 5 illustrates a general setup of the field situation.
  • a plantation field or a crop canopy 300 may be considered as an agglomeration of a plurality of predetermined ranges of a plantation field 310, 320, 330. Taking an image or at least recognizing and identifying may take place based on a predetermined range of a plantation field 310, 320, 330. It may be understood, that the image taken by an image capture device 220, 230, 240 may also be a large image of the entire plantation field 300 or larger portions of a plantation field 300 and may be divided in sub-images which then may serve for the recognizing and identifying process. For illustration purposes, in Fig.
  • a particular image capture device 220, 230, 240 is allocated to a particular predetermined range of a plantation field 310, 320, 330.
  • each of the image capturing devices takes an image which is used for the recognizing and identification process.
  • treatment units 61, 62, 63 as part of a treatment arrangement 60 are activated or deactivated based on the determination of the quantity of the particular type of insects. It should be noted, that as the smallest treatment spot, a spot may be considered which may be covered by a single treatment unit 61, 62, 63, which may be a nozzle of a sprayer.
  • a plurality of treatment units 61 may be allocated to a particular predetermined range of plantation field 310, as illustrated in Fig. 5 .
  • the image capturing and/or scanning may take place down to a plant or even leaf level, which means that the situation on a particular plant or even leaf can be detected.
  • the same applies to the treatment so that the number and density of nozzles can be in a way to treat a single plant or even a leaf.
  • the nozzles may be moved and inclined, and even may be arranged on a robot arm to focus on a particular plant or even leaf.
  • the image capturing devices may be cameras or laser scanners.
  • the image capturing devices may be adapted to not only take spatial resolved images, but also time resolved images, e.g. video sequences. Image capturing devices may also be adapted to optically detect time resolved parameters of particular insects, as e.g. a motion pattern and frequencies.
  • the wing beat frequency of an insect can be detected and used for identification of a particular type of insect.
  • the wing beat frequency can be detected by e.g. laser scanning of the plantation and evaluation the frequency spectrum detected. Particular characteristics in the spectrum can be indicative of particular types of insects.
  • Fig. 6 illustrates a controlling device 100 having an image interface 110 and a treatment control interface 160.
  • the image interface 110 serves for receiving an image which is captured by an image capturing device as described with respect to Fig. 5 .
  • This image or image data is provided to the image recognition unit 120 for recognizing items 20 on the respective image.
  • An identification unit 130 serves for identifying one or a plurality of patterns based on the recognized items of the image recognition unit 120 and patterns 931 to 939 of a database 90, which will be provided to the controlling device 100 via the database interface 190.
  • the identified patterns will be used for determining a measure of a quantity of insect population in the determination unit 140 which then will serve for controlling a treatment arrangement in the controlling unit 150.
  • the controlling unit 150 may output an instruction or an activation/deactivation signal or even a grade of activation which is then delivered to a treatment control interface 160.
  • the controlling device 100 may be provided remote from a treatment device and may also be provided remote from a database 90. Even if the treatment device is provided with image capturing devices 120, 130, 140, these images or image data may be transferred to the remote control device 100 and may be received by the image interface 110 of the controlling device 100.
  • the instruction signal or activation/deactivation signal then will be issued via the treatment control interface 160 and will be transferred to the treatment device which may be a sprayer with a plurality of nozzles 61, 62, 63.
  • controlling device may be located elsewhere and that the data transfer of the image data as well as the data transfer of the activation/deactivation signal may be carried out over wide distances. The same applies to the transfer of the sample data of the database 90, which also may be located elsewhere.
  • Fig. 7 illustrates a treatment device 200 having implemented a controlling device 100. It should be understood, that although Fig. 7 illustrates the controlling device 100 as part of the treatment device, the controlling device may be remotely positioned with respect to the treatment device 200. In this case, the communication between the interface 110, 160 of the controlling device 100 to the respective counter-interfaces 210, 260 of the treatment device 200 may take place via wireless communication over wide distances.
  • Database 90 may be provided within the treatment device 200 but also may be located remotely from the treatment device 200 which is illustrated by the two alternatives of the database in Fig. 7 . It should be noted, that a smaller database may be provided within the treatment device 200, whereas a more complex database 90 may be provided remote thereof.
  • the details of the controlling device 100 are described with respect to Fig. 6 and apply accordingly.
  • the image capturing devices 220, 230, 240 are illustrated and described with respect to Fig. 5 which applies accordingly.
  • the captured image will be provided via the image interface 210 of the treatment device 200 to the image interface 110 of the controlling device.
  • an instruction or activation/deactivation signal or activation grade is provided to the treatment control interface 160, which will transfer the respective instruction/data via the treatment control interface 260 of the treatment device 200 to a treatment arrangement 60.
  • the treatment arrangement 60 may be divided into respective treatment units 61, 62, 63 the application of which is illustrated with respect to Fig. 5 .
  • the decision will be based on a look-up table, there will be provided a correlated relation between the quantity of pest insects and beneficial insects so as to arrive at an activation grade.

Landscapes

  • Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Environmental Sciences (AREA)
  • Zoology (AREA)
  • Wood Science & Technology (AREA)
  • Pest Control & Pesticides (AREA)
  • Insects & Arthropods (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Quality & Reliability (AREA)
  • Radiology & Medical Imaging (AREA)
  • Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Catching Or Destruction (AREA)
  • Image Analysis (AREA)
EP19165176.9A 2019-03-26 2019-03-26 Camera based pest management sprayer Active EP3714688B1 (en)

Priority Applications (10)

Application Number Priority Date Filing Date Title
EP19165176.9A EP3714688B1 (en) 2019-03-26 2019-03-26 Camera based pest management sprayer
HUE19165176A HUE060153T2 (hu) 2019-03-26 2019-03-26 Kameraalapú kártevõírtó permetezõ
ES19165176T ES2929076T3 (es) 2019-03-26 2019-03-26 Rociador de control de plagas basado en cámara
US17/598,273 US12075770B2 (en) 2019-03-26 2020-03-25 Camera based pest management sprayer
JP2021557195A JP2022526157A (ja) 2019-03-26 2020-03-25 カメラベースの害虫管理噴霧器
CA3135084A CA3135084A1 (en) 2019-03-26 2020-03-25 Camera based pest management sprayer
PCT/EP2020/058438 WO2020193666A1 (en) 2019-03-26 2020-03-25 Camera based pest management sprayer
CN202080023887.XA CN113631035B (zh) 2019-03-26 2020-03-25 基于摄像头的害虫管理喷雾器
EP20714206.8A EP3945806A1 (en) 2019-03-26 2020-03-25 Camera based pest management sprayer
BR112021017157A BR112021017157A2 (pt) 2019-03-26 2020-03-25 Método para tratamento de pragas em um campo de plantação e dispositivos de controle para um dispositivo de tratamento e de tratamento para o tratamento de pragas de um campo de plantação

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP19165176.9A EP3714688B1 (en) 2019-03-26 2019-03-26 Camera based pest management sprayer

Publications (2)

Publication Number Publication Date
EP3714688A1 EP3714688A1 (en) 2020-09-30
EP3714688B1 true EP3714688B1 (en) 2022-08-03

Family

ID=65991601

Family Applications (2)

Application Number Title Priority Date Filing Date
EP19165176.9A Active EP3714688B1 (en) 2019-03-26 2019-03-26 Camera based pest management sprayer
EP20714206.8A Pending EP3945806A1 (en) 2019-03-26 2020-03-25 Camera based pest management sprayer

Family Applications After (1)

Application Number Title Priority Date Filing Date
EP20714206.8A Pending EP3945806A1 (en) 2019-03-26 2020-03-25 Camera based pest management sprayer

Country Status (9)

Country Link
US (1) US12075770B2 (zh)
EP (2) EP3714688B1 (zh)
JP (1) JP2022526157A (zh)
CN (1) CN113631035B (zh)
BR (1) BR112021017157A2 (zh)
CA (1) CA3135084A1 (zh)
ES (1) ES2929076T3 (zh)
HU (1) HUE060153T2 (zh)
WO (1) WO2020193666A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230363370A1 (en) * 2020-10-14 2023-11-16 Basf Agro Trademarks Gmbh Treatment system for weed specific treatment
CN113536011B (zh) * 2021-08-09 2022-08-19 河北科技师范学院 一种用于调查田间昆虫的植保方法及装置
CN113925039B (zh) * 2021-10-12 2022-10-11 一鼎(福建)生态园林建设有限公司 一种基于物联网的园林病虫害智能防护系统及防控方法

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20100127476A (ko) 2009-05-26 2010-12-06 대한민국(농촌진흥청장) 병해충 관리 시스템
WO2012090514A1 (en) * 2010-12-27 2012-07-05 Sumitomo Chemical Company, Limited Arthropod pest control composition and method for controlling arthropod pests
JP5812321B2 (ja) * 2011-02-04 2015-11-11 国立大学法人徳島大学 虫の画像処理装置、画像処理方法、画像処理プログラム及びコンピュータで読み取り可能な記憶媒体
US9563945B2 (en) * 2012-07-05 2017-02-07 Bernard Fryshman Object image recognition and instant active response with enhanced application and utility
EP3262934A1 (de) * 2016-06-28 2018-01-03 Bayer CropScience AG Verfahren zur unkrautbekämpfung
CN106956778A (zh) * 2017-05-23 2017-07-18 广东容祺智能科技有限公司 一种无人机农药喷洒方法及系统
CN108132655A (zh) * 2017-12-22 2018-06-08 长沙修恒信息科技有限公司 一种用于农田的多功能移动检测装置
JP2021534744A (ja) * 2018-08-31 2021-12-16 ファウナフォトニックス アグリカルチャー アンド エンヴァイロメンタル アー/エスFaunaphotonics Agriculture & Environmental A/S 殺虫剤噴霧装置
EP3847439A2 (en) * 2018-09-09 2021-07-14 Viewnetic Ltd. System and method for monitoring plants in plant growing areas
CN109077047B (zh) * 2018-09-20 2021-04-16 东南大学 一种基于蔬菜大棚的多功能自动雾化杀虫器及操作流程
BR112021017179A2 (pt) * 2019-03-29 2021-11-09 Basf Agro Trademarks Gmbh Método para o tratamento de uma plantação de campo, sistema de gerenciamento de campo, dispositivo de tratamento e sistema de tratamento

Also Published As

Publication number Publication date
US12075770B2 (en) 2024-09-03
JP2022526157A (ja) 2022-05-23
CN113631035B (zh) 2023-06-20
CA3135084A1 (en) 2020-10-01
WO2020193666A1 (en) 2020-10-01
ES2929076T3 (es) 2022-11-24
US20220183266A1 (en) 2022-06-16
HUE060153T2 (hu) 2023-01-28
BR112021017157A2 (pt) 2021-11-09
EP3714688A1 (en) 2020-09-30
EP3945806A1 (en) 2022-02-09
CN113631035A (zh) 2021-11-09

Similar Documents

Publication Publication Date Title
US12075770B2 (en) Camera based pest management sprayer
US10405535B2 (en) Methods, systems and devices relating to real-time object identification
US11974567B2 (en) Method for applying a spray onto a field
CN108369635B (zh) 用于航空图像获取与分析的方法
JP7562552B2 (ja) 化学的および機械的手段を用いるターゲットを定めた雑草防除
EP3741214B1 (en) Method for plantation treatment based on image recognition
US10269107B2 (en) Selective plant detection and treatment using green luminance photometric machine vision scan with real time chromaticity operations and image parameter floors for low processing load
US20190095710A1 (en) Methods, systems, and devices relating to shadow detection for real-time object identification
US20210209352A1 (en) Insect and other small object image recognition and instant active response with enhanced application and utility
US20230371493A1 (en) Treatment system for plant specific treatment
CN109496622A (zh) 害虫的识别方法及装置、施药信息的确定方法、植保系统
JP2022542764A (ja) 農業機器を用いて農場を処置するための適用マップを生成する方法
US20230363370A1 (en) Treatment system for weed specific treatment
AU2018250354B2 (en) Method and System for Extracting Shadows from an Image during Optical Based Selective Treatment of an Agricultural Field
KR20210101382A (ko) 야생동물 퇴치 시스템 및 그 방법
KR102531690B1 (ko) Oled 및 빅데이터와 파장을 기반으로 하는 겹눈해충 퇴치시스템
Siddiqi et al. Edge link detector based weed classifier
RU2021130475A (ru) Опрыскиватель для борьбы с вредителями на основе камеры

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

RIN1 Information on inventor provided before grant (corrected)

Inventor name: THE OTHER INVENTORS HAVE WAIVED THEIR RIGHT TO BE THUS MENTIONED.

Inventor name: KIEPE, BJOERN

Inventor name: PETERS, OLE

Inventor name: WAHABZADA, MIRWAES

RIN1 Information on inventor provided before grant (corrected)

Inventor name: KIEPE, BJOERN

Inventor name: WAHABZADA, MIRWAES

Inventor name: JANSSEN, OLE

Inventor name: THE OTHER INVENTORS HAVE WAIVED THEIR RIGHT TO BE THUS MENTIONED.

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

RAP3 Party data changed (applicant data changed or rights of an application transferred)

Owner name: BASF AGRO TRADEMARKS GMBH

17P Request for examination filed

Effective date: 20210330

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20210602

GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

INTC Intention to grant announced (deleted)
GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR1

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR1

INTG Intention to grant announced

Effective date: 20211221

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

INTG Intention to grant announced

Effective date: 20220111

GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR1

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR1

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20220125

INTG Intention to grant announced

Effective date: 20220204

INTC Intention to grant announced (deleted)
INTG Intention to grant announced

Effective date: 20220223

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1508016

Country of ref document: AT

Kind code of ref document: T

Effective date: 20220815

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602019017655

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: ES

Ref legal event code: FG2A

Ref document number: 2929076

Country of ref document: ES

Kind code of ref document: T3

Effective date: 20221124

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20220803

REG Reference to a national code

Ref country code: HU

Ref legal event code: AG4A

Ref document number: E060153

Country of ref document: HU

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220803

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220803

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221205

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221103

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220803

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220803

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220803

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220803

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1508016

Country of ref document: AT

Kind code of ref document: T

Effective date: 20220803

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220803

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221203

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220803

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220803

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220803

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220803

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220803

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220803

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602019017655

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220803

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220803

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220803

26N No opposition filed

Effective date: 20230504

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220803

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220803

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20230326

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20230331

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230326

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230326

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230331

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230326

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230326

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230331

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230331

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: HU

Payment date: 20240318

Year of fee payment: 6

Ref country code: DE

Payment date: 20240328

Year of fee payment: 6

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: IT

Payment date: 20240321

Year of fee payment: 6

Ref country code: FR

Payment date: 20240326

Year of fee payment: 6

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: ES

Payment date: 20240412

Year of fee payment: 6