EP1505556B1 - Method and apparatus for flight obstacle recognition - Google Patents

Method and apparatus for flight obstacle recognition Download PDF

Info

Publication number
EP1505556B1
EP1505556B1 EP20040018510 EP04018510A EP1505556B1 EP 1505556 B1 EP1505556 B1 EP 1505556B1 EP 20040018510 EP20040018510 EP 20040018510 EP 04018510 A EP04018510 A EP 04018510A EP 1505556 B1 EP1505556 B1 EP 1505556B1
Authority
EP
European Patent Office
Prior art keywords
flight
characterized
obstruction
identification
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP20040018510
Other languages
German (de)
French (fr)
Other versions
EP1505556A1 (en
Inventor
Martin Dr. Arndt
Claus Dr. Dähne
Michael Dr. Gross
Rene Dr. Koch
Holger Dr. Voos
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Diehl BGT Defence GmbH and Co KG
Original Assignee
Diehl BGT Defence GmbH and Co KG
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to DE10336928 priority Critical
Priority to DE10336928 priority
Application filed by Diehl BGT Defence GmbH and Co KG filed Critical Diehl BGT Defence GmbH and Co KG
Publication of EP1505556A1 publication Critical patent/EP1505556A1/en
Application granted granted Critical
Publication of EP1505556B1 publication Critical patent/EP1505556B1/en
Application status is Active legal-status Critical
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G08SIGNALLING
    • G08GTRAFFIC CONTROL SYSTEMS
    • G08G5/00Traffic control systems for aircraft, e.g. air-traffic control [ATC]
    • G08G5/0004Transmission of traffic-related information to or from an aircraft
    • G08G5/0013Transmission of traffic-related information to or from an aircraft with a ground station
    • GPHYSICS
    • G08SIGNALLING
    • G08GTRAFFIC CONTROL SYSTEMS
    • G08G5/00Traffic control systems for aircraft, e.g. air-traffic control [ATC]
    • G08G5/0047Navigation or guidance aids for a single aircraft
    • G08G5/0069Navigation or guidance aids for a single aircraft specially adapted for an unmanned aircraft
    • GPHYSICS
    • G08SIGNALLING
    • G08GTRAFFIC CONTROL SYSTEMS
    • G08G5/00Traffic control systems for aircraft, e.g. air-traffic control [ATC]
    • G08G5/0073Surveillance aids
    • G08G5/0078Surveillance aids for monitoring traffic from the aircraft
    • GPHYSICS
    • G08SIGNALLING
    • G08GTRAFFIC CONTROL SYSTEMS
    • G08G5/00Traffic control systems for aircraft, e.g. air-traffic control [ATC]
    • G08G5/04Anti-collision systems
    • G08G5/045Navigation or guidance aids, e.g. determination of anti-collision maneuvers

Description

The invention is based on a method for detecting a flying obstacle in an environment of an aircraft, in particular an unmanned aerial vehicle, at least two images each of at least a portion of the environment be recorded from the images the obstacle detected and with a Identification is provided and one of the identifier associated signal in a ground station is issued.

The invention is also based on a device for detecting a Flight obstacle in an environment of an aircraft, in particular an unmanned aerial vehicle Aircraft, with at least one camera unit for recording at least a part of the environment and an evaluation device for detecting the Flight obstacle and an assignment of an identifier to the obstacle.

From EP 1 296 213 A1 a method for displaying a flying object is known, in which an environment of an unmanned aerial vehicle with a number is captured by cameras and the images are sent to a ground station and be displayed on a display for a ground pilot. The ground pilot can control the aircraft based on the images shown, looking for flying objects near the aircraft by aids, such as flashing Symbols in the picture shown, is pointed. The constant active observation from electronically generated images in which only occasionally a flying object is recognizable, but is very tiring for a ground pilot, which he his responsibility for the safety of the aircraft with regard to a collision warning only with considerable concentration over a long period of time bill can carry.

The invention is therefore based on the object, in particular with respect to the Handleability improved method for representing a flying obstacle specify. The invention is also based on the object, a device indicate that such a procedure can be initiated in a simple manner can.

The former object is achieved by a method of the type mentioned solved, according to the invention to the detection and the assignment of the identifier Board of the aircraft are made and the identifier to the ground station is sent.

By sending the ID and possibly further detailed information to the ground station will be provided to the ground pilot Information reduced to a necessary and reasonable minimum. The ground pilot is therefore no longer the task, an airspace around the aircraft to actively observe images. This can increase the efficiency of Work of the ground pilot increased and thus increased safety. By the renunciation of the transmission of moving images from the entire environment the aircraft is also transmitted by the aircraft to the ground station Data volume kept low. The remote data transmission can thus with low capacity and cost-effective.

Airborne obstacles can be flying objects such as airplanes, helicopters, balloons or similar and ground-based objects such as towers, buildings, bridges, Masts, cables, etc. are detected. It is sufficient if the flight obstacles be recognized as such. The flight obstacles should be recognized, if they are in the vicinity of the aircraft, thus at a distance to maximum 3 km, advantageously up to 5 km and in particular up to 8 km, for example depending on visibility conditions and / or size and visibility of the hoverfly. For this, images of the surroundings of the aircraft are taken, are expected to be prioritized within the flight obstacles, for example in one Solid angle range of at least 110 ° in the horizontal direction on both sides Longitudinal axis of the aircraft and at least 30 ° in the vertical direction on both sides the longitudinal axis. The images are preferably from a passive wide-angle sensor recorded with a high resolution. The sensors are designed that there are sufficient environmental obstacles in the area and at a sufficient distance and with the lowest possible false alarm rate can be detected to allow sufficient time for an evasive maneuver to make available. The resolution of these pictures is expediently at least 2 mrad, in particular at least 0.3 mrad, which is the maximum corresponds to the foveal resolution of the human eye.

The images, from which an evaluation device detects the obstacle, can created by a continuous recording of image information be. It is also possible to take pictures in any or given temporal Record timing. At the same time, the frame rate, the field of view and the resolution be designed so that either with one or a few rigidly mounted cameras high pixel count or with one or more cameras with smaller but sufficient Pixel count and an additional scanner system the desired environment is covered.

To detect the obstacle, two or more can be consecutive taken pictures with the help of an image processing unit aboard the Processed aircraft, for example, compared to each other. An image processing in itself, which allows such recognition of an object from images, is known for example from Görz, Rollinger, Schneeberger: "Handbuch der Artificial Intelligence ", Oldenbourg Verlag, 2000, chapter 21.4 Detecting two or more images in a different spectral range recorded, for example in the visual and infrared Area, compared with each other. After detection, the Obstacle assigned an identifier. Such an identifier can be a signal for example, be a string. The ID is then sent to a Transmitter passed, which sends the identifier to the ground station. In this case, it is not necessary for the transmitting device to transmit the identifier directly sends to that ground station in which a signal associated with the identifier is issued. The identifier can indirectly via another ground station, a satellite, or via an aircraft serving as a relay station for issuing Ground station to be transmitted. There, the assigned signal is output, that the ground pilot can perceive it, for example as a visual one Object on a display means such as a screen or projection device, or as an acoustic signal or both together. The procedure is special suitable for use as a collision warning method.

In an advantageous embodiment of the invention, a classification of a Danger potential of the flying obstacle on board the aircraft made. Such a classification may be a control device on board the aircraft serve as a basis for decision whether an intervention in the currently flown Course carried out without a corresponding command of a ground pilot shall be. Conveniently, the classification of the risk potential sent to the ground station. A ground pilot can quickly identify in this way whether a rapid intervention is necessary due to a dangerous situation. The danger potential can be determined from the visual line rotation speed, the apparent size of the obstacle, its rate of change in the recorded images, the elevation angle or elevation angle, the relative Height to the aircraft or a remaining response time up to a calculated Collision result. Even an unsafe detection can be one or more Categories in the classification form, so that uncertain recognized Obstacles are only reported to the ground station if they are a significant Pose danger potential for the aircraft. This can cause false alarms be kept low. The classification can be in the classes "high", "medium" and "low", in a finer or continuous division or, still more differentiated, according to different types of dangers.

Important information about a detected obstacle can be obtained by making a measurement with respect to the flight obstacle with an active Measurement signal is performed. Such a measurement signal can be emitted by the aircraft be and is for example a laser beam or a radar signal. The Measurement signal can be used for a distance measurement or for a measurement of the differential speed between the flight obstacle and the aircraft used become. Such measurements may be used to calculate a remaining reaction time be used up to a possible collision. The won Information can be passed on to the ground pilot and / or for classification the danger potential are used. Carrying out the measurement with the active measuring signal can be controlled by the ground pilot or automatically be performed, for example, when exceeding a predetermined Hazard potential. It is also possible for any detected flight obstacle basically an active measurement, for example a distance measurement, perform these types of measurements for disregard of For example, can serve to remove flying obstacles for a ground pilots are of little or no interest. An active sensor has usually a limited field of view, which is why they expediently independently movable and swiveled to an interesting obstacle is.

As a further embodiment of the invention it is proposed that on board a Ausweichtrajektorie is determined and in particular the evasion trajectory associated Data is transmitted to the ground station. By the determination the avoidance trajectory on board the aircraft not only eliminates the calculation a corresponding evasion trajectory in the ground station, but it will the basic requirement for a flight of the aircraft with autopilot without one absolutely necessary radio contact to ground station created. This can The safety of the aircraft can be substantially increased, as in case of failure of a Radio contact between a ground station and the aircraft's capability of the aircraft is maintained to independently determine an evasion trajectory and fly along this evasion trajectory. A control of the aircraft With the help of an autopilot independently around an obstacle around then can make sense be when the reaction time to a calculated collision is no longer sufficient to give the ground pilot the decision-making responsibility for an evasive maneuver to hand over. With a sufficient reaction time of the Ausweichtrajektorie expediently to the ground station transmitted. For a ground pilot, the avoidance trajectory can be used as an alternative are displayed.

Advantageously, the avoidance trajectory is included with on board determined information about the environment and / or the flight situation of Aircraft detected. Information about the environment can be further recognized Obstacles and in particular their potential danger or be the Aircraft surrounding landscape, such as mountains and valleys. information about the flight situation, the speed of the aircraft, whose Location in space, its distance from a destination or the position of Be Flugaktuatoren to influence the trajectory. In this way can be prevented be that flying the aircraft while guiding along the avoidance trajectory in danger of being crushed, for example on a mountain or on a mountain Another obstacle is brought. Information about the environment and in particular about the flight situation of the aircraft are usually in one Ground station to a lesser extent than in the aircraft before. Calculating the Evasion trajectory aboard the aircraft thus allows for inclusion much more complete information than is usually the case in a calculation would be possible in the ground station.

Typically, a ground pilot will fly over on an evasion trajectory decide and initiate such an evasive maneuver. At a very low Reaction time, however, may be beneficial to the safety of the aircraft Decide if the aircraft should fly on an evasion trajectory a control device is taken on board the aircraft. Furthermore can the ability to act of the aircraft in a disturbed communication between the aircraft and the ground station in a dangerous situation stay. Conveniently, such automatic from ground pilots manual disengageable, so that the aircraft does not have unforeseen evasive maneuvers performs, which are explicitly not wanted by the ground pilot.

The ground pilot can as a basis for decision, whether an evasive maneuver should be initiated, meaningful additional information offered by a sent the flying obstacle performing detail image to the ground station and there is shown. Such a detail picture shows only a detail of the overall picture of the monitored room. The ground pilot will also use this detail image for information in the form of a symbol a real image, for example in the infrared or in the visual spectral range, by which it detects the object of the flight outgoing danger potential can better estimate. This detail picture, the the obstacle and preferably a small section of the environment of Flying obstacle represents, at the request of the ground pilot or automatically be sent with the identifier to the ground station. It is also possible, that the detail image automatically starts from a preset hazard potential of Aircraft is transmitted to the ground station.

The detail image can be arranged by a movable in the aircraft and on the Airborne detectable detail camera be recorded. In this case is the detail image expediently optically zoomable, causing the ground pilots the easy recognition of the flying obstacle is facilitated. In another Embodiment of the invention, the detail image is a picture detail of a recognition of the obstacle used. It may be on an additional shot the detail image is omitted and the detail image very quickly from existing Data are selected and transmitted to the ground station. It is also possible that the detail image first selected from existing artwork and is sent to the ground station, for example automatically with the identifier, and in addition to a special request of the ground pilot in addition a detail camera directed to the specified obstacle.

Another advantage can be achieved by adding images that make up the flight obstacle is detected, are recorded in the infrared spectral range. It can Obstacles due to poor visibility or darkness for the human eye could not be recognized. An infrared camera unit can be based on a line scanner or a wide-angle scanning method be conceived.

For the estimation of the danger potential important additional information for the Ground pilots can be obtained by taking the detail image in a spectral range which is different from the spectral range for detection of the obstacle used distinguishes images. In the dark or Twilight, for example, used to detect the obstacle Be included in the infrared spectral range, the Detail image taken for easier interpretation in the visible spectral range becomes. Conversely, it is also possible that the detection of the flying obstacle taken pictures in the visible spectral range and the ground pilot requests a detail image in the infrared spectral range, for example, to assess the risk potential of the flying obstacle.

The detail image may be a single image that is similar to the flying obstacle Photo shows. Also possible is an image processed by image processing e.g. only contours. Appropriately, a moving image of the flying obstacle sent to the ground station, for example in the manner of a video, whereby the Ground pilots additional information about the movement of the flying obstacle can be delivered.

The object related to the device is achieved by a device of the initially solved type mentioned, in the present invention, the evaluation device electrically with the camera and a transmitting device for sending the identifier connected to a ground station. By the electrical connection of the evaluation device with the camera and the transmitting device eliminates the need the data transmission of complete images and thus large amounts of data from the aircraft to the ground station. In addition, the aircraft may be in a dangerous situation be guided independently of a radio contact to the ground station. In addition, the information offered to the ground pilot is essential Minimum limited. The electrical connection can, for example via a wire or indirectly via an electrical circuit can be achieved. Instead of The electrical connection is also a mechanical connection between the evaluation device and the camera and transmitting device possible, for example, by these devices for common arrangement in the aircraft are provided.

A detail image can be transmitted to a ground pilot quickly and with little effort be when the device is a detail image unit for the extraction of a the flight obstacle imaging detail image and passing on the detail image to Transmitting device has.

The device expediently comprises a first camera unit for recording an overall image in a first spectral range and a second camera unit for capturing an image, in particular a detail image, in one of first different spectral range. The picture can be a full picture or a Detail picture, wherein the second camera unit for recording in the visual or infrared spectral range or for recording reflected laser or Radar beams is provided.

drawing

Further advantages emerge from the following description of the drawing. In the Drawing is an embodiment of the invention shown. The drawing, the description and claims contain numerous features in combination. The expert will expediently also consider the features individually and summarize to meaningful further combinations.

Show it:

Fig. 1
a schematic block diagram of an apparatus for detecting and guiding an aircraft,
Fig. 2
a schematic representation of an environment of an aircraft and
Fig. 3
a screen representation of the environment of FIG. 2.
Description of the embodiments

FIG. 1 shows a block diagram of a device 2 for detecting and for guiding an aircraft 4 (FIG. 2) completely on board the aircraft 4 is arranged. The aircraft 4 is an unmanned aerial vehicle 4, for example a reconnaissance plane or transport plane. In a ground station 6 are a device 8 for graphical representation and a device 10 for Communication with a ground pilot to work with the device 2 arranged. The device 10 for communication with a ground pilot comprises a display means 12 in the form of a screen, two further screens for displaying detail images 38, another screen 13 for written advertisements of additional information and a number of tax credits 14 for input of control commands.

By the dashed arrows 16 is a radio contact between the ground station. 6 and a control device 18 of the device 2 indicated. To communication via remote data transmission with the ground station 6 comprises the control device 18 a transmitting device for the transmission of identifiers and a Receiving device. The control device 18 is electrically connected to a unit 20 for emitting and receiving laser light and a unit 22 for distance calculation. Also with the control device 18 electrically connected is a camera unit 24 for recording an overall image in the spectral range of visible light. Between the camera unit 24 and the Control device 18, an evaluation device 26 is arranged, which is a Image processing unit includes. The evaluation device 26 is for detection an obstacle 28, 30, 32, 34 (Figure 2) and to assign an identifier to the obstacle 28, 30, 32, 34 is provided. In addition, the device includes 2, a second camera unit 36, which is provided for receiving a detail image 38 is.

The control device 18 is, inter alia, for the calculation of an evasion trajectory 42 (Figure 3) and is for this purpose in a data connection with a flight information unit 44. The flight information unit 44 comprises Data about the immediate environment of the aircraft 4, such as the Scenic environment and data about the flight situation of the aircraft 4, such as For example, the current airspeed and the orientation of the Aircraft 4 in the room. Also electrically with the control device 18 connected to a flight control unit 46, which is used to control a Flugaktorik 48 with Engines and hydraulic devices is provided.

A method for visualizing flight obstacles 28, 30, 32, 34 in the environment of the aircraft 4 and for guiding the aircraft 4 is based on the schematic Representation in Figures 2 and 3 explained in more detail below. With The camera unit 24 for capturing an overall image becomes an environment 50 of the aircraft 4 recorded in one or more images. This overall picture is arranged symmetrically about the direction of flight 52 of the aircraft 4 and covers a solid angle range of 220 ° in the horizontal and 60 ° in the Vertical. To capture the overall picture, the camera unit 24 comprises four Cameras, each with a sensor array with an image resolution of 1 mrad. After taking a first overall picture of the environment 50 is of the Camera unit 24 a second and possibly other overall images of the environment 50 recorded. The overall images become the evaluation device 26 passed and there with the help of image processing methods on flying obstacles 28, 30, 32, 34 in the environment 50 examined. Here are two obstacles 28, 30 are recognized directly and two more objects are considered as possible candidates detected for further flight obstacles 32, 34. Based on the following from the Camera unit 24 recorded and processed by the evaluation device 26 Images, the flight obstacles 28, 30, 32, 34 further processed and Information about them stored in the evaluation device 26. In addition, will the flight obstacles 28, 30, 32, 34 each provided with an identifier and This total information is forwarded to the control device 18. The control device 18 calculates a danger potential that can be developed from the images the flight obstacles 28, 30, 32, 34 for the aircraft 4 with the help of Line of sight rotation speed and distance of the flying obstacles 28, 30, 32, 34, a shape and size changing from image to image, as well as the relative Height of flight obstacles 28, 30, 32, 34.

By the interaction of the control device 18 and the evaluation device 26, the two flying obstacles 28, 30 are considered to be on a trajectory 54 or 56 moving aircraft detected. Here is the obstacle obstacle 28th No danger to the aircraft 4, because the flying obstacle 28 much lower flies as the aircraft 4. The danger potential of the flying obstacle 28 is therefore classified as low. However, the obstacle 30 is considered to be on a collision course calculated with the aircraft 4 flying. This can also be an expected Collision point 60 are determined. The danger potential of the flying matter 30 is therefore classified as very high. Due to this high classification a calculation of the avoidance trajectory 42 is automatically initiated. The calculation is included by the control device 18 of information about the environment 50 and the flight situation of the aircraft 4, which are stored in the flight information unit 44 performed.

The ID associated with the flight obstacles 28, 30 is used together with the determined classification of the hazard potential of the flight obstacles 28, 30 of the Ground station 6 transmitted by radio, where these data using the device. 8 prepared for graphical representation and displayed on the display means 12 become. A possible representation on the display means 12 is shown in FIG. The display means 12 comprises a screen 62 on which an aircraft icon 64 is shown, with which the position of the aircraft 4 are shown should. On screen 62 is also the obstacle 28 through an object 66 and the trajectory 54 of the flying obstacle 28. In another and more conspicuous color and / or shape is an object 68 that represents the position of the flying obstacle 30. The objects 66, 68 are common, in TCAS (Traffic Collision Avoiding System) used symbols. In addition, the Ground pilot with a synthetic voice on all flight obstacles 30, 34 attentive made that exceed a given hazard class. It is also possible, the transmitted from the aircraft 4 identifier only in the form of an acoustic Output signals such as a voice or a tone sequence.

From the colors and / or shapes of the objects 66, 68 and / or an acoustic Signal, a ground pilot can directly detect the potential danger of flight obstacles 28, 30 recognize. As additional information, the trajectories 54, 56 serve him and the possible collision point 60. In addition, on the screen 13 (Figure 1) further additional information, such as the remaining reaction time until reaching the collision point 60 and information about the Flight situation of the aircraft and possibly on the environment of the aircraft 4, displayed.

If the remaining reaction time is above a predetermined time value, then The ground pilot can initiate an evasive maneuver, whereby he is free, the fallback proposal to follow along evasion trajectory 42 or another Route to choose. If the remaining reaction time is below the preset one Time value, then the flying of an evasive maneuver is automatically by the Control device 18 initiated. In this case, the control device act 18, the flight information unit 44, the flight control unit 46 and the flight actuator 48 in such a way that the aircraft 4 along the evasion trajectory 42 to be led.

The two of the evaluation device 26 not identified as aircraft Flight obstacles 32, 34 are examined on the basis of further pictures. This is detected by the evaluation device 26 that the flying obstacle 32 a very small Airspeed on a trajectory 72 has. This trajectory 72 is coming not near the trajectory 58 of the aircraft 4. In addition, it is recognized that the obstacle 32 is far away from the aircraft 4. The obstacle 32 associated hazard potential is therefore used by the control device 18 as classified so low that the ID assigned to the flight obstacle 32 is not the ground station 6 is transmitted. The decision as to the potential danger Flight obstacles displayed on the display means 12 of the ground station 6 can be set by a ground pilot by means of the control means 14 become.

The also not identifiable as aircraft aircraft obstacle 34 has a very low visual line rotation speed, so that the flying obstacle 34 a hazard potential is assigned, which is set by the ground pilot Display threshold exceeds. Because the object size in the sequence of recorded images does not grow, but the obstacle 34 can not be assigned more or less exact distance, with only detectable is that the flight obstacle is due to the non-growing object size in a relatively large distance. The the obstacle 34 associated Identification is thus transmitted to the ground station 6 and it becomes a possible Stay area 74 of the flying obstacle 34 34 displayed on the screen.

For additional information, the ground pilot can use the second camera unit 36 for taking a detail image in the visible spectral range sensitive to the obstacles 28, 30, 34. Through the camera unit 36 is an optically zoomable to a maximum of 10 ° by 10 ° detail image with a resolution of not more than 0.1 mrad recorded, the control device 18 passed, the ground station 6 transmitted and on the device 10 (FIG 1). The ground pilot can see on these detail images 38 that it is at the obstacle 30 to a traffic machine and the obstacle 28 is about a helicopter. The detail screens 38 are constantly updated, so that the ground pilot in each case a moving image of the flight obstacles 28, 30 is shown.

In the case of a lack of the second camera unit 36, a corresponding Control command of the ground pilot via the control device 18 to the evaluation device 26 are sent, the one the flying obstacles 28, 30 respectively Detail image extracted from a previously taken whole picture and the Control device 18 for transmission to the ground station 6 passes. For this the evaluation device 26 comprises a detail image unit for extracting a an obstacle obstacle 28, 30, 34 depicting detail image 38 and for passing the Detail picture 38 to the transmitting device.

Since the object size of the flying obstacle 34 is indicated as very small, the Ground pilot on a representation of the flying obstacle 34 in a further detail image Dispense on the flying obstacle 34 as a small weather balloon would have been recognized. Instead, the unit 20 for transmission and be activated to receive laser light and with the help of the unit 22 the Removal of the flying obstacle 34 are determined. The determined distance is supplied by the unit 22 of the control device 18, which is the obstacle If necessary, assign a new hazard potential and this together with the identifier of the flying obstacle 34 sends to the ground station 6. Relatively short time after activation of the unit 20 by the ground pilot Therefore, the residence area 74 on the screen 62 by another Object replaces the distance of the flying obstacle 34 reproduces. The color and shape of the object is the classification of the danger potential of the flying matter 34 adjusted.

The four cameras of the camera unit 24 for taking a complete picture are in visible spectral range sensitive. As well it is possible that these four Cameras in the infrared spectral range are sensitive, creating a night vision capability could be reached. As described above, the recognition of the Obstacles 28, 30, 32, 34 also in this case using one of the camera unit 24 recorded image sequence made. As another variant it is possible to equip the camera unit 24 with a number of cameras, from some in the infrared spectral range and others in the visual spectral range are sensitive. Detection of the flight obstacles 28, 30, 32, 34 can in this case, based on a sequence of images or comparing images be performed in different spectral ranges.

reference numeral

2
contraption
4
aircraft
6
ground station
8th
contraption
10
contraption
12
display means
13
screen
14
control means
16
arrow
18
control device
20
unit
22
unit
24
camera unit
26
evaluation
28
aviation obstruction
30
aviation obstruction
32
aviation obstruction
34
aviation obstruction
36
camera unit
38
detail screen
42
evasion
44
Flight information unit
46
Flight control unit
48
Flugaktorik
50
Surroundings
52
flight direction
54
trajectory
56
trajectory
58
trajectory
60
collision point
62
screen
64
Aircraft symbol
66
object
68
object
72
trajectory
74
lounge area

Claims (14)

  1. Method for identification of an obstruction to flight (28, 30, 32, 34) in an area (50) surrounding an airborne vehicle (4), in which at least two images of in each case at least one part of the surrounding area (50) are recorded, from which images the obstruction to flight (28, 30, 32, 34) is identified and is provided with an identification, and a signal (66, 68, 76) which is associated with the identification is emitted in a ground station (6)
    characterized in that
    the identification process and the allocation of the identification are carried out on board the airborne vehicle (4), and the identification is sent to the ground station (6).
  2. Method according to Claim 1,
    characterized in that
    the hazard potential of the obstruction to flight (28, 30, 32, 34) is classified on board the airborne vehicle (4) and, in particular, is sent to the ground station (6).
  3. Method according to Claim 1 or 2,
    characterized in that
    a measurement with respect to the obstruction to flight (28, 30, 32, 34) is carried out by means of an active measurement signal.
  4. Method according to one of the preceding claims,
    characterized in that
    an escape trajectory (42) is determined on board, and data associated with the escape trajectory (42) is transmitted to the ground station (6).
  5. Method according to Claim 4,
    characterized in that
    the escape trajectory (42) is determined including information, determined on board, about the surrounding area (50) and/or the flight situation of the airborne vehicle (4).
  6. Method according to Claim 4 or 5,
    characterized in that
    a decision as to whether the airborne vehicle (4) should fly on the escape trajectory (42) is made by a control apparatus (18) on board the airborne vehicle (4).
  7. Method according to one of the preceding claims,
    characterized in that
    a detailed image (38), which shows the obstruction to flight (28, 30), is sent to the ground station (6), where it is displayed.
  8. Method according to Claim 7,
    characterized in that
    an image detail of an image that is used for identification of the obstruction to flight (28, 30) is used as the detailed image (38).
  9. Method according to Claim 7 or 8,
    characterized in that the detailed image (38) is recorded in a spectral range which differs from the spectral range of the images which are used for identification of the obstruction to flight (28, 30, 32, 34).
  10. Method according to one of the preceding claims,
    characterized in that
    the images from which the obstruction to flight (28, 30, 32, 34) is identified are recorded in the infrared spectral range.
  11. Method according to one of the preceding claims,
    characterized in that
    a moving image of the obstruction to flight (28, 30) is sent to the ground station (6).
  12. Apparatus for identification of an obstruction to flight (28, 30, 32, 34) in an area (50) surrounding an airborne vehicle (4), having at least one camera unit (24) for recording at least a part of the surrounding area (50), and having an evaluation apparatus (26) for identification of the obstruction to flight (28, 30, 32, 34) and association of an identification with the obstruction to flight (28, 30, 32, 34),
    characterized in that
    the evaluation apparatus (26) is electrically connected to the camera unit (24) and to a transmission device for transmission of the identification to a ground station (6).
  13. Apparatus according to Claim 12,
    characterized by
    a detailed image unit for extraction of a detailed image (38), which images the obstruction to flight (28, 30), and for passing on the detailed image (38) to the transmission device.
  14. Apparatus according to Claim 12 or 13,
    characterized by
    a first camera unit (24) for recording an overall image in a first spectral range, and a second camera unit (36) for recording an image, in particular a detailed image (38), in a spectral range which is not the same as the first.
EP20040018510 2003-08-07 2004-08-05 Method and apparatus for flight obstacle recognition Active EP1505556B1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
DE10336928 2003-08-07
DE10336928 2003-08-07

Publications (2)

Publication Number Publication Date
EP1505556A1 EP1505556A1 (en) 2005-02-09
EP1505556B1 true EP1505556B1 (en) 2005-10-05

Family

ID=33547187

Family Applications (1)

Application Number Title Priority Date Filing Date
EP20040018510 Active EP1505556B1 (en) 2003-08-07 2004-08-05 Method and apparatus for flight obstacle recognition

Country Status (4)

Country Link
US (1) US7061401B2 (en)
EP (1) EP1505556B1 (en)
AT (1) AT306112T (en)
DE (1) DE502004000085D1 (en)

Families Citing this family (45)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4376653B2 (en) * 2004-02-17 2009-12-02 富士重工業株式会社 Outside monitoring device
US7228232B2 (en) * 2005-01-24 2007-06-05 International Business Machines Corporation Navigating a UAV with obstacle avoidance algorithms
US20060235584A1 (en) * 2005-04-14 2006-10-19 Honeywell International Inc. Decentralized maneuver control in heterogeneous autonomous vehicle networks
DE102005047273B4 (en) * 2005-10-01 2008-01-03 Eads Deutschland Gmbh Method for supporting low-level flights
US8855846B2 (en) * 2005-10-20 2014-10-07 Jason W. Grzywna System and method for onboard vision processing
US7876258B2 (en) 2006-03-13 2011-01-25 The Boeing Company Aircraft collision sense and avoidance system and method
US20100121575A1 (en) * 2006-04-04 2010-05-13 Arinc Inc. Systems and methods for aerial system collision avoidance
FR2913800B1 (en) * 2007-03-13 2010-08-20 Thales Sa Devices and methods for filtering field anti-collision alerts and obstacles for an aircraft
WO2008134815A1 (en) * 2007-05-04 2008-11-13 Teledyne Australia Pty Ltd. Collision avoidance system and method
DE102007032084A1 (en) * 2007-07-09 2009-01-22 Eads Deutschland Gmbh Collision and Conflict Prevention System for autonomous unmanned aerial vehicles (UAV)
US8868787B2 (en) * 2007-12-26 2014-10-21 Honeywell International Inc. Incremental downlink of flight information
DE102008024308B4 (en) * 2008-05-20 2010-12-09 Eads Deutschland Gmbh Method for detecting non-cooperative aviation on board an aircraft
FR2933523B1 (en) * 2008-07-01 2014-04-18 Airbus France Method and device for assisting the driving of a vehicle, in particular an aircraft, for the avoidance of obstacles
EP2159779B1 (en) * 2008-08-27 2013-01-16 Saab Ab Using image sensor and tracking filter time-to-go to avoid mid-air collisions
US8543265B2 (en) * 2008-10-20 2013-09-24 Honeywell International Inc. Systems and methods for unmanned aerial vehicle navigation
US8626361B2 (en) * 2008-11-25 2014-01-07 Honeywell International Inc. System and methods for unmanned aerial vehicle navigation
US8570211B1 (en) * 2009-01-22 2013-10-29 Gregory Hubert Piesinger Aircraft bird strike avoidance method and apparatus
US8380367B2 (en) * 2009-03-26 2013-02-19 The University Of North Dakota Adaptive surveillance and guidance system for vehicle collision avoidance and interception
US8494760B2 (en) * 2009-12-14 2013-07-23 American Aerospace Advisors, Inc. Airborne widefield airspace imaging and monitoring
US9355565B2 (en) * 2010-06-23 2016-05-31 Honeywell International Inc. Crossing traffic depiction in an ITP display
DE102011010679A1 (en) 2011-02-08 2012-08-09 Eads Deutschland Gmbh Unmanned aircraft with built-in collision warning system
DE102011016964A1 (en) 2011-04-13 2012-10-18 Diehl Bgt Defence Gmbh & Co. Kg Method for monitoring an air space around an aircraft
US8626428B2 (en) * 2011-06-28 2014-01-07 Honeywell International Inc. Selectable display of aircraft traffic on tracks
DE102011121616A1 (en) * 2011-12-20 2013-06-20 Audi Ag Method for controlling a display device of a motor vehicle
WO2013112016A1 (en) * 2012-01-27 2013-08-01 두산인프라코어 주식회사 Operational stability enhancing device for construction machinery
US9002719B2 (en) 2012-10-08 2015-04-07 State Farm Mutual Automobile Insurance Company Device and method for building claim assessment
CA2833985A1 (en) * 2012-11-19 2014-05-19 Rosemount Aerospace Inc. Collision avoidance system for aircraft ground operations
US10502584B1 (en) 2012-12-28 2019-12-10 Sean Patrick Suiter Mission monitor and controller for autonomous unmanned vehicles
US8872818B2 (en) 2013-03-15 2014-10-28 State Farm Mutual Automobile Insurance Company Methods and systems for capturing the condition of a physical structure
US9082015B2 (en) 2013-03-15 2015-07-14 State Farm Mutual Automobile Insurance Company Automatic building assessment
US8818572B1 (en) 2013-03-15 2014-08-26 State Farm Mutual Automobile Insurance Company System and method for controlling a remote aerial device for up-close inspection
US9674490B2 (en) * 2013-04-18 2017-06-06 Magna Electronics Inc. Vision system for vehicle with adjustable cameras
CN104854428B (en) 2013-12-10 2018-12-14 深圳市大疆创新科技有限公司 sensor fusion
US9562773B2 (en) 2014-03-15 2017-02-07 Aurora Flight Sciences Corporation Autonomous vehicle navigation system and method
US9875661B2 (en) 2014-05-10 2018-01-23 Aurora Flight Sciences Corporation Dynamic collision-avoidance system and method
EP3103043B1 (en) 2014-09-05 2018-08-15 SZ DJI Technology Co., Ltd. Multi-sensor environmental mapping
WO2016033796A1 (en) 2014-09-05 2016-03-10 SZ DJI Technology Co., Ltd. Context-based flight mode selection
US9583011B2 (en) * 2015-01-28 2017-02-28 Airbus Helicopters Aircraft system for signaling the presence of an obstacle, an aircraft equipped with this system, and method for the detection of an obstacle
US9607447B2 (en) 2015-03-31 2017-03-28 Honeywell International Inc. Automated transmission of aircraft anomalous incident data via preferred transmission modes
US9821910B1 (en) * 2015-05-19 2017-11-21 uAvionix Corporation Unmanned vehicle control system and apparatus
RU2661269C2 (en) * 2015-08-04 2018-07-13 Акционерное общество "Концерн радиостроения "Вега" Method for trajectory control of unmanned aerial vehicle to fly around urban territories in vertical plane
US9787951B2 (en) * 2015-12-18 2017-10-10 Serge Kannon Vehicle proximity warning system
US10332409B2 (en) * 2016-09-27 2019-06-25 Rockwell Collins, Inc. Midair collision threat detection and assessment using visual information
EP3566221A1 (en) 2017-01-06 2019-11-13 Aurora Flight Sciences Corporation Collision-avoidance system and method for unmanned aircraft
US10243647B2 (en) * 2017-05-30 2019-03-26 Bell Helicopter Textron Inc. Aircraft visual sensor system

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3992707A (en) 1974-09-13 1976-11-16 Vereinigte Flugtechnische Werke-Fokker Gesellschaft Mit Beschrankter Haftung Reproduction of a field of view as scanned by a remote controlled aircraft
US4405943A (en) * 1981-08-19 1983-09-20 Harris Corporation Low bandwidth closed loop imagery control and communication system for remotely piloted vehicle
US4918442A (en) 1988-10-03 1990-04-17 Bogart Jr Donald W Airplane collision avoidance system
DE4240789C2 (en) 1992-12-04 2003-08-28 Bodenseewerk Geraetetech Procedure for identifying objects
US5414439A (en) 1994-06-09 1995-05-09 Delco Electronics Corporation Head up display with night vision enhancement
US5581250A (en) 1995-02-24 1996-12-03 Khvilivitzky; Alexander Visual collision avoidance system for unmanned aerial vehicles
US5883586A (en) 1996-07-25 1999-03-16 Honeywell Inc. Embedded mission avionics data link system
DE19741896C2 (en) 1997-09-23 1999-08-12 Opel Adam Ag Apparatus for imaging of regions of the environment of a motor vehicle
US6400831B2 (en) * 1998-04-02 2002-06-04 Microsoft Corporation Semantic video object segmentation and tracking
US6480789B2 (en) 2000-12-04 2002-11-12 American Gnc Corporation Positioning and proximity warning method and system thereof for vehicle
DE10065180A1 (en) 2000-12-23 2002-07-11 Eads Deutschland Gmbh Aircraft collision avoidance method, involves determining edge information and object center of gravity profiles, extrapolating and predicting possible courses for comparison with aircraft path
US6804607B1 (en) * 2001-04-17 2004-10-12 Derek Wood Collision avoidance system and method utilizing variable surveillance envelope
DE10146514A1 (en) 2001-09-21 2003-04-10 Emt Ingenieurbuero Fuer Elektr Method and device for operating an unmanned aerial vehicle

Also Published As

Publication number Publication date
AT306112T (en) 2005-10-15
US7061401B2 (en) 2006-06-13
DE502004000085D1 (en) 2006-02-16
US20050109872A1 (en) 2005-05-26
EP1505556A1 (en) 2005-02-09

Similar Documents

Publication Publication Date Title
USRE44604E1 (en) System for and method of synchronous acquisition of pulsed source light in performance of monitoring aircraft flight operation
US7180476B1 (en) Exterior aircraft vision system using a helmet-mounted display
EP1709611B1 (en) Automatic taxi manager
KR101281899B1 (en) Aircraft collision sense and avoidance system and method
DE102004006033B3 (en) Method for detection and control of forest and wildfires
EP2177966A2 (en) Systems and methods for unmanned aerial vehicle navigation
EP2442134A1 (en) Image acquisition unit, acquisition method, and associated control unit
US9959774B2 (en) Systems and methods for displaying obstacle-avoidance information during surface operations
CA1338747C (en) Automatic landing and navigation system
EP2089677B1 (en) Methods, apparatus and systems for enhanced synthetic vision and multi-sensor data fusion to improve operational capabilities of unmanned aerial vehicles
KR100434747B1 (en) System to improve the function and operate surveillance in low-retry state
US7948404B2 (en) Obstacle avoidance situation display generator
US20100238161A1 (en) Computer-aided system for 360º heads up display of safety/mission critical data
US20100250022A1 (en) Useful unmanned aerial vehicle
JP2006027448A (en) Aerial photographing method and device using unmanned flying body
RU2523167C2 (en) Method of monitoring flight strip and system for implementing method
EP2820632B1 (en) System and method for multipurpose traffic detection and characterization
US8970401B2 (en) Using image sensor and tracking filter time-to-go to avoid mid-air collisions
JP4763610B2 (en) Video on demand method and apparatus
JP6110316B2 (en) Surveillance system and method for detecting foreign objects, debris, or damage in an airfield
US6804607B1 (en) Collision avoidance system and method utilizing variable surveillance envelope
EP1936584B1 (en) A device at an airborne vehicle and a method for collision avoidance
US5448233A (en) Airborne obstacle collision avoidance apparatus
JP2005265699A (en) System and method for inspecting power transmission line using unmanned flying body
US3053932A (en) Aircraft warning system

Legal Events

Date Code Title Description
AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PL PT RO SE SI SK TR

AX Request for extension of the european patent to:

Extension state: AL HR LT LV MK

RAP1 Rights of an application transferred

Owner name: DIEHL BGT DEFENCE GMBH & CO.KG

17P Request for examination filed

Effective date: 20050122

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20051005

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20051005

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20051005

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20051005

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20051005

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20051005

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20051005

AX Request for extension of the european patent to:

Extension state: AL HR LT LV MK

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PL PT RO SE SI SK TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

Free format text: NOT ENGLISH

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

AKX Designation fees paid

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PL PT RO SE SI SK TR

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

Free format text: LANGUAGE OF EP DOCUMENT: GERMAN

GBT Gb: translation of ep patent filed (gb section 77(6)(a)/1977)
PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20060105

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20060105

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20060105

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20060116

REG Reference to a national code

Ref country code: SE

Ref legal event code: TRGR

REF Corresponds to:

Ref document number: 502004000085

Country of ref document: DE

Date of ref document: 20060216

Kind code of ref document: P

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20060306

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20060406

REG Reference to a national code

Ref country code: IE

Ref legal event code: FD4D

ET Fr: translation filed
PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20060831

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20060831

26N No opposition filed

Effective date: 20060706

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20060805

BERE Be: lapsed

Owner name: DIEHL BGT DEFENCE G.M.B.H. & CO.KG

Effective date: 20060831

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20051005

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20060805

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20051005

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 12

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 13

REG Reference to a national code

Ref country code: DE

Ref legal event code: R081

Ref document number: 502004000085

Country of ref document: DE

Owner name: DIEHL DEFENCE GMBH & CO. KG, DE

Free format text: FORMER OWNER: DIEHL BGT DEFENCE GMBH & CO. KG, 88662 UEBERLINGEN, DE

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 14

PGFP Annual fee paid to national office [announced from national office to epo]

Ref country code: NL

Payment date: 20170821

Year of fee payment: 14

PGFP Annual fee paid to national office [announced from national office to epo]

Ref country code: GB

Payment date: 20170822

Year of fee payment: 14

Ref country code: FR

Payment date: 20170822

Year of fee payment: 14

Ref country code: IT

Payment date: 20170828

Year of fee payment: 14

PGFP Annual fee paid to national office [announced from national office to epo]

Ref country code: TR

Payment date: 20170726

Year of fee payment: 14

PGFP Annual fee paid to national office [announced from national office to epo]

Ref country code: CH

Payment date: 20180822

Year of fee payment: 15

PGFP Annual fee paid to national office [announced from national office to epo]

Ref country code: DE

Payment date: 20181023

Year of fee payment: 15

REG Reference to a national code

Ref country code: NL

Ref legal event code: MM

Effective date: 20180901

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20180805

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180901

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180805

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180831

PGFP Annual fee paid to national office [announced from national office to epo]

Ref country code: SE

Payment date: 20190821

Year of fee payment: 16

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180805