WO2023287276A1 - Procédés et systèmes de traitement de données géographiques pour détecter un empiètement par des objets dans un couloir géographique - Google Patents

Procédés et systèmes de traitement de données géographiques pour détecter un empiètement par des objets dans un couloir géographique Download PDF

Info

Publication number
WO2023287276A1
WO2023287276A1 PCT/MY2022/050064 MY2022050064W WO2023287276A1 WO 2023287276 A1 WO2023287276 A1 WO 2023287276A1 MY 2022050064 W MY2022050064 W MY 2022050064W WO 2023287276 A1 WO2023287276 A1 WO 2023287276A1
Authority
WO
WIPO (PCT)
Prior art keywords
geographic
machine learning
detection results
object detection
encroachment
Prior art date
Application number
PCT/MY2022/050064
Other languages
English (en)
Inventor
Tareq Aziz Hasan ALQUTAMI
Nian Wei TAN
Muhammad Arif B ABDUL RAHMAN
Naila Rusma Bt IDRIS
Mohd Nazri B AHMAD
Swee Ling KOI
Mohd Syazwan Kamil BIN ABDULLAH
Original Assignee
Petroliam Nasional Berhad (Petronas)
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Petroliam Nasional Berhad (Petronas) filed Critical Petroliam Nasional Berhad (Petronas)
Publication of WO2023287276A1 publication Critical patent/WO2023287276A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/20—Ensemble learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G06T7/001—Industrial image inspection using an image reference approach
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/254—Analysis of motion involving subtraction of images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G06V20/17—Terrestrial scenes taken from planes or by drones
    • G—PHYSICS
    • G08—SIGNALLING
    • G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
    • G08B13/00—Burglar, theft or intruder alarms
    • G08B13/18—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength
    • G08B13/189—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems
    • G08B13/194—Actuation by interference with heat, light, or radiation of shorter wavelength; Actuation by intruding sources of heat, light, or radiation of shorter wavelength using passive radiation detection systems using image scanning and comparing systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10032—Satellite or aerial image; Remote sensing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30181—Earth observation
    • G06T2207/30184—Infrastructure
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30232—Surveillance

Definitions

  • the present disclosure relates to geographic data processing methods and systems.
  • the present disclosure relates to methods and systems for detecting encroachment by objects into a geographic corridor.
  • Third party encroachment is a major threat to geographic corridors such as pipelines. Managing the risk from such encroachment is a real challenge to pipeline operators. If this threat is not properly managed, the consequences can lead to major catastrophe which impacts public safety, asset damage, financial loss and tarnish a company’s reputation.
  • a method of detecting encroachment by objects into a geographic corridor from geographic data comprises: receiving current geographic data comprising current image data of a geographic area comprising the geographic corridor and location tag data; generating a plurality of sets of machine learning object detection results from the current image data, each set of machine learning object detection results being generated using a different machine learning object detection methodology; combining the plurality of sets of machine learning object detection results to generate a combined set of machine learning object detection results; filtering the combined set of machine learning object detection results using location tag data associated with machine learning object detection results of the combined set of machine learning object detection results to generate a set of machine learning encroachment detections; and generating a set of encroachment detections comprising the set of machine learning encroachment detections.
  • the plurality of sets of machine learning object detection results comprises a set of machine learning object detection results generated from the current image data at a first resolution and a set of a set of machine learning object detection results generated from the current image data at a second resolution.
  • the plurality of sets of machine learning object detection results comprises a set of machine learning object detection results generated with a first object detection algorithm and a set of object detection results generated with a second object detection algorithm.
  • the set of encroachment detections comprises a set of confidence indications determined from the number of sets of machine learning object detection results from plurality of sets of machine learning object defection results in which each encroachment detection of the set of encroachment detections occurs.
  • the method further comprises: receiving historicai orthophoto data comprising historicai geographic data of the geographic area comprising the geographic corridor, the historic geographic data comprising historical image and location tag data; detecting changes between the historical image data and the current image data to generate a set of change detection results; and filtering the set change detection results using location tag data associated with the set of change detection results to generate a set of change detection encroachment detections, wherein the set of encroachment detections further comprises the set of change detection encroachment detections.
  • detecting changes between the historical image data and the current image data to generate a set of change detection results comprises appiying a piurality of image comparison methodologies to the historical image data and the current image data.
  • the method further comprises accessing a database storing information on mapped objects and locations corresponding to the mapped objects, and filtering the mapped objects to generate a set of mapped object detections comprising mapped objects having a location overlapping with the geographic corridor, wherein the set of encroachment detections further comprises the set of mapped object detections.
  • the method further comprises: receiving land lot information indicating an expected position of a target object in the geographic corridor; identifying an actual position of the target object in the geographic corridor; and determining a deviation between the actual position of the target object and the actual position of the target object in the geographic corridor.
  • the method further comprises generating a risk score for the geographic corridor using the set of encroachment detections.
  • a computer readable medium storing processor executable instructions which when executed on a processor cause the processor to carry out a method as set out above is provided.
  • a geographic data processing system for detecting encroachment by objects into a geographic corridor.
  • the geographic data processing system comprises: a processor and a data storage device storing computer program instructions operable to cause the processor to: receive current geographic data comprising current image data of a geographic area comprising the geographic corridor and location tag data; generate a plurality of sets of machine learning object detection results from the current image data, each set of machine learning object detection results being generated using a different machine learning object detection methodology; combine the plurality of sets of machine learning object detection results to generate a combined set of machine learning object detection results; filter the combined set of machine learning object detection results using location tag data associated with machine learning object detection results of the combined set of machine learning object detection results to generate a set of machine learning encroachment detections; and generate a set of encroachment detections comprising the set of machine learning encroachment detections.
  • the plurality of sets of machine learning object detection results comprises a set of machine learning object detection results generated from the
  • the plurality of sets of machine learning object detection results comprises a set of machine learning object detection results generated with a first object detection algorithm and a set of object detection results generated with a second object detection algorithm.
  • the set of encroachment detections comprises a set of confidence indications determined from the number of sets of machine learning object detection results from plurality of sets of machine learning object detection results in which each encroachment detection of the set of encroachment defections occurs.
  • the data storage device further stores computer program instructions operable to cause the processor to: receive historical orthophoto data comprising historical geographic data of the geographic area comprising the geographic corridor, the historic geographic data comprising historical image and location tag data; detect changes between the historical image data and the current image data to generate a set of change detection results; and filter the set change detection results using location tag data associated with the set of change detection results to generate a set of change detection encroachment detections, wherein the set of encroachment detections further comprises the set of change detection encroachment detections.
  • the data storage device further stores computer program instructions operable to cause the processor to: detect changes between the historical image data and the current image data to generate a set of change detection results by applying a plurality of image comparison methodologies to the historical image data and the current image data.
  • the data storage device further stores computer program instructions operable to cause the processor to: access a database storing information on mapped objects and locations corresponding to the mapped objects, and filter the mapped objects to generate a set of mapped object detections comprising mapped objects having a location overlapping with the geographic corridor, wherein the set of encroachment detections further comprises the set of mapped object detections.
  • the data storage device further stores computer program instructions operable to cause the processor to: receive land lot information indicating an expected position of a target object in the geographic corridor; identify an actual position of the target object in the geographic corridor; and determine a deviation between the actual position of the target object and the actual position of the target object in the geographic corridor.
  • the data storage device further stores computer program instructions operable to cause the processor to: generate a risk score for the geographic corridor using the set of encroachment detections.
  • a method of detecting encroachment by objects into a geographic corridor from geographic data comprising: receiving current geographic data comprising current image data of a geographic area comprising the geographic corridor and location tag data; generating a plurality of sets of machine learning object detection results from the current image data, each set of machine learning object detection results being generated using a different machine learning object detection methodology; combining the plurality of sets of machine learning object detection results to generate a combined set of machine learning object detection results; filtering the combined set of machine learning object defection results using location tag data associated with machine learning object detection results of the combined set of machine learning object detection results to generate a set of machine learning encroachment detections; and generating a set of encroachment detections comprising the set of machine learning encroachment detections.
  • the plurality of sets of machine learning object detection results comprises a set of machine learning object detection results generated from the current image data at a first resolution and a set of a set of machine learning object detection results generated from the current image data at a second resolution.
  • a method according to clause 1 or clause 2, wherein the plurality of sets of machine learning object detection results comprises a set of machine learning object detection results generated with a first object detection algorithm and a set of object detection results generated with a second object detection algorithm.
  • the set of encroachment detections comprises a set of confidence indications determined from the number of sets of machine learning object detection results from plurality of sets of machine learning object detection results in which each encroachment detection of the set of encroachment detections occurs.
  • a method further comprising: receiving historical orthophoto data comprising historical geographic data of the geographic area comprising the geographic corridor, the historic geographic data comprising historical image and location tag data; detecting changes between the historical image data and the current image data to generate a set of change detection results; and filtering the set change detection results using location tag data associated with the set of change detection results to generate a set of change detection encroachment detections, wherein the set of encroachment detections further comprises the set of change detection encroachment detections.
  • detecting changes between the historical image data and the current image data to generate a set of change detection results comprises applying a plurality of image comparison methodologies to the historical image data and the current image data.
  • a method further comprising accessing a database storing information on mapped objects and locations corresponding to the mapped objects, and filtering the mapped objects to generate a set of mapped object detections comprising mapped objects having a location overlapping with the geographic corridor, wherein the set of encroachment detections further comprises the set of mapped object detections,
  • a method further comprising: receiving land lot information indicating an expected position of a target object in the geographic corridor; identifying an actual position of the target object in the geographic corridor; and determining a deviation between the actual position of the target object and the actual position of the target object in the geographic corridor.
  • a method according to any preceding clause further comprising generating a risk score for the geographic corridor using the set of encroachment detections.
  • a computer readable medium storing processor executable instructions which when executed on a processor cause the processor to carry out a method according to any one of clauses 1 to 9.
  • a geographic data processing system for detecting encroachment by objects into a geographic corridor, the geographic data processing system comprising: a processor and a data storage device storing computer program instructions operable to cause the processor to: receive current geographic data comprising current image data of a geographic area comprising the geographic corridor and location tag data; generate a plurality of sets of machine learning object detection results from the current image data, each set of machine learning object detection results being generated using a different machine learning object detection methodology; combine the plurality of sets of machine learning object detection results to generate a combined set of machine learning object detection results; filter the combined set of machine learning object detection results using location tag data associated with machine learning object detection results of the combined set of machine learning object detection results to generate a set of machine learning encroachment detections; and generate a set of encroachment detections comprising the set of machine learning encroachment detections.
  • a geographic data processing system according to clause 11, wherein the plurality of sets of machine learning object detection results comprises a set of machine learning object detection results generated from the current image data at a first resolution and a set of a set of machine learning object detection results generated from the current image data at a second resolution.
  • a geographic data processing system according to clause 11 or clause 12, wherein the plurality of sets of machine learning object detection results comprises a set of machine learning object detection results generated with a first object detection algorithm and a set of object detection results generated with a second object detection algorithm,
  • a geographic data processing system according to any one of clauses 11 to 13, wherein the set of encroachment detections comprises a set of confidence indications determined from the number of sets of machine learning object detection results from plurality of sets of machine learning object detection results in which each encroachment detection of the set of encroachment detections occurs.
  • a geographic data processing system according to clause 11 , wherein the data storage device further stores computer program instructions operable to cause the processor to: receive historical orthophoto data comprising historical geographic data of the geographic area comprising the geographic corridor, the historic geographic data comprising historical image and location tag data; detect changes between the historical image data and the current image data to generate a set of change detection results; and filter the set change detection results using location tag data associated with the set of change detection results to generate a set of change detection encroachment detections, wherein the set of encroachment detections further comprises the set of change detection encroachment detections. 18.
  • a geographic data processing system according to clause 15, wherein the data storage device further stores computer program instructions operable to cause the processor to: detect changes between the historical image data and the current image data to generate a set of change detection results by applying a plurality of image comparison methodologies to the historical image data and the current image data.
  • a geographic data processing system according to any one of clauses 11 to 16 wherein the data storage device further stores computer program instructions operable to cause the processor to: access a database storing information on mapped objects and locations corresponding to the mapped objects, and filter the mapped objects to generate a set of mapped object detections comprising mapped objects having a location overlapping with the geographic corridor, wherein the set of encroachment detections further comprises the set of mapped object detections.
  • a geographic data processing system according to any one of clauses 11 to 17 wherein the data storage device further stores computer program instructions operable to cause the processor to: receive land lot information indicating an expected position of a target object in the geographic corridor; identify an actual position of the target object in the geographic corridor; and determine a deviation between the actual position of the target object and the actual position of the target object in the geographic corridor,
  • a geographic data processing system according to any one of clauses 11 to 18 wherein the data storage device further stores computer program instructions operable to cause the processor to: generate a risk score for the geographic corridor using the set of encroachment detections,
  • FIG.1 is a block diagram showing a geographic data processing system according to an embodiment of the present invention.
  • FIG.2 is a flow chart showing method of processing geographic data to detect encroachment by objects into a geographic corridor according to an embodiment of the present invention
  • FIG.3 is a flow chart showing method of processing geographic data to detect encroachment by objects into a geographic corridor according to an embodiment of the present invention
  • FIG.4 is a flowchart showing an overview of processing in a method according to an embodiment of the present invention.
  • FIG.5 is a flowchart showing a machine learning workflow in a method according to an embodiment of the present invention.
  • FIG.6 is a flowchart showing processing of mapped object data in a method according to an embodiment of the present invention
  • FIG.7 is a flowchart showing an image processing workflow in a method according to an embodiment of the present invention.
  • FIG.8 is a flowchart showing an image processing workflow in a method according to an embodiment of the present invention.
  • the present disclosure relates to the autonomous detection of third-party threats such as intrusion, encroachment, and crossing of a geographic corridor such a pipeline from aerial visual imagery collected by aerial vehicle survey or satellite imagery.
  • the system may be capable of detecting the following: vehicles and machinery within and beside the geographic corridor, buildings or structures encroaching or beside the geographic corridor, any suspicious changes within the geographic corridor such as foreign objects, and utilities such as transmission towers within or encroaching the geographic corridor.
  • the system may also be configured for the following: identification of location class through counting the number of buildings within the geographic corridor, deviations of objects such as pipelines from their land lots, and detection of roads or waterways crossing the geographic corridor.
  • FIG.1 shows a geographic data processing system according to an embodiment of the present invention.
  • the geographic data processing system 100 is a computer system with memory that stores computer program modules which implement geographic data processing methods according to embodiments of the present invention.
  • the geographic data processing system 100 comprises a processor 110, a working memory 112, an input interface 114, a user interface 116, an output interface 118, program storage 120 and data storage 140.
  • the processor 110 may be implemented as one or more central processing unit (CPU) chips.
  • the program storage 120 is a non-volatile storage device such as a hard disk drive which stores computer program modules. The computer program modules are loaded into the working memory 112 for execution by the processor 110.
  • the input interface 114 is an interface which allows data, such as image data to be received by the geographic data processing system 100.
  • the input interface 114 may be a wireless network interface such as a Wi-Fi or Bluetooth interface, alternatively it may be a wired interface.
  • the user interface 116 allows a user of the geographic data processing system 100 to input selections and commands and may be implemented as a graphical user interface.
  • the output interface 118 outputs data and may be implemented as a display or a data interface.
  • the program storage 120 stores a machine learning object detection module 122, an image processing change detection module 124, a mapped object detection module 126, a land lot deviation detection module 128, an encroachment detection module 130, and a risk score calculation module 132.
  • the computer program modules cause the processor 110 to execute various geographic data processing which is described in more detail below.
  • the program storage 120 may be referred to in some contexts as computer readable storage media and/or non-transitory computer readable media.
  • the computer program modules are distinct modules which perform respective functions implemented by the geographic data processing system 100. It will be appreciated that the boundaries between these modules are exemplary only, and that alternative embodiments may merge modules or impose an alternative decomposition of functionality of modules.
  • modules discussed herein may be decomposed into sub-modules to be executed as multiple computer processes, and, optionally, on multiple computers.
  • alternative embodiments may combine multiple instances of a particular module or sub-module.
  • software implementation of the computer program modules is described herein, these may alternatively be implemented as one or more hardware modules (such as field-programmable gate array(s) or application-specific integrated circuit(s)) comprising circuitry which implements equivalent functionality to that implemented in software.
  • the data storage 140 stores machine learning model 150.
  • the machine learning models 150 comprise multiple models which are trained for image recognition. As shown in FIG.1 , the machine learning models 150 comprise a first model for recognizing a first object 152, a second model for recognizing the first object 154, a first model for recognizing a second object 156 and a second model for recognizing the second object 158.
  • the geographic data processing system 100 is coupled to a first object map database 160 and a second object map database 162.
  • the first object map database 160 and the second object map database 162 are databases which store labelled maps of objects such as roads, utility towers and transmission lines.
  • FIG.2 is a flow chart showing method of processing geographic data to detect encroachment by objects into a geographic corridor according to an embodiment of the present invention.
  • the method 200 shown in FIG.2 is carried out by the geographic data processing system 100 shown in FIG.1.
  • the geographic data processing system 100 receives current geographic data.
  • the current geographic data is received by the input interface 114 of the geographic data processing system 100.
  • the geographic data comprises current image data of a geographic corridor and location tag data which identifies the locations of the current image data.
  • the current geographic data may comprise orthophoto data, or any image captured from above data such as satellite imagery, or drone images and videos.
  • the machine learning object detection module 122 is executed by the processor 110 of the geographic data processing system 100 to generate a plurality of sets of machine learning object detection results from the current image data.
  • Each of the sets of machine learning object detection results is generated using a different machine learning methodology.
  • different machine learning object detection results may be generated by using different machine learning models stored in the data storage 140.
  • a first set of machine learning object detection results may be generated using the first machine learning model for the first object 152 and a second set of machine learning object detection results may be generated using the second machine learning model for the first object 156.
  • different machine learning object detection results may be generated by converting the current image data to different resolutions and applying the machine learning models to the different resolutions of image data.
  • step 204 multiple sets of machine learning object detection results are generated, these sets of machine learning object detections may comprise duplicate detections corresponding to the same object being detected by different methodologies.
  • the machine learning object detection module 122 is executed by the processor 110 of the geographic data processing system 100 to combine the plurality of sets of machine learning object detection results to obtain a combined set of machine learning object detection results.
  • the combined results may be given a confidence indication depending on the number object detection sets from the plurality of object detection sets in which a detection occurs. Thus, if an object is detected by multiple machine learning object detection methodologies then that detection is given a high confidence indication.
  • the machine learning object detection module 122 is executed by the processor 110 of the geographic data processing system 100 to filter the machine learning detections based on location.
  • the location tag data indicates locations associated with the current image data.
  • a set of machine learning encroachment detections is generated by filtering the combined set of machine learning object detection results based on their location. Object detection results having a location corresponding to the geographic corridor are identified as being in the set of machine learning encroachment detections.
  • the encroachment detection module 130 is executed by the processor 110 of the geographic data processing system 100 to generate a set of encroachment detections.
  • the set of encroachment detections may correspond to the set of machine learning encroachment detections. In some embodiments as described more detail below, the set of encroachment detections may include detections generated by other detection methods.
  • the risk score calculation module 132 is executed by the processor 110 of the geographic data processing system to calculate a risk score for the geographic corridor using the set of encroachment detections.
  • the final list of detections are passed to a rule-based engine that consumes the detections and their meta data (such as the type of detection such as vehicle, building, road crossing, and machinery) to produce a final risk score for each area of interest segment (for example a pipe segment).
  • the geographic corridor may be divided into segments (i.e. each 1km is one segment), the goal is to come up with a final third-party threat score for each segment based on all the detections from drone surveys and manual patrolling.
  • the score may be used to provision resources to each segment for further routine inspections patrolling.
  • the score may be calculated according to the following rules: if a segment is found to have machinery (e.g. excavator) within 20m from a pipeline, then the risk score is increased by 10, if there is a highway crossing the segment then the risk score is increased by 2; if there is a 1-lane road crossing the segment, then the risk score is increased by 5; if there are objects (such as buildings, garbage, etc.) detected within pipeline right of way, then the risk score is increased by 10; if there are industrial facilities within 50m from pipeline then the risk score is increased by 5; if the segment is within a predefined location class A (more than 10 buildings within 200m) then the risk score is increased by 10.
  • the final score represents an estimate of the chance of third-party encroachments/intrusion or damage to pipeline.
  • FIG.3 is a flow chart showing method of processing geographic data to detect encroachment by objects into a geographic corridor according to an embodiment of the present invention.
  • the method 300 shown in FIG.3 includes the steps of the method 200 described above with reference to FIG.2 and also includes additional steps which allow additional processing to influence the encroachment detections and the risk score.
  • the method 300 shown in FIG.3 is carried out by the geographic data processing system 100 shown in FIG.1.
  • the geographic data processing system 100 receives current geographic data.
  • the current geographic data is received by the input interface 114 of the geographic data processing system 100.
  • the geographic data comprises current image data of a geographic corridor and location tag data which identifies the locations of the current image data.
  • the current geographic data may comprise orthophoto data, or any image captured from above such as satellite imagery, or drone images and videos.
  • step 304 the machine learning object detection module 122 is executed by the processor 110 of the geographic data processing system 100 to perform machine learning object detection.
  • Step 304 corresponds to steps 204, 206 and 208 described above with reference to FIG.2, and the output is a set of machine learning encroachment detections.
  • the geographic data processing system 100 receives historic geographic data.
  • the historic geographic data comprises historic image data of the geographic corridor and location tag data which identifies the locations of the historic image data.
  • the historic geographic data may comprise orthophoto data, or any image captured from above such as satellite imagery, or drone images and videos.
  • the image processing change detection module is executed by the processor 100 of the geographic data processing system 100 to perform change detection on the geographic data to generate a set of change detection results. As will be described in more detail below, the change detection results are generated by comparing the current image data with the historic image data.
  • step 310 the mapped object detection module 126 is executed by the processor 100 of the geographic data processing system 100 to access mapped object data from the first object map database 160 and the second object map database 162. This returns indications of mapped objects within the geographic corridor.
  • the mapped object detection module 126 is executed by the processor 100 of the geographic data processing system 100 to generate mapped object detections.
  • the mapped object detections are a set of indications of mapped objects which are located in the geographic corridor.
  • the geographic data processing system 100 receives land lot information.
  • the land lot information comprises indications expected positions of a target object in the geographic corridor.
  • the land lot information may comprise indications of land purchased as a corridor for a pipeline, in which case, the pipeline would be expected to be positioned within the corridor for the pipeline.
  • step 316 the land lot deviation module 128 is executed by the processor 110 of the geographic data processing system to determine a deviation between an actual position of the target object in the geographic corridor and an expected position from the land lot information received in step 314.
  • the encroachment detection module 130 is executed by the processor 110 of the geographic data processing system 100 to generate a set of encroachment detections.
  • the set of encroachment detections is generated from the set of machine learning encroachment detections, the set of change detection results, and the set of mapped object detections.
  • the risk score calculation module 132 is executed by the processor 110 of the geographic data processing system 100 to generate a risk score based on the encroachment detections.
  • the risk score may be output from the output interface 118 of the geographic data processing system.
  • FIG.4 is a flowchart showing an overview of processing in a method according to an embodiment of the present invention.
  • the processing 400 shown in FIG.4 is carried out by the geographic data processing system 100 shown in FIG.1 and corresponds to parts of the methods 200 and 300 shown in FIG.2 and FIG.3.
  • the inputs to the processing 400 are image data 410, pipeline coordinates 412 and land lot data 414.
  • the processing 400 relates to a pipeline.
  • the image data 410 comprises images captured of the geographic corridor including the pipeline and may comprise orthophoto data including tags which identify the geographic locations of the image data.
  • the pipeline coordinates 412 comprises a set of coordinates of the pipeline.
  • the land lot data 414 comprises a set of indications of the planned locations of the pipeline and the land lots purchased for the pipeline.
  • the image data 416 is subjected to pre-processing 416 which includes image normalization, image scaling, band arrangement, format transformation. Then the pre- processed image data is subjected to three parallel processing streams: a machine learning workflow 420, data mining 430 and computer vision 440.
  • machine learning workflow 420 machine learning object detection results are generated.
  • the machine learning workflow 420 is followed by a morphology 422 step in which the detections from the machine learning workflow are cleaned. This is followed by a vectorization 424 step in which the detections are converted into polygons.
  • mapped object detections are generated by querying the first object map database 160 and the second object map database 162.
  • change detection algorithms are used to generate change detection results by comparing current image data with historical image data. Detections from the machine learning workflow 420, the data mining 430 and the computer vision 440 processing are combined by a combiner 450.
  • the combiner 450 Combines all detections from machine learning, computer vision and data mining.
  • the pipeline coordinates 412 are subjected to buffering 418 which adds a certain distance around the vector line of the pipeline and transforms the vector line into polygons which indicate the geographic corridor 460 which corresponds to the pipeline right of way onto which encroachments are detected.
  • the geographic corridor 460 is used to filter the detections. The detections are filtered to those within the area of interest only.
  • the land lot data 414 is are used to detect deviations between pipeline coordinates and land lot coordinates so that the actual route of the pipeline can be compared with the planned route.
  • a geo-filtering 470 processing step the combined detections are filtered using the geographic corridor 460 in order to identify objects within the right of way of the pipeline.
  • the outputs from the processing 400 are detections 472, statistics 474 and integration 476.
  • the final detections may be fed into a rule-based engine to produce threat score of each pipeline segment.
  • Each detection may comprise: the location and shape (polygon or square around the object). Represented in vector format (GEOJSON geometry), meta data: this can include: Detection id: id of this detection (unique); class type: what is the detection (car, machinery, road, building, transmission tower, waterway, unknown object, etc); road name: for roads only; road type: highway, primary, secondary, pathway, etc; color: color to represent this object in the map; and confidence indicator: how confident the system is of this detection.
  • FIG.5 is a flowchart showing a machine learning workflow in a method according to an embodiment of the present invention.
  • the machine learning workflow 500 shown in FIG.5 corresponds to the machine learning workflow 420 shown in FIG.4 and is an implementation of step 304 of the method 300 shown in FIG.3 and steps 204 to 206 shown in FIG.2.
  • the machine learning workflow 500 shown in FIG.5 is carried out by the machine learning object detection module 122 shown in FIG.1.
  • the input into the machine learning workflow 500 is image data 502 which corresponds to images of the geographic corridor.
  • the image data 502 is subjected to pre-processing 504.
  • the pre-processing 504 comprises transforming the image data into different resolutions. This transformation is done to fit different machine learning models as different models may only accept certain resolutions of image.
  • the image data 502 is transformed into 5cm / pixel resolution 506, 15 cm / pixel resolution 508 and other resolutions 510.
  • the pre-processed image data is then fed into a plurality of machine learning models.
  • the machine learning models may be for example artificial neural networks trained on image data sets to recognize certain objects.
  • the machine learning models comprise a first car model 520, a second car model 522, a first road model 524 and a second car model 526. It will be appreciated that these are intended as illustrative examples and a plurality of different models for each of a plurality of different object types may be used.
  • the different resolution images may be fed into each of the machine learning models to generate multiple sets of results.
  • results from the different models are combined by a combiner algorithm.
  • the results from the first car model 520 and the second car model 522 are combined by an ensemble algorithm 530 and the results from the first road model 524 and the second road model 526 are combined by an ensemble algorithm 532.
  • the ensemble algorithms may be majority voting ensembles.
  • the results from the different models are combined in an aggregate algorithm 540.
  • the outputs from the machine learning models are images which are huge, and these may be transformed into vector formats based on GIS vector format standards which are much more lightweight.
  • each of the models may yield different results.
  • the different models which detect the same object may be trained using different datasets.
  • the combined results may include a confidence indicator. If more models produce the same prediction, the confidence for that prediction will be higher.
  • Results from different resolutions may be compared and if a detection in the same location is found in both resolutions, take one of them and increase confidence.
  • the use of different resolutions improves the performance and reduces the rate of false negatives.
  • Machine learning models are trained on specific resolution and images come in different resolutions, having detections from multiple resolutions will ensure the maximum number of detections are found.
  • FIG.6 is a flowchart showing processing of mapped object data in a method according to an embodiment of the present invention.
  • the method 600 shown in FIG.6 corresponds to steps 310 and 312 of the method 300 shown in FIG.3 and the data mining 430 processing shown in FIG.4.
  • the method 600 shown in FIG.6 is carried out by the mapped object detection module 126 of the geographic data processing system 100 shown in FIG.1.
  • the inputs to the method 600 are image data 602 which as discussed above is images such as orthophotos of the geographic corridor and tags 604 which are a manual collection of keywords to be used in data mining based on application program interface (API) docs.
  • API application program interface
  • Boundary extraction 606 is performed on the image data 606. Then the tags 604 are used to obtain open street map (OSM) overpass data from an application program interface (API) 608. Alternative mapping data such as Google Maps or Bing Maps may also be used.
  • the collected data is in OSM vector format.
  • the collected data is then subjected to transformation 610 to convert from OSM format and labeling to GEOJSON format.
  • filtering 612 is performed to remove unnecessary or irrelevant data.
  • the outputs from the method are detections 614 and metadata 616.
  • the metadata 616 comprises additional information relating to the detections 614 such as the name of the street or road and the type of the road.
  • FIG.7 is a flowchart showing an image processing workflow in a method according to an embodiment of the present invention.
  • the method 700 shown in FIG.7 corresponds to steps 306 and 308 of the method 300 shown in FIG.3 and the computer vision 440 processing shown in FIG.4.
  • the method 700 shown in FIG.7 is carried out by the image processing change detection module 124 of the geographic data processing system 100 shown in FIG.1.
  • the inputs to the method 700 are current image data 702 and historic image data 704.
  • the current image data 702 and the historic image data 704 are sets of image data captured from above, for example orthophoto data and may include location data indicating the locations corresponding to the image data.
  • the historic image data 704 may be image data captured during a previous survey of the geographic corridor which is used as a reference to detect changes.
  • the current image data 702 is subjected to pre-processing 706 and the historic image data 704 is subjected to pre-processing 708.
  • Each of the pre-processing steps 706 and 708 comprise normalization and scaling.
  • the pre-processed current image data and the pre-processed historic image data are input into a change detection ensemble 710.
  • the change detection ensemble 710 is an ensemble of algorithms used to detect changes between the current image data 702 and the historic image data 704.
  • detections are subjected to morphology 712 which comprises filtering noise.
  • the filtered detections are subjected to masking 716 and vectorization 716.
  • the masking 716 extracts detections as a binary mask and the vectorization converts the binary mask into polygons.
  • the output is detections 718.
  • FIG.8 is a flowchart showing an image processing workflow in a method according to an embodiment of the present invention.
  • the method 800 shown in FIG.8 corresponds to the change detection ensemble 710 described above with reference to FIG.7.
  • the inputs to the method 800 are current image data 802 and historic image data 804.
  • the current image data 802 and the historic image data 804 are sets of image data captured from above, for example orthophoto data and may include location data indicating the locations corresponding to the image data.
  • the historic image data 804 may be image data captured during a previous survey of the geographic corridor which is used as a reference to detect changes.
  • the current image data 802 is subjected to pre-processing 806 and the historic image data 804 is subjected to pre-processing 808.
  • Each of the pre-processing steps 806 and 808 comprise normalization and scaling.
  • the pre-processed current image data and the pre-processed historic image data are input into three different image change detection algorithms: a Gaussian mixture background subtractor 820, a pixel subtraction 830 workflow, and a structural similarity index measure 840 workflow.
  • the Gaussian mixture background subtractor 820 is a segmentation algorithm for background segmentation. It is followed by a morphology 822 operation.
  • the morphological operations may comprise the use of opening followed by closing operations.
  • the pixel subtraction 830 workflow comprises a Gaussian filter 832 followed by a morphology 836 operation and an adaptive Gaussian filter 834 followed by a morphology operation 838.
  • the structural similarity index measure 840 workflow comprises determination of a structural similarity index followed by binary thresholding 842, a find contours operation 844 followed by a morphology operation 846.
  • the detections from the different workflows are combined by a combiner 850 to give the final set of change detection encroachment detections.
  • the combiner 850 may be implemented using an OR binary operation. Alternatively, intensity summation may be used and areas with higher intensity means higher confidence of change detection.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Remote Sensing (AREA)
  • Health & Medical Sciences (AREA)
  • Quality & Reliability (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Image Analysis (AREA)

Abstract

L'invention divulgue des procédés et des systèmes de traitement de données géographiques permettant de détecter l'empiètement par des objets dans une zone géographique. Un procédé de détection d'empiètement par des objets dans un couloir géographique à partir de données géographiques comprend les étapes suivantes : recevoir des données géographiques actuelles contenant des données d'image actuelle d'une zone géographique comprenant le corridor géographique et des données d'étiquette de localisation ; générer une pluralité d'ensembles de résultats de détection d'objet d'apprentissage automatique à partir des données d'image actuelles, chaque ensemble de résultats de détection d'objet d'apprentissage automatique étant généré à l'aide d'une méthodologie de détection d'objet d'apprentissage automatique différente ; combiner la pluralité d'ensembles de résultats de détection d'objet d'apprentissage automatique pour générer un ensemble combiné de résultats de détection d'objet d'apprentissage automatique ; filtrer l'ensemble combiné de résultats de détection d'objet d'apprentissage automatique à l'aide de données d'étiquette de localisation associées à des résultats de détection d'objet d'apprentissage automatique de l'ensemble combiné de résultats de détection d'objet d'apprentissage automatique pour générer un ensemble de détections d'empiètement par apprentissage automatique ; et générer un ensemble de détections d'empiètement comprenant l'ensemble de détections d'empiètement par apprentissage automatique.
PCT/MY2022/050064 2021-07-16 2022-07-15 Procédés et systèmes de traitement de données géographiques pour détecter un empiètement par des objets dans un couloir géographique WO2023287276A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
MYPI2021004069 2021-07-16
MYPI2021004069 2021-07-16

Publications (1)

Publication Number Publication Date
WO2023287276A1 true WO2023287276A1 (fr) 2023-01-19

Family

ID=84919589

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/MY2022/050064 WO2023287276A1 (fr) 2021-07-16 2022-07-15 Procédés et systèmes de traitement de données géographiques pour détecter un empiètement par des objets dans un couloir géographique

Country Status (1)

Country Link
WO (1) WO2023287276A1 (fr)

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11752472B2 (en) 2019-12-30 2023-09-12 Marathon Petroleum Company Lp Methods and systems for spillback control of in-line mixing of hydrocarbon liquids
US11754225B2 (en) 2021-03-16 2023-09-12 Marathon Petroleum Company Lp Systems and methods for transporting fuel and carbon dioxide in a dual fluid vessel
US11774990B2 (en) 2019-12-30 2023-10-03 Marathon Petroleum Company Lp Methods and systems for inline mixing of hydrocarbon liquids based on density or gravity
US11794153B2 (en) 2019-12-30 2023-10-24 Marathon Petroleum Company Lp Methods and systems for in-line mixing of hydrocarbon liquids
US11807945B2 (en) 2021-08-26 2023-11-07 Marathon Petroleum Company Lp Assemblies and methods for monitoring cathodic protection of structures
US11808013B1 (en) 2022-05-04 2023-11-07 Marathon Petroleum Company Lp Systems, methods, and controllers to enhance heavy equipment warning
US11815227B2 (en) 2021-03-16 2023-11-14 Marathon Petroleum Company Lp Scalable greenhouse gas capture systems and methods
US12006014B1 (en) 2023-02-18 2024-06-11 Marathon Petroleum Company Lp Exhaust vent hoods for marine vessels and related methods
US12012082B1 (en) 2022-12-30 2024-06-18 Marathon Petroleum Company Lp Systems and methods for a hydraulic vent interlock
US12012883B2 (en) 2021-03-16 2024-06-18 Marathon Petroleum Company Lp Systems and methods for backhaul transportation of liquefied gas and CO2 using liquefied gas carriers
US12043361B1 (en) 2023-02-18 2024-07-23 Marathon Petroleum Company Lp Exhaust handling systems for marine vessels and related methods
US12043905B2 (en) 2021-08-26 2024-07-23 Marathon Petroleum Company Lp Electrode watering assemblies and methods for maintaining cathodic monitoring of structures
US12087002B1 (en) 2023-09-18 2024-09-10 Marathon Petroleum Company Lp Systems and methods to determine depth of soil coverage along a right-of-way
US12109543B2 (en) 2019-12-30 2024-10-08 Marathon Petroleum Company Lp Methods and systems for operating a pump at an efficiency point
US12129559B2 (en) 2021-08-26 2024-10-29 Marathon Petroleum Company Lp Test station assemblies for monitoring cathodic protection of structures and related methods

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017117147A (ja) * 2015-12-24 2017-06-29 前田建設工業株式会社 構造物の施工管理方法
KR101765235B1 (ko) * 2016-11-28 2017-08-04 한국건설기술연구원 사물인터넷 기반 센서와 무인비행체를 이용한 시설물 유지관리 시스템 및 그 방법
KR101772916B1 (ko) * 2016-12-30 2017-08-31 한양대학교 에리카산학협력단 촬영영상과 인공지능 알고리즘 기반의 균열검출 자동화 프로그램을 이용한 터널 라이닝 표면의 균열 검출 방법 및 시스템
US20190057507A1 (en) * 2017-08-18 2019-02-21 Samsung Electronics Co., Ltd. System and method for semantic segmentation of images
KR101949525B1 (ko) * 2017-11-16 2019-03-18 주식회사 아이오티봇 무인탐지장치를 이용한 현장 안전관리 시스템

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017117147A (ja) * 2015-12-24 2017-06-29 前田建設工業株式会社 構造物の施工管理方法
KR101765235B1 (ko) * 2016-11-28 2017-08-04 한국건설기술연구원 사물인터넷 기반 센서와 무인비행체를 이용한 시설물 유지관리 시스템 및 그 방법
KR101772916B1 (ko) * 2016-12-30 2017-08-31 한양대학교 에리카산학협력단 촬영영상과 인공지능 알고리즘 기반의 균열검출 자동화 프로그램을 이용한 터널 라이닝 표면의 균열 검출 방법 및 시스템
US20190057507A1 (en) * 2017-08-18 2019-02-21 Samsung Electronics Co., Ltd. System and method for semantic segmentation of images
KR101949525B1 (ko) * 2017-11-16 2019-03-18 주식회사 아이오티봇 무인탐지장치를 이용한 현장 안전관리 시스템

Cited By (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12128369B2 (en) 2019-12-30 2024-10-29 Marathon Petroleum Company Lp Methods and systems for in-line mixing of hydrocarbon liquids
US11774990B2 (en) 2019-12-30 2023-10-03 Marathon Petroleum Company Lp Methods and systems for inline mixing of hydrocarbon liquids based on density or gravity
US11794153B2 (en) 2019-12-30 2023-10-24 Marathon Petroleum Company Lp Methods and systems for in-line mixing of hydrocarbon liquids
US11752472B2 (en) 2019-12-30 2023-09-12 Marathon Petroleum Company Lp Methods and systems for spillback control of in-line mixing of hydrocarbon liquids
US12011697B2 (en) 2019-12-30 2024-06-18 Marathon Petroleum Company Lp Methods and systems for spillback control of in-line mixing of hydrocarbon liquids
US12109543B2 (en) 2019-12-30 2024-10-08 Marathon Petroleum Company Lp Methods and systems for operating a pump at an efficiency point
US12066843B2 (en) 2019-12-30 2024-08-20 Marathon Petroleum Company Lp Methods and systems for inline mixing of hydrocarbon liquids based on density or gravity
US11754225B2 (en) 2021-03-16 2023-09-12 Marathon Petroleum Company Lp Systems and methods for transporting fuel and carbon dioxide in a dual fluid vessel
US11774042B2 (en) 2021-03-16 2023-10-03 Marathon Petroleum Company Lp Systems and methods for transporting fuel and carbon dioxide in a dual fluid vessel
US11815227B2 (en) 2021-03-16 2023-11-14 Marathon Petroleum Company Lp Scalable greenhouse gas capture systems and methods
US11988336B2 (en) 2021-03-16 2024-05-21 Marathon Petroleum Company Lp Scalable greenhouse gas capture systems and methods
US12000538B2 (en) 2021-03-16 2024-06-04 Marathon Petroleum Company Lp Systems and methods for transporting fuel and carbon dioxide in a dual fluid vessel
US12012883B2 (en) 2021-03-16 2024-06-18 Marathon Petroleum Company Lp Systems and methods for backhaul transportation of liquefied gas and CO2 using liquefied gas carriers
US11807945B2 (en) 2021-08-26 2023-11-07 Marathon Petroleum Company Lp Assemblies and methods for monitoring cathodic protection of structures
US12043906B2 (en) 2021-08-26 2024-07-23 Marathon Petroleum Company Lp Assemblies and methods for monitoring cathodic protection of structures
US12043905B2 (en) 2021-08-26 2024-07-23 Marathon Petroleum Company Lp Electrode watering assemblies and methods for maintaining cathodic monitoring of structures
US12129559B2 (en) 2021-08-26 2024-10-29 Marathon Petroleum Company Lp Test station assemblies for monitoring cathodic protection of structures and related methods
US11965317B2 (en) 2022-05-04 2024-04-23 Marathon Petroleum Company Lp Systems, methods, and controllers to enhance heavy equipment warning
US11808013B1 (en) 2022-05-04 2023-11-07 Marathon Petroleum Company Lp Systems, methods, and controllers to enhance heavy equipment warning
US12012082B1 (en) 2022-12-30 2024-06-18 Marathon Petroleum Company Lp Systems and methods for a hydraulic vent interlock
US12043361B1 (en) 2023-02-18 2024-07-23 Marathon Petroleum Company Lp Exhaust handling systems for marine vessels and related methods
US12006014B1 (en) 2023-02-18 2024-06-11 Marathon Petroleum Company Lp Exhaust vent hoods for marine vessels and related methods
US12087002B1 (en) 2023-09-18 2024-09-10 Marathon Petroleum Company Lp Systems and methods to determine depth of soil coverage along a right-of-way

Similar Documents

Publication Publication Date Title
WO2023287276A1 (fr) Procédés et systèmes de traitement de données géographiques pour détecter un empiètement par des objets dans un couloir géographique
Tan et al. Automatic detection of sewer defects based on improved you only look once algorithm
Mirzaei et al. 3D point cloud data processing with machine learning for construction and infrastructure applications: A comprehensive review
Kamari et al. AI-based risk assessment for construction site disaster preparedness through deep learning-based digital twinning
Zhang et al. Review of artificial intelligence-based bridge damage detection
Kong et al. Computer vision and long short-term memory: Learning to predict unsafe behaviour in construction
Wang et al. Deep semantic segmentation for visual understanding on construction sites
Minetto et al. Measuring human and economic activity from satellite imagery to support city-scale decision-making during covid-19 pandemic
CN109858367B (zh) 工人通过支撑不安全行为的视觉自动化检测方法及系统
Guo et al. Evaluation-oriented façade defects detection using rule-based deep learning method
Wang et al. Development and improvement of deep learning based automated defect detection for sewer pipe inspection using faster R-CNN
US11709870B2 (en) Comprehensive utility line database and user interface for excavation sites
Arman et al. Detection and classification of road damage using R-CNN and faster R-CNN: a deep learning approach
Sun et al. Roads and Intersections Extraction from High‐Resolution Remote Sensing Imagery Based on Tensor Voting under Big Data Environment
CN117372875A (zh) 一种航空遥感目标识别方法
Nogueira et al. A tool for bridge detection in major infrastructure works using satellite images
Cao et al. Posthurricane damage assessment using satellite imagery and geolocation features
Katsamenis et al. Real time road defect monitoring from UAV visual data sources
Ejaz et al. Computer vision in drone imagery for infrastructure management
Han et al. Ceiling damage detection and safety assessment in large public buildings using semantic segmentation
Zhao et al. High-resolution infrastructure defect detection dataset sourced by unmanned systems and validated with deep learning
Roberts et al. Optimization and sensitivity analysis of existing deep learning models for pavement surface monitoring using low-quality images
Samadzadegan et al. Automatic Road Crack Recognition Based on Deep Learning Networks from UAV Imagery
Wang et al. Instance segmentation of soft‐story buildings from street‐view images with semiautomatic annotation
Watanabe Electric pole detection using deep network based object detector

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22842536

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 14.05.2024)