EP3680648B1 - Port machinery inspection device and inspection method - Google Patents

Port machinery inspection device and inspection method Download PDF

Info

Publication number
EP3680648B1
EP3680648B1 EP17924086.6A EP17924086A EP3680648B1 EP 3680648 B1 EP3680648 B1 EP 3680648B1 EP 17924086 A EP17924086 A EP 17924086A EP 3680648 B1 EP3680648 B1 EP 3680648B1
Authority
EP
European Patent Office
Prior art keywords
drone
data
port machinery
module
structural defect
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP17924086.6A
Other languages
German (de)
French (fr)
Other versions
EP3680648A4 (en
EP3680648A1 (en
Inventor
Wenjun Li
Yunfu YAN
Lei Luo
Zijian Zhao
Jun Yu
Wei Dai
Qing Yi
Yao LU
Jianguo Shan
Dinggen XIONG
Maohua FU
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Zhenghua Heavy Industries Co Ltd
Original Assignee
Shanghai Zhenghua Heavy Industries Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Zhenghua Heavy Industries Co Ltd filed Critical Shanghai Zhenghua Heavy Industries Co Ltd
Publication of EP3680648A1 publication Critical patent/EP3680648A1/en
Publication of EP3680648A4 publication Critical patent/EP3680648A4/en
Application granted granted Critical
Publication of EP3680648B1 publication Critical patent/EP3680648B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/0094—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
    • G01C11/02—Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/95—Investigating the presence of flaws or contamination characterised by the material or shape of the object to be examined
    • G01N21/9515—Objects of complex shape, e.g. examined with use of a surface follower device
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U10/00—Type of UAV
    • B64U10/10—Rotorcrafts
    • B64U10/13—Flying platforms
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U2101/00—UAVs specially adapted for particular uses or applications
    • B64U2101/30—UAVs specially adapted for particular uses or applications for imaging, photography or videography
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U2201/00—UAVs characterised by their flight controls
    • B64U2201/10—UAVs characterised by their flight controls autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U2201/00—UAVs characterised by their flight controls
    • B64U2201/20—Remote controls
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/95—Investigating the presence of flaws or contamination characterised by the material or shape of the object to be examined
    • G01N21/9515—Objects of complex shape, e.g. examined with use of a surface follower device
    • G01N2021/9518—Objects of complex shape, e.g. examined with use of a surface follower device using a surface follower, e.g. robot

Definitions

  • the present invention relates to the field of port machinery, and more particularly, to inspection and evaluation technologies for port machinery.
  • Port machineries are widely used in the field of logistics. Port machineries mainly include shore container cranes, rail cranes, tire cranes, portal cranes and ship unloaders. Due to the heavy transportation tasks and long service life of port machinery, the health inspection of port machinery structure is very important for the normal operation of the port. The key of safe work for port machinery is the steel mechanism. The working position of port machinery is close to the shore, which is a corrosive environment. Therefore, it is necessary to perform regular inspections for port machinery to check its health.
  • the current inspection of port machinery is mainly realized as manual inspections.
  • the inspectors climb the port machinery to inspect the structure of the port machinery, take pictures of the structural defect areas, and draft an evaluation report according to the inspection results and pictures.
  • the size of existing port machinery reaches 150m*30m* 100m. It is difficult for inspectors to climb and the risk is very high.
  • some areas can not be reached, or can not be photographed because of the perspective of the inspectors. Therefore some blind spots for inspection are formed.
  • the existing manual inspection methods are inefficient, have long detection cycles, have poor detection results, have many blind spots, and have high safety risks.
  • Arial drones have been used for taking precise images of engineering structures such as described in EP 3193496 .
  • the present invention discloses an inspection technology for port machinery based on drones.
  • an inspection apparatus for port machinery comprises: a group of drones, a group of ground locators, a ground monitoring and processing device and a data management and evaluation device.
  • the group of drones includes one or more drones, the drone carries out inspections of the port machinery and obtains images of the port machinery.
  • the group of ground locators includes one or more ground locators, the ground locator locates the three-dimensional position of the drone.
  • the ground monitoring and processing device communicates with the drone, controls the flight of the drone, receives the images acquired by the drone and monitors the status of the drone.
  • the data management and evaluation device communicates with the ground monitoring and processing device, the data management and evaluation device stores the basic data of the port machinery, the data management and evaluation device receives the image acquired by the drone from the ground monitoring and processing device, performs structural defect identification and structural defect analysis based on the image, and generates evaluation results based on the structural defect identification and structural defect analysis.
  • the ground monitoring and processing device comprises: a main control machine and a hand-held remote controller.
  • the main control machine plans an inspection route for the drone, controls the drone to fly along the inspection route and performs aerial photography so as to obtain the image of the port machinery, the main control machine also monitors the status of the drone.
  • the hand-held remote controller, the hand-held remote controller receives operation instructions, and controls the drone to fly and perform aerial photography so as to obtain images of port machinery according to the operation instructions. Wherein, when the hand-held remote controller is not in operation, the drone is controlled by the main control machine and operates automatically, when the hand-held remote controller is in operation, the drone quits from automatic operation and is controlled by the hand-held remote controller.
  • the main control machine comprises: an interface interaction module, a front-end data management module, a route planning module, an aerial photography control module, a monitoring module and a communication module.
  • the interface interaction module generates an interactive interface for receiving instructions and displaying information.
  • the front-end data management module obtains the basic data of the port machinery from the data management and evaluation device or the interface interaction module.
  • the route planning module plans an automatic inspection route based on the basic data of port machinery obtained by the front-end data management module, or plans a designate inspection route based on the instructions obtained from the interactive interface.
  • the aerial photography control module controls the drone to fly and perform aerial photography according to the automatic inspection route or designated inspection route planned by the route planning module to obtain the image of port machinery.
  • the monitoring module monitors the 3D position and status of the drone.
  • the communication module communicates with the data management and evaluation device and the hand-held remote controller.
  • the hand-held remote controller comprises: a hand-held manipulator, a data transmission module and an image transmission module.
  • the hand-held manipulator is held by an operator, receives operation instructions and controls the drone according to the operation instructions.
  • the data transmission module performs data communication with the drone and the main control machine, and monitors the status of the drone.
  • the image transmission module performs image communication with the drone and the main control machine, and obtains the images and videos of the port machinery obtained by the aerial photography of the drone.
  • the data management and evaluation device comprises: a back-end data management module, a structural defect image recognition module, a structural defect image analysis module and an evaluation module.
  • the back-end data management module processes basic data of port machinery, route data, status data of drones, and image data of port machinery.
  • the structural defect image recognition module is configured to identify structural defect areas in images of port machinery, mark and classify structural defect areas.
  • the structural defect image analysis module is configured to perform quantitative calculation on the structural defect areas marked and classified by the structural defect image identification module, and correlate the results of the quantitative calculation with the structural defect areas.
  • the evaluation module is configured to generate an evaluation result based on the result of the quantitative calculation and the correlation between the result of the quantitative calculation and the structural defect area.
  • the classification of structural defect areas includes: corrosion, cracks, missing bolts, paint stripping, or other structural defects.
  • the quantitative calculation of the structural defect areas includes: the size of the corrosion area, the size of the paint stripped area, the length and width of the crack, the number and location of the missing bolts.
  • the basic data of the port machinery includes size data and position data of the port machinery.
  • the route data includes planned route data and actual route data, wherein the planned route data includes automatic inspection route data and designated inspection route data.
  • the status data of drones includes: time, detection point number, plane horizontal coordinate, plane longitudinal coordinate, altitude coordinate, pitch angle, tilt angle, yaw angle, pitch angle of the photography gimbal, tilt angle of the photography gimbal, and yaw angle of the of the photography gimbal.
  • the processing of the basic data, the route data, the status data of drones and image data of port machinery includes: saving, deleting, modifying, encrypting, decrypting, classifying, filtering, statistics, importing, and exporting.
  • the drone includes: a fuselage, a flight module, a flight control module, a data transmission module, an image transmission module, an obstacle avoidance module, a positioning module, an aerial camera and a tracking camera.
  • an inspection method for port machinery utilizes the inspection apparatus for port machinery as mentioned above.
  • the inspection method comprises:
  • the inspection method further comprises: a step of route adjusting, generating a designated inspection route according to the instructions obtained from the interactive interface, wherein the drone flies according to the designated inspection route to perform aerial photography of the port machinery.
  • the inspection method further comprises: a step of manual control, operating the hand-held remote controller, wherein the drone quits automatic operation and is controlled to fly and perform aerial photography of the port machinery under the control of the hand-held remote controller.
  • the inspection apparatus and inspection method for port machinery can automatically plan the route of the drone based on the basic data of the port machinery.
  • the drone automatically flies and performs aerial photography according to the planned route to carry out the inspection and collect images and videos of the port machinery.
  • the image data and video data are transmitted to a data management and evaluation device, which performs structural defect identification and structural defect analysis on the images.
  • the data management and evaluation device generates evaluation results based on the structural defect identification and structural defect analysis. When necessary, it is able to manually modify the route of the drone or switch the drone to manual operation mode.
  • the inspection device and inspection method have high degree of automation, high detection efficiency, wide application range, and high safety factor.
  • FIG. 1 is a schematic structural diagram of an inspection apparatus for port machinery according to an embodiment of the present invention.
  • the inspection apparatus for port machinery includes: a group of drones 101, a group of ground locators 102, a ground monitoring and processing device 103 and a data management and evaluation device 104.
  • the group of drones 101 includes one or more drones, the drone carries out inspections of a port machinery and obtains images of the port machinery.
  • the group of ground locators 102 includes one or more ground locators, the ground locator locates the three-dimensional position of the drone.
  • the ground monitoring and processing device 103 communicates with the drone, controls the flight of the drone, receives the images acquired by the drone and monitors the status of the drone.
  • the data management and evaluation device 104 communicates with the ground monitoring and processing device, the data management and evaluation device stores the basic data of the port machinery, the data management and evaluation device receives the image acquired by the drone from the ground monitoring and processing device, performs structural defect identification and structural defect analysis based on the image, and generates evaluation results based on the structural defect identification and structural defect analysis.
  • FIG. 2 illustrates a structural block diagram of an inspection apparatus for port machinery according to an embodiment of the present invention.
  • FIG. 2 illustrates the structure of the inspection apparatus for port machinery in detail.
  • the group of drones 101 includes one or more drones, and in FIG. 2 , each drone in the group of drones 101 is marked as 101.1 ⁇ 101.n.
  • the drone used here is a drone with the function of automatic tracking flight and aerial photography.
  • the drone includes: a fuselage, a flight module, a flight control module, a data transmission module, an image transmission module, an obstacle avoidance module, a positioning module, an aerial camera and a tracking camera.
  • the drone with the function of automatic tracking flight and aerial photography is purchased from the market.
  • the group of ground locators 102 includes one or more ground locators, and in FIG. 2 , each ground locator in the group of ground locators 102 is marked as 102.1 ⁇ 102.m.
  • the ground locator locates the three-dimensional position of the drone. In an embodiment, the ground locator locates the three-dimensional position of the drone according to a wireless pulse signal or a satellite signal.
  • a wireless pulse signal or a satellite signal a wireless pulse signal or satellite signal.
  • the method of locating the three-dimensional position of the drone by the ground locator is not limited to the wireless pulse signal or satellite signal described above.
  • the ground monitoring and processing device 103 comprises: a main control machine 131 and a hand-held remote controller 132.
  • the main control machine 131 plans an inspection route for the drone, controls the drone to fly along the inspection route and performs aerial photography so as to obtain the image of the port machinery, the main control machine also monitors the status of the drone.
  • the hand-held remote controller 132 receives operation instructions, and controls the drone to fly and perform aerial photography so as to obtain images of port machinery according to the operation instructions.
  • the hand-held remote controller 132 is not in operation, the drone is controlled by the main control machine 131 and operates automatically.
  • the hand-held remote controller 132 is in operation, the drone quits from automatic operation and is controlled by the hand-held remote controller 132.
  • the main control machine 131 includes: an interface interaction module 301, a front-end data management module 302, a route planning module 303, an aerial photography control module 304, a monitoring module 305 and a communication module 306.
  • the interface interaction module 301 generates an interactive interface for receiving instructions and displaying information.
  • the front-end data management module 302 obtains the basic data of the port machinery from the data management and evaluation device 104 or the interface interaction module 301.
  • the basic data of the port machinery includes size data and position data of the port machinery.
  • the route planning module 303 plans an automatic inspection route based on the basic data of port machinery obtained by the front-end data management module 302, that is, plans the automatic inspection route based on the size and position of port machinery. Or, the route planning module 303 plans a designate inspection route based on the instructions obtained from the interactive interface. In some cases, it is necessary to modify the automatic inspection route. At this time, an instruction may be input through the interactive interface generated by the interface interaction module 301 to manually modify the inspection route to generate a designated inspection route. After the designated inspection route is generated, the drone flies and performs aerial photography according to the newly modified designated inspection route.
  • the aerial photography control module 304 controls the drone to fly and perform aerial photography according to the automatic inspection route or designated inspection route planned by the route planning module to obtain the image of port machinery.
  • the monitoring module 305 monitors the 3D position and status of the drone.
  • the communication module 306 communicates with the data management and evaluation device 104 and the hand-held remote controller 132.
  • the front-end data management module 302 processes the basic data, the route data, the status data of drones and image data of port machinery, and the processing of date includes: saving, deleting, modifying, encrypting, decrypting, classifying, filtering, statistics, importing, and exporting.
  • the hand-held remote controller 132 comprises: a hand-held manipulator 307, a data transmission module 308 and an image transmission module 309.
  • the hand-held manipulator 307 is held by an operator, receives operation instructions and controls the drone according to the operation instructions.
  • the hand-held manipulator 307 includes a joystick, buttons, and a wheel.
  • the data transmission module 308 performs data communication with the drone and the main control machine, and monitors the status of the drone.
  • the data transmission module 308 is used for transmission of the basic data of port machinery, the route data and the status data of drones.
  • the image transmission module 309 performs image communication with the drone and the main control machine, and obtains the images and videos of the port machinery obtained by the aerial photography of the drone.
  • the image transmission module 309 is used for transmission of the image date or video data of port machinery.
  • the data management and evaluation device 104 comprises: a back-end data management module 141, a structural defect image recognition module 142, a structural defect image analysis module 143 and an evaluation module 144.
  • the back-end data management module 141 processes basic data of port machinery, route data, status data of drones, and image data of port machinery.
  • the basic data of the port machinery includes size data and position data of the port machinery.
  • the route data includes planned route data and actual route data.
  • the status data of drones includes: time, detection point number, plane horizontal coordinate (or longitude coordinates), plane longitudinal coordinate (or latitude coordinates), altitude coordinate, pitch angle, tilt angle, yaw angle, pitch angle of the photography gimbal, tilt angle of the photography gimbal, and yaw angle of the of the photography gimbal.
  • the processing of the basic data, the route data, the status data of drones and image data of port machinery includes: saving, deleting, modifying, encrypting, decrypting, classifying, filtering, statistics, importing, and exporting, etc,.
  • the structural defect image recognition module 142 identifies structural defect areas in images of port machinery, marks and classifies structural defect areas.
  • the classification of structural defect areas includes: corrosion, cracks, missing bolts, paint stripping, or other structural defects.
  • the structural defect image analysis module 143 performs quantitative calculation on the structural defect areas marked and classified by the image structural defect identification module, and correlates the results of the quantitative calculation with the structural defect areas.
  • the quantitative calculation of the structural defect areas includes: the size of the corrosion area, the size of the paint stripped area, the length and width of the crack, the number and location of the missing bolts.
  • the evaluation module generates an evaluation result based on the result of the quantitative calculation and the correlation between the result of the quantitative calculation and the structural defect area.
  • the present invention further discloses an inspection method for port machinery utilizing the inspection apparatus for port machinery mentioned above, the inspection method comprises:
  • the inspection method for port machinery when it is necessary to adjust the automatic inspection route, further comprises: a step of route adjusting, generating a designated inspection route according to the instructions obtained from the interactive interface, wherein the drone flies according to the designated inspection route to perform aerial photography of the port machinery.
  • the inspection method for port machinery further comprises: a step of manual control, operating the hand-held remote controller, wherein the drone quits automatic operation and is controlled to fly and perform aerial photography of the port machinery under the control of the hand-held remote controller.
  • FIG. 4a and FIG. 4b disclose an exemplary flow of an inspection method for port machinery according to an embodiment of the present invention.
  • the exemplary flow includes the following steps:
  • the inspection apparatus and inspection method for port machinery can automatically plan the route of the drone based on the basic data of the port machinery.
  • the drone automatically flies and performs aerial photography according to the planned route to carry out the inspection and collect images and videos of the port machinery.
  • the image data and video data are transmitted to a data management and evaluation device, which performs structural defect identification and structural defect analysis on the images.
  • the data management and evaluation device generates evaluation results based on the structural defect identification and structural defect analysis. When necessary, it is able to manually modify the route of the drone or switch the drone to manual operation mode.
  • the inspection device and inspection method have high degree of automation, high detection efficiency, wide application range, and high safety factor.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Remote Sensing (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Chemical & Material Sciences (AREA)
  • Immunology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Multimedia (AREA)
  • Analytical Chemistry (AREA)
  • Biochemistry (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Pathology (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Automation & Control Theory (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)
  • Image Analysis (AREA)

Description

    BACKGROUND OF THE INVENTION 1. Field of the Invention
  • The present invention relates to the field of port machinery, and more particularly, to inspection and evaluation technologies for port machinery.
    • 2. The Related Art
  • Port machineries are widely used in the field of logistics. Port machineries mainly include shore container cranes, rail cranes, tire cranes, portal cranes and ship unloaders. Due to the heavy transportation tasks and long service life of port machinery, the health inspection of port machinery structure is very important for the normal operation of the port. The key of safe work for port machinery is the steel mechanism. The working position of port machinery is close to the shore, which is a corrosive environment. Therefore, it is necessary to perform regular inspections for port machinery to check its health.
  • The current inspection of port machinery is mainly realized as manual inspections. The inspectors climb the port machinery to inspect the structure of the port machinery, take pictures of the structural defect areas, and draft an evaluation report according to the inspection results and pictures. With the enlargement of port machinery, the size of existing port machinery reaches 150m*30m* 100m. It is difficult for inspectors to climb and the risk is very high. In addition, due to the special structure of the port machinery, some areas can not be reached, or can not be photographed because of the perspective of the inspectors. Therefore some blind spots for inspection are formed. In general, the existing manual inspection methods are inefficient, have long detection cycles, have poor detection results, have many blind spots, and have high safety risks. Arial drones have been used for taking precise images of engineering structures such as described in EP 3193496 .
    • SUMMARY
  • The present invention discloses an inspection technology for port machinery based on drones.
  • According to an embodiment of the present invention, an inspection apparatus for port machinery is disclosed. The apparatus comprises: a group of drones, a group of ground locators, a ground monitoring and processing device and a data management and evaluation device. The group of drones includes one or more drones, the drone carries out inspections of the port machinery and obtains images of the port machinery. The group of ground locators includes one or more ground locators, the ground locator locates the three-dimensional position of the drone. The ground monitoring and processing device communicates with the drone, controls the flight of the drone, receives the images acquired by the drone and monitors the status of the drone. The data management and evaluation device communicates with the ground monitoring and processing device, the data management and evaluation device stores the basic data of the port machinery, the data management and evaluation device receives the image acquired by the drone from the ground monitoring and processing device, performs structural defect identification and structural defect analysis based on the image, and generates evaluation results based on the structural defect identification and structural defect analysis.
  • In an embodiment, the ground monitoring and processing device comprises: a main control machine and a hand-held remote controller. The main control machine plans an inspection route for the drone, controls the drone to fly along the inspection route and performs aerial photography so as to obtain the image of the port machinery, the main control machine also monitors the status of the drone. The hand-held remote controller, the hand-held remote controller receives operation instructions, and controls the drone to fly and perform aerial photography so as to obtain images of port machinery according to the operation instructions. Wherein, when the hand-held remote controller is not in operation, the drone is controlled by the main control machine and operates automatically, when the hand-held remote controller is in operation, the drone quits from automatic operation and is controlled by the hand-held remote controller.
  • In an embodiment, the main control machine comprises: an interface interaction module, a front-end data management module, a route planning module, an aerial photography control module, a monitoring module and a communication module. The interface interaction module generates an interactive interface for receiving instructions and displaying information. The front-end data management module obtains the basic data of the port machinery from the data management and evaluation device or the interface interaction module. The route planning module plans an automatic inspection route based on the basic data of port machinery obtained by the front-end data management module, or plans a designate inspection route based on the instructions obtained from the interactive interface. The aerial photography control module controls the drone to fly and perform aerial photography according to the automatic inspection route or designated inspection route planned by the route planning module to obtain the image of port machinery. The monitoring module monitors the 3D position and status of the drone. The communication module communicates with the data management and evaluation device and the hand-held remote controller.
  • In an embodiment, the hand-held remote controller comprises: a hand-held manipulator, a data transmission module and an image transmission module. The hand-held manipulator is held by an operator, receives operation instructions and controls the drone according to the operation instructions. The data transmission module performs data communication with the drone and the main control machine, and monitors the status of the drone. The image transmission module performs image communication with the drone and the main control machine, and obtains the images and videos of the port machinery obtained by the aerial photography of the drone.
  • In an embodiment, the data management and evaluation device comprises: a back-end data management module, a structural defect image recognition module, a structural defect image analysis module and an evaluation module. The back-end data management module processes basic data of port machinery, route data, status data of drones, and image data of port machinery. The structural defect image recognition module is configured to identify structural defect areas in images of port machinery, mark and classify structural defect areas. The structural defect image analysis module is configured to perform quantitative calculation on the structural defect areas marked and classified by the structural defect image identification module, and correlate the results of the quantitative calculation with the structural defect areas. The evaluation module is configured to generate an evaluation result based on the result of the quantitative calculation and the correlation between the result of the quantitative calculation and the structural defect area.
  • In an embodiment, the classification of structural defect areas includes: corrosion, cracks, missing bolts, paint stripping, or other structural defects. The quantitative calculation of the structural defect areas includes: the size of the corrosion area, the size of the paint stripped area, the length and width of the crack, the number and location of the missing bolts.
  • In an embodiment the basic data of the port machinery includes size data and position data of the port machinery. The route data includes planned route data and actual route data, wherein the planned route data includes automatic inspection route data and designated inspection route data. The status data of drones includes: time, detection point number, plane horizontal coordinate, plane longitudinal coordinate, altitude coordinate, pitch angle, tilt angle, yaw angle, pitch angle of the photography gimbal, tilt angle of the photography gimbal, and yaw angle of the of the photography gimbal. The processing of the basic data, the route data, the status data of drones and image data of port machinery includes: saving, deleting, modifying, encrypting, decrypting, classifying, filtering, statistics, importing, and exporting.
  • In an embodiment, the drone includes: a fuselage, a flight module, a flight control module, a data transmission module, an image transmission module, an obstacle avoidance module, a positioning module, an aerial camera and a tracking camera.
  • According to an embodiment of the present invention, an inspection method for port machinery is disclosed. The inspection method utilizes the inspection apparatus for port machinery as mentioned above. The inspection method comprises:
    • a step of route planning, planning an automatic inspection route based on the basic data of port machinery;
    • a step of drone aerial photography, controlling the drone to fly according to automatic inspection route and taking aerial photos of the port machinery, monitoring the status of the drone, and receiving images acquired by the drone;
    • a step of data processing, performing structural defect recognition and structural defect analysis on the images acquired by the drone, and generating evaluation results based on the structural defect recognition and structural defect analysis.
  • In an embodiment, the inspection method further comprises:
    a step of route adjusting, generating a designated inspection route according to the instructions obtained from the interactive interface, wherein the drone flies according to the designated inspection route to perform aerial photography of the port machinery.
  • In an embodiment, the inspection method further comprises:
    a step of manual control, operating the hand-held remote controller, wherein the drone quits automatic operation and is controlled to fly and perform aerial photography of the port machinery under the control of the hand-held remote controller.
  • The inspection apparatus and inspection method for port machinery according to the present invention can automatically plan the route of the drone based on the basic data of the port machinery. The drone automatically flies and performs aerial photography according to the planned route to carry out the inspection and collect images and videos of the port machinery. The image data and video data are transmitted to a data management and evaluation device, which performs structural defect identification and structural defect analysis on the images. The data management and evaluation device generates evaluation results based on the structural defect identification and structural defect analysis. When necessary, it is able to manually modify the route of the drone or switch the drone to manual operation mode. The inspection device and inspection method have high degree of automation, high detection efficiency, wide application range, and high safety factor.
    • BRIEF DESCRIPTION OF THE DRAWINGS
  • The above and other features, natures, and advantages of the invention will be apparent by the following description of the embodiments incorporating the drawings, wherein,
    • FIG. 1 is a schematic structural diagram of an inspection apparatus for port machinery according to an embodiment of the present invention.
    • FIG. 2 illustrates a structural block diagram of an inspection apparatus for port machinery according to an embodiment of the present invention.
    • FIG. 3 illustrates a structural block diagram of a ground monitoring and processing device in the inspection apparatus for port machinery according to an embodiment of the present invention.
    • FIG. 4a and FIG. 4b disclose an exemplary flow of an inspection method for port machinery according to an embodiment of the present invention.
    DETAILED DESCRIPTION OF EMBODIMENTS
  • Referring to FIG. 1, FIG. 1 is a schematic structural diagram of an inspection apparatus for port machinery according to an embodiment of the present invention. The inspection apparatus for port machinery includes: a group of drones 101, a group of ground locators 102, a ground monitoring and processing device 103 and a data management and evaluation device 104.
  • The group of drones 101 includes one or more drones, the drone carries out inspections of a port machinery and obtains images of the port machinery. The group of ground locators 102 includes one or more ground locators, the ground locator locates the three-dimensional position of the drone. The ground monitoring and processing device 103 communicates with the drone, controls the flight of the drone, receives the images acquired by the drone and monitors the status of the drone. The data management and evaluation device 104 communicates with the ground monitoring and processing device, the data management and evaluation device stores the basic data of the port machinery, the data management and evaluation device receives the image acquired by the drone from the ground monitoring and processing device, performs structural defect identification and structural defect analysis based on the image, and generates evaluation results based on the structural defect identification and structural defect analysis.
  • Referring to FIG. 2, FIG. 2 illustrates a structural block diagram of an inspection apparatus for port machinery according to an embodiment of the present invention. FIG. 2 illustrates the structure of the inspection apparatus for port machinery in detail. The group of drones 101 includes one or more drones, and in FIG. 2, each drone in the group of drones 101 is marked as 101.1~101.n. The drone used here is a drone with the function of automatic tracking flight and aerial photography. In an embodiment, the drone includes: a fuselage, a flight module, a flight control module, a data transmission module, an image transmission module, an obstacle avoidance module, a positioning module, an aerial camera and a tracking camera. The drone with the function of automatic tracking flight and aerial photography is purchased from the market. The basic structure and functions of the drone will not be described here in detail. The group of ground locators 102 includes one or more ground locators, and in FIG. 2, each ground locator in the group of ground locators 102 is marked as 102.1~102.m. The ground locator locates the three-dimensional position of the drone. In an embodiment, the ground locator locates the three-dimensional position of the drone according to a wireless pulse signal or a satellite signal. Of course, those skilled in the art shall understand that the method of locating the three-dimensional position of the drone by the ground locator is not limited to the wireless pulse signal or satellite signal described above.
  • The ground monitoring and processing device 103 comprises: a main control machine 131 and a hand-held remote controller 132. The main control machine 131 plans an inspection route for the drone, controls the drone to fly along the inspection route and performs aerial photography so as to obtain the image of the port machinery, the main control machine also monitors the status of the drone. The hand-held remote controller 132 receives operation instructions, and controls the drone to fly and perform aerial photography so as to obtain images of port machinery according to the operation instructions. When the hand-held remote controller 132 is not in operation, the drone is controlled by the main control machine 131 and operates automatically. When the hand-held remote controller 132 is in operation, the drone quits from automatic operation and is controlled by the hand-held remote controller 132. FIG. 3 illustrates a structural block diagram of a ground monitoring and processing device in the inspection apparatus for port machinery according to an embodiment of the present invention. As shown in FIG. 3, the main control machine 131 includes: an interface interaction module 301, a front-end data management module 302, a route planning module 303, an aerial photography control module 304, a monitoring module 305 and a communication module 306. The interface interaction module 301 generates an interactive interface for receiving instructions and displaying information. The front-end data management module 302 obtains the basic data of the port machinery from the data management and evaluation device 104 or the interface interaction module 301. In an embodiment, the basic data of the port machinery includes size data and position data of the port machinery. The route planning module 303 plans an automatic inspection route based on the basic data of port machinery obtained by the front-end data management module 302, that is, plans the automatic inspection route based on the size and position of port machinery. Or, the route planning module 303 plans a designate inspection route based on the instructions obtained from the interactive interface. In some cases, it is necessary to modify the automatic inspection route. At this time, an instruction may be input through the interactive interface generated by the interface interaction module 301 to manually modify the inspection route to generate a designated inspection route. After the designated inspection route is generated, the drone flies and performs aerial photography according to the newly modified designated inspection route. The aerial photography control module 304 controls the drone to fly and perform aerial photography according to the automatic inspection route or designated inspection route planned by the route planning module to obtain the image of port machinery. The monitoring module 305 monitors the 3D position and status of the drone. The communication module 306 communicates with the data management and evaluation device 104 and the hand-held remote controller 132. In an embodiment, the front-end data management module 302 processes the basic data, the route data, the status data of drones and image data of port machinery, and the processing of date includes: saving, deleting, modifying, encrypting, decrypting, classifying, filtering, statistics, importing, and exporting. Continue to refer to FIG. 3, the hand-held remote controller 132 comprises: a hand-held manipulator 307, a data transmission module 308 and an image transmission module 309. The hand-held manipulator 307 is held by an operator, receives operation instructions and controls the drone according to the operation instructions. In an embodiment, the hand-held manipulator 307 includes a joystick, buttons, and a wheel. The data transmission module 308 performs data communication with the drone and the main control machine, and monitors the status of the drone. The data transmission module 308 is used for transmission of the basic data of port machinery, the route data and the status data of drones. The image transmission module 309 performs image communication with the drone and the main control machine, and obtains the images and videos of the port machinery obtained by the aerial photography of the drone. The image transmission module 309 is used for transmission of the image date or video data of port machinery.
  • Back to FIG. 2, the data management and evaluation device 104 comprises: a back-end data management module 141, a structural defect image recognition module 142, a structural defect image analysis module 143 and an evaluation module 144. The back-end data management module 141 processes basic data of port machinery, route data, status data of drones, and image data of port machinery. In an embodiment, the basic data of the port machinery includes size data and position data of the port machinery. The route data includes planned route data and actual route data. The status data of drones includes: time, detection point number, plane horizontal coordinate (or longitude coordinates), plane longitudinal coordinate (or latitude coordinates), altitude coordinate, pitch angle, tilt angle, yaw angle, pitch angle of the photography gimbal, tilt angle of the photography gimbal, and yaw angle of the of the photography gimbal. The processing of the basic data, the route data, the status data of drones and image data of port machinery includes: saving, deleting, modifying, encrypting, decrypting, classifying, filtering, statistics, importing, and exporting, etc,. The structural defect image recognition module 142 identifies structural defect areas in images of port machinery, marks and classifies structural defect areas. In an embodiment, the classification of structural defect areas includes: corrosion, cracks, missing bolts, paint stripping, or other structural defects. The structural defect image analysis module 143 performs quantitative calculation on the structural defect areas marked and classified by the image structural defect identification module, and correlates the results of the quantitative calculation with the structural defect areas. In an embodiment, the quantitative calculation of the structural defect areas includes: the size of the corrosion area, the size of the paint stripped area, the length and width of the crack, the number and location of the missing bolts. The evaluation module generates an evaluation result based on the result of the quantitative calculation and the correlation between the result of the quantitative calculation and the structural defect area.
  • The present invention further discloses an inspection method for port machinery utilizing the inspection apparatus for port machinery mentioned above, the inspection method comprises:
    • a step of route planning, planning an automatic inspection route based on the basic data of port machinery;
    • a step of drone aerial photography, controlling the drone to fly according to automatic inspection route and taking aerial photos of the port machinery, monitoring the status of the drone, and receiving images acquired by the drone;
    • a step of data processing, performing structural defect recognition and structural defect analysis on the images acquired by the drone, and generates evaluation results based on the structural defect recognition and structural defect analysis.
  • In an embodiment, when it is necessary to adjust the automatic inspection route, the inspection method for port machinery further comprises: a step of route adjusting, generating a designated inspection route according to the instructions obtained from the interactive interface, wherein the drone flies according to the designated inspection route to perform aerial photography of the port machinery.
  • In an embodiment, when it is necessary to manually operate the drone, the inspection method for port machinery further comprises: a step of manual control, operating the hand-held remote controller, wherein the drone quits automatic operation and is controlled to fly and perform aerial photography of the port machinery under the control of the hand-held remote controller.
  • FIG. 4a and FIG. 4b disclose an exemplary flow of an inspection method for port machinery according to an embodiment of the present invention. The exemplary flow includes the following steps:
    • Step S1: Initiate the ground monitoring and processing device 103, and check to determine that the main controller 131 and the hand-held remote controller 132 are in a normal state.
    • Step S2: Input the size data of the port machinery to be inspected through the interface interaction module 301 of the main control machine 131, or download the size data of the port machinery to be inspected from the back-end data management module 141 through the communication module 306 of the main control machine 131.
    • Step S3: Put the drone 101 and the ground locator 102 in the initial positions respectively. The initial position of the drone 101 is a predefined relative position that refers to the position of the port machinery to be inspected. The ground locator 102 is arranged at an appropriate position on the ground according to the required positioning accuracy of the drone.
    • Step S4: Initiate the drone 101 and the ground locator 102, and check to confirm that both the drone and the ground locator are in a normal state.
    • Step S5: Record the initial position coordinates of the drone 101 in the main control machine 131, including its plane horizontal coordinate (or longitude coordinates), plane longitudinal coordinate (or latitude coordinates), altitude coordinate.
    • Step S6: Based on the size data of the port machinery that had been imported into the main control machine 131, the route planning module 303 executes the route planning program to automatically generate an automatic inspection route matching the port machinery to be inspected. When it is necessary, the operator may manually modify the automatic inspection route through the interface interaction module 301, modify it to be the designated inspection route, and save the designated inspection route as the latest planned route.
    • Step S7: Automatically control the drone 101 to perform aerial photography according to the latest planned route.
    • Step S8: The ground monitoring processing system 103 receives the mission status of the drone 101, and determines whether the aerial photography mission is completed. If it is determined that the aerial photography mission is not completed, proceed to step S9; if it is determined that the aerial photography mission is completed, proceed to step S15.
    • Step S9: The main controller 131 detects the control instruction of the hand-held remote controller 132 in real time, and determines whether any instruction is received. If an instruction is received, proceed to step S10; if no instruction is received, return to step S7.
    • Step S10: The ground monitoring processing system 103 sends an instruction to the drone 101 to control the drone 101 to suspend aerial photography and hover.
    • Step S11: The main control machine 131 displays three options through the interactive interface: route modification, manual control and misoperation, and wait for the operator to make a choice. If the option route modification is chosen, proceed to step S12; if the option manual control is chosen, proceed to step S14; if the option misoperation is chosen, return to step S7.
    • Step S12: The operator manually modifies the route parameters in the main control machine 131 according to the actual requirements, designates the inspection route and saves the designated inspection route as the latest planned route.
    • Step S13: The main control machine 131 inquires the operator through the interactive interface whether to continue the automatic control. If the operator selects No, proceed to step S14, and if the operator selects Yes, return to step S7.
    • Step S14: The operator manually operates the hand-held remote controller 132 to control the drone 101 to perform aerial photography according to the manual control instruction.
    • Step S15: In the automatic control mode, the ground monitoring and processing system 103 automatically controls the drone 101 to complete the flight mission and return to the ground. In the manual control mode, the operator operates the hand-held remote controller 132 to manually control the drone 101 and complete the flight mission and return to the ground.
    • Step S16: The operator imports the image data and video data collected by the drone 101 into the main control machine 131.
    • Step S17: The operator manually checks and filters the images that have been imported into the main control machine 131 according to actual requirements.
    • Step S18: The operator transmits the route data, all or filtered images and video data in the main control machine 131 to the back-end data management module 141 in the data management and evaluation device 104.
    • Step S19: The structural defect image recognition module 142 in the data management and evaluation device 104 automatically recognizes the structural surface structural defects in the image, classifies the surface structural defects into corrosion, cracks, missing bolts, paint stripping, or other structural defects, and marks the structurally defective area and type in the image that recognized with structural defect. The processed image is saved to the back-end data management module 141.
    • Step S20: The structural defect image analysis module 143 performs quantitative calculation on images that have been recognized and classified as corrosion, cracks, missing bolts or paint stripping. The result of the quantitative calculation includes: the size of the corrosion area, the size of the paint stripped area, the length and width of the crack, the number and location of the missing bolts. The results are correlated with the structural defect images.
    • Step S21: The evaluation module 144 automatically writes the structural defect images and analysis results as an evaluation result into a pre-stored evaluation report template file to form a basic evaluation report.
    • Step S22: A professional evaluator checks the basic evaluation report in the evaluation module 144, writes recommendations such as maintenance comments into the basic evaluation report, and saves it as a complete evaluation report.
  • The inspection apparatus and inspection method for port machinery according to the present invention can automatically plan the route of the drone based on the basic data of the port machinery. The drone automatically flies and performs aerial photography according to the planned route to carry out the inspection and collect images and videos of the port machinery. The image data and video data are transmitted to a data management and evaluation device, which performs structural defect identification and structural defect analysis on the images. The data management and evaluation device generates evaluation results based on the structural defect identification and structural defect analysis. When necessary, it is able to manually modify the route of the drone or switch the drone to manual operation mode. The inspection device and inspection method have high degree of automation, high detection efficiency, wide application range, and high safety factor.
  • The above embodiments are provided to those skilled in the art to realize or use the invention, under the condition that various modifications or changes being made by those skilled in the art without departing from the scope of the invention, which is defined by the appended claims.

Claims (9)

  1. An inspection apparatus for port machinery comprising:
    one or more drones (101) suitable for carrying out inspections of a port machinery and obtaining images of the port machinery;
    one or more ground locators (102) suitable for locating the three-dimensional position of each drone;
    a ground monitoring and processing device (103), suitable for communicating with each drone, controlling the flight of each drone, receiving the images acquired by each drone and monitoring the status of each drone;
    a data management and evaluation device (104) suitable for communicating with the ground monitoring and processing device, storing basic data of the port machinery, receiving the image acquired by each drone from the ground monitoring and processing device, performing structural defect identification and structural defect analysis based on the image, and generating evaluation results based on the structural defect identification and structural defect analysis;
    wherein the data management and evaluation device comprises:
    a back-end data management module (141) suitable for processing the basic data of port machinery, route data, status data of each drone, and image data of port machinery;
    characterised in that the data management and evaluation device further comprises a structural defect image recognition module (142) configured to identify structural defect areas in images of port machinery, mark and classify structural defect areas, wherein the classification of structural defect areas includes: corrosion, cracks, missing bolts, paint stripping, and other structural defects;
    a structural defect image analysis module (143) configured to perform quantitative calculation on the structural defect areas marked and classified by the image structural defect identification module, and correlate the results of the quantitative calculation with the structural defect areas, wherein the quantitative calculation of the structural defect areas includes: the size of the corrosion area, the size of the paint stripped area, the length and width of the crack, the number and location of the missing bolts;
    an evaluation module (144) configured to generate an evaluation result based on the result of the quantitative calculation and the correlation between the result of the quantitative calculation and the structural defect area.
  2. The inspection apparatus for port machinery according to claim 1, characterized in that the ground monitoring and processing device comprises:
    a main control machine (131) suitable for planning an inspection route for each drone, controlling each drone to fly along the inspection route and performing aerial photography so as to obtain the image of the port machinery, and monitoring the status of each drone;
    a hand-held remote controller (132) suitable for receiving operation instructions, and controlling each drone to fly and perform aerial photography so as to obtain images of port machinery according to the operation instructions;
    wherein, when the hand-held remote controller is not in operation, each drone is controlled by the main control machine and operates automatically, when the hand-held remote controller is in operation, each drone quits from automatic operation and is controlled by the hand-held remote controller.
  3. The inspection apparatus for port machinery according to claim 2, characterized in that the main control machine comprises:
    an interface interaction module (301), suitable for generating an interactive interface for receiving instructions and displaying information;
    a front-end data management module (302) suitable for obtaining the basic data of the port machinery from the data management and evaluation device or the interface interaction module;
    a route planning module (303) suitable for planning an automatic inspection route based on the basic data of port machinery obtained by the front-end data management module, or planning a designate inspection route based on the instructions obtained from the interactive interface;
    an aerial photography control module (304) suitable for controlling each drone to fly and perform aerial photography according to the automatic inspection route or designated inspection route planned by the route planning module to obtain the image of port machinery;
    a monitoring module (305) suitable for monitoring the 3D position and status of each drone;
    a communication module (306) suitable for communicating with the data management and evaluation device and the hand-held remote controller.
  4. The inspection apparatus for port machinery according to claim 2, characterized in that the hand-held remote controller comprises:
    a hand-held manipulator (307) suitable for being held by an operator, receiving operation instructions and controlling each drone according to the operation instructions;
    a data transmission module (308) suitable for performing data communication with each drone and the main control machine, and monitoring the status of each drone;
    an image transmission module (309) suitable for performing image communication with each drone and the main control machine, and obtaining the images and videos of the port machinery obtained by the aerial photography of each drone.
  5. The inspection apparatus for port machinery according to claim 1, characterized in that,
    the basic data of the port machinery includes size data and position data of the port machinery;
    the route data includes planned route data and actual route data, wherein the planned route data includes automatic inspection route data and designated inspection route data;
    the status data of drone includes: time, detection point number, plane horizontal coordinate, plane longitudinal coordinate, altitude coordinate, pitch angle, tilt angle, yaw angle, pitch angle of the photography gimbal, tilt angle of the photography gimbal, and yaw angle of the of the photography gimbal;
    the processing of the basic data, the route data, the status data of drone and image data of port machinery includes: saving, deleting, modifying, encrypting, decrypting, classifying, filtering, statistics, importing, and exporting.
  6. The inspection apparatus for port machinery according to claim 1, characterized in that each drone includes: a fuselage, a flight module, a flight control module, a data transmission module, an image transmission module, an obstacle avoidance module, a positioning module, an aerial camera and a tracking camera.
  7. An inspection method for port machinery utilizing the inspection apparatus for port machinery of any one of claims 1 to 6, characterized in that the inspection method comprises:
    a step of route planning, planning an automatic inspection route based on the basic data of port machinery;
    a step of drone aerial photography, controlling each drone to fly according to automatic inspection route and taking aerial photos of the port machinery, monitoring the status of each drone, and receiving images acquired by the drone;
    a step of data processing, performing structural defect recognition and structural defect analysis on the images acquired by each drone, and generates evaluation results based on the structural defect recognition and structural defect analysis,
    wherein the defect recognition comprises identifying structural defect areas in images of port machinery by the structural defect image recognition module, marking and classifying structural defect areas, wherein the classification of structural defect areas includes: corrosion, cracks, missing bolts, paint stripping, and other structural defects;
    wherein the structural defect analysis comprises quantitative calculation by the structural defect image analysis module on the structural defect areas marked and classified by the image structural defect identification module, and correlating the results of the quantitative calculation with the structural defect areas, wherein the quantitative calculation of the structural defect areas includes: the size of the corrosion area, the size of the paint stripped area, the length and width of the crack, the number and location of the missing bolts.
  8. The inspection method for port machinery according to claim 7, characterized in that further comprises:
    a step of route adjusting, generating a designated inspection route according to the instructions obtained from the interactive interface, wherein each drone flies according to the designated inspection route to perform aerial photography of the port machinery.
  9. The inspection method for port machinery according to claim 7, characterized in that further comprising:
    a step of manual control, operating the hand-held remote controller, wherein each drone quits automatic operation and is controlled to fly and perform aerial photography of the port machinery under the control of the hand-held remote controller.
EP17924086.6A 2017-09-08 2017-10-19 Port machinery inspection device and inspection method Active EP3680648B1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710805734.0A CN107607543A (en) 2017-09-08 2017-09-08 The inspection device and method for inspecting of harbour machinery
PCT/CN2017/106796 WO2019047338A1 (en) 2017-09-08 2017-10-19 Port machinery inspection device and inspection method

Publications (3)

Publication Number Publication Date
EP3680648A1 EP3680648A1 (en) 2020-07-15
EP3680648A4 EP3680648A4 (en) 2021-06-23
EP3680648B1 true EP3680648B1 (en) 2023-07-19

Family

ID=61062825

Family Applications (1)

Application Number Title Priority Date Filing Date
EP17924086.6A Active EP3680648B1 (en) 2017-09-08 2017-10-19 Port machinery inspection device and inspection method

Country Status (3)

Country Link
EP (1) EP3680648B1 (en)
CN (1) CN107607543A (en)
WO (1) WO2019047338A1 (en)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109471450B (en) * 2018-03-10 2019-08-27 福建通图信息技术有限公司 Drone flying height adjusting method
CN109405830B (en) * 2018-09-04 2021-09-21 西安爱生无人机技术有限公司 Unmanned aerial vehicle automatic inspection method based on line coordinate sequence
EP3960688A4 (en) * 2019-04-26 2022-07-13 Sumitomo Heavy Industries Construction Cranes Co., Ltd. Crane inspection system and crane
CN111102967A (en) * 2019-11-25 2020-05-05 桂林航天工业学院 Intelligent navigation mark supervision system and method based on unmanned aerial vehicle
CN111524128A (en) * 2020-04-29 2020-08-11 三一海洋重工有限公司 Steel structure detection method and system, electronic device and storage medium
CN111625023A (en) * 2020-06-18 2020-09-04 上海海事大学 Port unmanned aerial vehicle routing inspection path planning method based on improved Astar algorithm
CN111924119B (en) * 2020-07-15 2022-04-08 招商局国际信息技术有限公司 Wharf intelligent point inspection method and system based on unmanned aerial vehicle
CN112348034A (en) * 2020-10-21 2021-02-09 中电鸿信信息科技有限公司 Crane defect detection system based on unmanned aerial vehicle image recognition and working method
CN112577472A (en) * 2020-12-03 2021-03-30 上海振华重工电气有限公司 Unmanned aerial vehicle photovoltaic inspection positioning system and method
CN113533369B (en) * 2021-06-21 2024-05-17 国网山东省电力公司鱼台县供电公司 Power transmission line inspection device
CN113495580A (en) * 2021-08-06 2021-10-12 澜枢(上海)智能科技有限公司 Unmanned aerial vehicle system for detecting crane structure of port and wharf and using method thereof

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6842534B1 (en) * 2000-09-28 2005-01-11 Itt Manufacturing Enterprises, Inc. Detecting material failures in ground locations
CN103869819B (en) * 2014-03-10 2016-08-17 中国矿业大学 A kind of ribbon conveyer automatic detecting method based on many rotor wing unmanned aerial vehicles
CN203845022U (en) * 2014-04-28 2014-09-24 赵子滨 Automation platform for providing service for routing inspection unmanned plane
CN106797438B (en) * 2014-10-17 2020-03-06 索尼公司 Control device, control method and aircraft device
CN104713527A (en) * 2015-03-26 2015-06-17 江苏光导通信科技有限公司 Single-time multi-angle shooting mechanism
WO2017103982A1 (en) * 2015-12-14 2017-06-22 株式会社 ニコン・トリンブル Defect detection apparatus and program
CA3008620A1 (en) * 2015-12-16 2017-06-22 Walmart Apollo, Llc Systems and methods of capturing and distributing imaging content captured through unmanned aircraft systems
KR101707865B1 (en) * 2016-11-28 2017-02-27 한국건설기술연구원 Unmanned air vehicle system for approaching to photograph facility, and closeup method using the same
CN106954042B (en) * 2017-03-13 2023-04-28 兰州交通大学 Unmanned aerial vehicle railway line inspection device, system and method
CN106970639A (en) * 2017-03-15 2017-07-21 武汉理工大学 A kind of harbour outdoor scene monitoring system and method based on unmanned aerial vehicle platform

Also Published As

Publication number Publication date
WO2019047338A1 (en) 2019-03-14
EP3680648A4 (en) 2021-06-23
CN107607543A (en) 2018-01-19
EP3680648A1 (en) 2020-07-15

Similar Documents

Publication Publication Date Title
EP3680648B1 (en) Port machinery inspection device and inspection method
CN111512256B (en) Automated and adaptive three-dimensional robotic site survey
JP7030431B2 (en) Inspection support system and inspection support control program
US10031529B2 (en) Unmanned vehicle (UV) control system
KR102675315B1 (en) Systems and methods for automatically inspecting surfaces
EP2908202B1 (en) Unmanned vehicle (UV) control system
US10682677B2 (en) System and method providing situational awareness for autonomous asset inspection robot monitor
JP6484695B1 (en) Ship block joint welding defect marking method
WO2018208995A1 (en) Self-localized mobile sensor network for autonomous robotic inspection
EP3312095B1 (en) Lightning strike inconsistency aircraft dispatch mobile disposition tool
CN112068539A (en) Unmanned aerial vehicle automatic driving inspection method for blades of wind turbine generator
CN110673141A (en) Mobile airport pavement foreign matter detection method and system
WO2024093420A1 (en) Patrol method and apparatus for cooperative operation of unmanned aerial vehicle and ground patrol robot
JP2024515632A (en) System and method for assisting a crane operator - Patents.com
CN112801432B (en) Intelligent inspection system for fan set blades and inspection method for fan set blades
CN114147389A (en) Welding auxiliary method and system for detecting welding process parameters
CN111780761A (en) Autonomous navigation method for inspection robot
CN117576920B (en) Traffic control system based on unmanned aerial vehicle
EP4223648A1 (en) Automated method and system for aircraft inspection with data validation
Silva Filho et al. Semi-Autonomous Industrial Robotic Inspection: Remote Methane Detection in Oilfield
CN114047780A (en) Fine intelligent inspection device based on unmanned aerial vehicle technology
WO2024121618A1 (en) Systems, methods, and devices for building a robot mission
CN118258398A (en) Routing inspection path planning method and device, electronic equipment and storage medium
CN118192666A (en) Unmanned aerial vehicle-based tunnel inspection system and method
CN118196923A (en) Safe docking system based on unmanned aerial vehicle inspection information

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20200408

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20210521

RIC1 Information provided on ipc code assigned before grant

Ipc: G01N 21/88 20060101AFI20210517BHEP

Ipc: G01C 11/02 20060101ALI20210517BHEP

Ipc: B64C 39/02 20060101ALI20210517BHEP

Ipc: G05D 1/10 20060101ALI20210517BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20230206

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602017071661

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: FP

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20231031

Year of fee payment: 7

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1589946

Country of ref document: AT

Kind code of ref document: T

Effective date: 20230719

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231020

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231119

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231120

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231019

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231119

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231020

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20231026

Year of fee payment: 7

Ref country code: DE

Payment date: 20231027

Year of fee payment: 7

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: BE

Payment date: 20231026

Year of fee payment: 7

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602017071661

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231019

26N No opposition filed

Effective date: 20240422

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20231019