CN111325943B - Method and system for carrying out large-range forest fire prevention patrol early warning based on unmanned aerial vehicle - Google Patents
Method and system for carrying out large-range forest fire prevention patrol early warning based on unmanned aerial vehicle Download PDFInfo
- Publication number
- CN111325943B CN111325943B CN202010118233.7A CN202010118233A CN111325943B CN 111325943 B CN111325943 B CN 111325943B CN 202010118233 A CN202010118233 A CN 202010118233A CN 111325943 B CN111325943 B CN 111325943B
- Authority
- CN
- China
- Prior art keywords
- unmanned aerial
- aerial vehicle
- automatic
- data
- patrol
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000002265 prevention Effects 0.000 title claims abstract description 37
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000004891 communication Methods 0.000 claims abstract description 19
- 239000002131 composite material Substances 0.000 claims description 35
- 230000005540 biological transmission Effects 0.000 claims description 11
- 150000001875 compounds Chemical class 0.000 claims description 10
- 230000008569 process Effects 0.000 claims description 9
- 238000001228 spectrum Methods 0.000 claims description 9
- 238000005516 engineering process Methods 0.000 claims description 7
- 238000004458 analytical method Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 abstract description 9
- 238000004364 calculation method Methods 0.000 abstract description 5
- 230000006870 function Effects 0.000 abstract description 4
- 238000007689 inspection Methods 0.000 abstract description 3
- 210000001015 abdomen Anatomy 0.000 abstract description 2
- 239000000779 smoke Substances 0.000 description 12
- 238000012544 monitoring process Methods 0.000 description 6
- 230000008901 benefit Effects 0.000 description 5
- 238000001931 thermography Methods 0.000 description 5
- RZVHIXYEVGDQDX-UHFFFAOYSA-N 9,10-anthraquinone Chemical compound C1=CC=C2C(=O)C3=CC=CC=C3C(=O)C2=C1 RZVHIXYEVGDQDX-UHFFFAOYSA-N 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000006378 damage Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000010521 absorption reaction Methods 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000034994 death Effects 0.000 description 1
- 231100000517 death Toxicity 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000002329 infrared spectrum Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000012797 qualification Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000035939 shock Effects 0.000 description 1
- 230000007306 turnover Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B17/00—Fire alarms; Alarms responsive to explosion
- G08B17/12—Actuation by presence of radiation or particles, e.g. of infrared radiation or of ions
- G08B17/125—Actuation by presence of radiation or particles, e.g. of infrared radiation or of ions by using a video camera to detect fire or smoke
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/10—Simultaneous control of position or course in three dimensions
- G05D1/101—Simultaneous control of position or course in three dimensions specially adapted for aircraft
- G05D1/102—Simultaneous control of position or course in three dimensions specially adapted for aircraft specially adapted for vertical take-off of aircraft
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B17/00—Fire alarms; Alarms responsive to explosion
- G08B17/005—Fire alarms; Alarms responsive to explosion for forest fires, e.g. detecting fires spread over a large or outdoors area
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B29/00—Checking or monitoring of signalling or alarm systems; Prevention or correction of operating errors, e.g. preventing unauthorised operation
- G08B29/18—Prevention or correction of operating errors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04B—TRANSMISSION
- H04B7/00—Radio transmission systems, i.e. using radiation field
- H04B7/14—Relay systems
- H04B7/15—Active relay systems
- H04B7/185—Space-based or airborne stations; Stations for satellite systems
- H04B7/1851—Systems using a satellite or space-based relay
- H04B7/18513—Transmission in a satellite or space-based system
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/02—Services making use of location information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/30—Services specially adapted for particular environments, situations or purposes
- H04W4/40—Services specially adapted for particular environments, situations or purposes for vehicles, e.g. vehicle-to-pedestrians [V2P]
- H04W4/44—Services specially adapted for particular environments, situations or purposes for vehicles, e.g. vehicle-to-pedestrians [V2P] for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/90—Services for handling of emergency or hazardous situations, e.g. earthquake and tsunami warning systems [ETWS]
Landscapes
- Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Business, Economics & Management (AREA)
- Emergency Management (AREA)
- Multimedia (AREA)
- Aviation & Aerospace Engineering (AREA)
- Biodiversity & Conservation Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Public Health (AREA)
- Environmental & Geological Engineering (AREA)
- Computer Security & Cryptography (AREA)
- Health & Medical Sciences (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Automation & Control Theory (AREA)
- Astronomy & Astrophysics (AREA)
- Alarm Systems (AREA)
Abstract
The invention discloses a method and a system for carrying out large-scale forest fire prevention patrol early warning based on an unmanned aerial vehicle, wherein high-quality network coverage and data nearby calculation processing of forest edges are realized by constructing a 5G base station and mounting edge calculation equipment on the forest edges; the automatic unmanned aerial vehicle hangar is deployed in a forest to realize network bridging, information receiving, data processing, and automatic dispatching of an unmanned aerial vehicle to execute an accurate patrol observation task under a specific condition, so that the functions of unmanned aerial vehicle endurance and the like are guaranteed; deploying a plurality of long-endurance unmanned aerial vehicles to and from all hangars in a forest edge 5G network coverage range, and realizing 24-hour uninterrupted all-dimensional forest coverage fire prevention inspection; the communication satellite is in full coverage, so that the forest belly non-mobile network coverage area can be ensured to transmit data such as fire alarm early warning coordinates and the like through the satellite narrow band. The invention is suitable for scenes such as large-scale forest fire prevention automatic patrol and protection in complex terrains, emergency dispatching and commanding of forest fires and the like.
Description
Technical Field
The invention particularly relates to a method and a system for carrying out large-range forest fire prevention patrol early warning based on an unmanned aerial vehicle.
Background
Various natural disasters are frequently encountered in the forest reproduction process, and the damage caused by forest fires is extremely serious. The forest area of China is about 2.08 multiplied by 107hm2, the forest accumulation amount is 1.5137 multiplied by 109m3, the forest coverage rate is 21.63%, the forest fire happens tens of thousands of times every year on average, and the annual area of the damaged forest is hundreds of thousands of hectares. The frequent and frequent forest fires cause huge loss to valuable forest resources in China. Forest fires often have high difficulty in fighting fire, huge manpower and capital investment are needed, personnel injuries and deaths are easily caused, the ecological environment recovery after the fire disaster also needs dozens of or even hundreds of years, and therefore the forest fire needs to be protected from the rain and the fire disaster is quite important in monitoring hidden dangers and quickly and effectively early warning in the early stage of the fire disaster.
At present, the main forest fire prevention monitoring modes in China are satellite monitoring, observation of watchtowers, manned observation and manual inspection. This approach has the following disadvantages: 1. the satellite monitoring has high weather requirements (generally, the satellite remote sensing needs the cloud-free weather), only pictures and images can be shot, and the resolution is low. 2. The fire point cannot be accurately positioned by observation of the manual observation tower, and the optimal fire fighting time is delayed;
the manned aircraft has higher aviation forest protection cost and is an industry with higher risk. The manned machine is complex in operation and running, and safety accidents are easy to happen in the processes of patrol observation, fire treatment, mechanical descent and the like. 3. The manual patrol is greatly influenced by terrain factors, is easy to cause danger, and has small coverage and low efficiency.
Fly and patrol and protect and can solve above problem based on unmanned aerial vehicle, but present solution need take unmanned aerial vehicle to the operation scene before patrolling and protecting to manually carry out corresponding operation with the remote controller, can't accomplish unmanned aerial vehicle full-automatic independently operation. Unmanned aerial vehicle often is far away from practical application place, can not react immediately, carries unmanned aerial vehicle to arrive the scene, and the debugging takes off the wait delay time, can't effectively deal with promptness work. The working environment of the forest region is complex and the climate is bad, particularly in mountainous regions, patrolling personnel need to turn over mountains and go over mountains, the safety risk is high, and high-frequency and 24-hour patrol protection cannot be realized. The advantage of unmanned aerial vehicle in the aspect of forest fire prevention is seriously restricted.
Disclosure of Invention
The invention aims to provide a method and a system for carrying out large-range forest fire prevention patrol early warning based on an unmanned aerial vehicle.
The technical scheme adopted by the invention is as follows:
the system comprises a composite wing unmanned aerial vehicle, a multi-rotor unmanned aerial vehicle and 5G base stations, wherein the 5G base stations are arranged on a boundary line of a patrol area at intervals, the composite wing unmanned aerial vehicle travels between the 5G base stations and cruises the patrol area at a long distance, the multi-rotor unmanned aerial vehicle is used for low-altitude fixed-point reconnaissance in the patrol area, a plurality of automatic hangars are dispersedly arranged in the patrol area, the multi-rotor unmanned aerial vehicle is dispersedly arranged in the automatic hangars and reconnaissance the coverage area of the automatic hangars according to instructions, the multi-rotor unmanned aerial vehicle is in communication connection with the automatic hangars, and the automatic hangars are in communication connection with a data center; the automatic hangar is used for charging continuation of journey of composite wing unmanned aerial vehicle and many rotor unmanned aerial vehicle, dispose marginal computing equipment on the 5G basic station, marginal computing equipment calculates the data of cruising of processing composite wing unmanned aerial vehicle nearby and transmits to data center through the 5G basic station, and the last satellite communication module that carries of composite wing unmanned aerial vehicle and the satellite communication that covers this region of patrolling and protecting to in time send the police situation to data center through the satellite.
Furthermore, a plurality of fixed point watchtowers are distributed in the patrol area, watchers are configured on the watchtowers, and the watchers transmit the acquired data to a data center.
Further, be equipped with the unmanned aerial vehicle mounting cloud platform that is used for taking off and land of compound wing unmanned aerial vehicle and many rotor unmanned aerial vehicle on the automatic hangar.
Furthermore, the multi-rotor unmanned aerial vehicle transmits videos in real time through microwaves and an automatic hangar, the automatic hangar forwards the videos to a data center, the data center manually sends commands through a network to control the multi-rotor unmanned aerial vehicle to hover, and the shooting angle of the holder is adjusted to return video acquisition data in real time and provide the video acquisition data for the data center to be rechecked.
Furthermore, a double-spectrum cloud platform or edge computing device is carried on the composite-wing unmanned aerial vehicle, a smoke and fire point double-identification algorithm and a smoke and fire secondary judgment algorithm are arranged in the double-spectrum cloud platform or edge computing device, the smoke and fire point double-identification algorithm is used for finding fire in time and clearly confirming the fire by utilizing visible light, the smoke and fire secondary judgment algorithm carries out secondary judgment on the fire and filtering interference, the occurrence of false alarm is reduced, and accurate fire alarm is realized.
Further, the composite wing unmanned aerial vehicle goes to the nearest hangar to be charged or the battery is replaced under the condition of insufficient endurance.
A large-scale forest fire prevention patrol early warning method based on an unmanned aerial vehicle comprises the following steps:
step 3, the composite wing unmanned aerial vehicle cruises and acquires image data in a long distance according to the cruise line in the coverage area of the 5G base station;
step 4, carrying out image contrast analysis and identification on the fire point by extracting image features through a computer vision technology by the composite wing unmanned aerial vehicle;
step 5, judging whether a fire is formed according to the alarm threshold; if yes, sending alarm and position information to the data center through the satellite link and executing the step 6; otherwise, executing step 3;
step 6, the data center judges whether the suspected fire point position is in the coverage range of the automatic hangar; if so, the composite wing unmanned aerial vehicle continues the original route cruise task and executes the step 7; otherwise, the composite wing unmanned aerial vehicle flies to the nearest 5G base station to upload the cruise data in time, continues the original air route cruise task after the cruise data are uploaded, and executes the step 8;
7, dispatching a multi-rotor unmanned aerial vehicle in an automatic hangar by the data center to perform low altitude patrol corresponding to a suspected fire point and returning video data in real time;
and 9, completing the cruise task by the composite wing unmanned aerial vehicle, uploading all cruise data and finishing the cruise.
Further, the cruise task formulated in the step 2 comprises unmanned aerial vehicle model selection, takeoff hangar configuration, landing hangar configuration, flight trajectory planning, flight height setting, flight time configuration and mounting equipment configuration.
Further, the image features extracted in step 4 by the computer vision technology include color features, texture features, shape features, local feature points and the like.
Further, in step 7, the multi-rotor unmanned aerial vehicle transmits videos to an automatic hangar in real time through microwaves, and forwards the videos to the data center through the automatic hangar.
Further, in step 7, the multi-rotor unmanned aerial vehicle is manually controlled to hover by sending commands through the network, the shooting angle of the holder is adjusted, and the video acquisition data is returned in real time and provided to the data center for rechecking.
By adopting the technical scheme, the invention is not influenced by the landform and the landform, realizes high-frequency uninterrupted automatic fire prevention and patrol for forests in a large range for 24 hours, recognizes and warns, can remotely control the unmanned aerial vehicle to enter the periphery of a fire scene for reconnaissance through a network when forest fire occurs, and can check the fire scene state in real time. The method is suitable for scenes such as large-scale forest fire prevention automatic patrol and protection of complex terrains, emergency dispatching and commanding of forest fires and the like.
The invention takes full-automatic patrol of the unmanned aerial vehicle and automatic identification of the real-time remote sensing image as a technical means for forest fire prevention, and has the advantages of low cost and high efficiency. Can break away from artificial participation, reduce personnel's potential safety hazard, realize the smog to forest fire, flame automatic identification and carry out early warning processing. Meanwhile, the unmanned aerial vehicle is used for carrying out remote sensing image data acquisition and comparison to find geological disaster risks such as geological settlement, mountain cracks and the like and carrying out forest geological natural disaster warning prevention by replacing the unmanned aerial vehicle mounting and the algorithm pool algorithm. The method has good practicability, can reduce forest fires and geological disasters, reduce economic loss and casualties, and create considerable value for the society.
Drawings
The invention is described in further detail below with reference to the accompanying drawings and the detailed description;
FIG. 1 is a schematic structural diagram of a large-scale forest fire prevention patrol early warning system based on an unmanned aerial vehicle;
FIG. 2 is a schematic view of a patrol area configuration of a large-scale forest fire prevention patrol early warning system based on an unmanned aerial vehicle;
FIG. 3 is a schematic flow chart of a method for performing large-scale forest fire prevention patrol early warning based on an unmanned aerial vehicle.
Detailed Description
The invention aims to solve the problems that in a set of complete unmanned aerial vehicle forest fire prevention emergency command system which is established by taking an unmanned aerial vehicle as a core and covers the air, the sky and the ground, the unmanned aerial vehicle driver has high operating qualification requirement, complex patrol operation and small coverage range, and data such as returned collected videos cannot be processed in time. Finally, the unmanned aerial vehicle is used for realizing high-frequency and 24-hour full-range automatic cruising of the forest and accurate judgment of fire points in key areas, so that manual patrol is replaced, manpower resources are reduced, patrol risks are reduced, and the purpose of early warning at the beginning of a fire disaster in a patrol range and frequency is increased. The existing unmanned aerial vehicle forest fire prevention emergency command system is erected, and the original traditional satellite, manned aircraft, observation tower, forest checkpoint and individual patrol mode are adjusted to carry out large-area patrol by taking an unmanned aerial vehicle and a full-automatic hangar as cores to mount various detection devices; carrying out data convergence transmission by taking the lookout tower as a data relay platform; a satellite is used as a real-time communication channel and a large-range fire source trend monitoring tool; data are collected through a network, and finally forest fires are found and early warned through methods such as artificial intelligence, a big data platform and neural network calculation, warning pushing is carried out through multiple channels, and rechecking command is carried out manually.
As shown in fig. 1 to 3, the invention discloses a large-scale forest fire prevention patrol early warning system based on an unmanned aerial vehicle, which comprises a composite-wing unmanned aerial vehicle, a multi-rotor unmanned aerial vehicle and 5G base stations, wherein the plurality of 5G base stations are arranged on a boundary line of a patrol area at intervals, the composite-wing unmanned aerial vehicle travels between the 5G base stations and cruises the patrol area at a long distance, the multi-rotor unmanned aerial vehicle is used for low altitude fixed point reconnaissance in the patrol area, a plurality of automatic hangars are dispersedly arranged in the patrol area, the multi-rotor unmanned aerial vehicle is dispersedly arranged in the automatic hangars and reconnaissance the coverage area of the automatic hangars according to instructions, the multi-rotor unmanned aerial vehicle is in communication connection with the automatic hangars, and the automatic hangars are in communication connection with a data center; the automatic hangar is used for charging endurance of the compound wing unmanned aerial vehicle and the multi-rotor unmanned aerial vehicle, and the compound wing unmanned aerial vehicle goes to the nearest hangar to charge or replace the battery and other technologies in the condition of insufficient endurance, and belongs to the prior art. Dispose marginal computing equipment on the 5G basic station, marginal computing equipment calculates the data of cruising of processing composite wing unmanned aerial vehicle nearby and transmits to data center through the 5G basic station, and the last satellite communication module that carries of composite wing unmanned aerial vehicle and the satellite communication that covers this region of patrolling, and in time send the police situation to data center through the satellite.
Specifically, the 5G chip/base station is used for realizing high-bandwidth and low-delay transmission of data. A large amount of data can be transmitted back to the data center in the short time course of the unmanned aerial vehicle flying or hovering. The compound wing unmanned aerial vehicle is also called VTOL fixed wing unmanned aerial vehicle, has both possessed the advantage of many rotors VTOL, does not need the runway just can take off and land, possesses the advantages such as long duration, low noise, the gliding of fixed wing again simultaneously for long distance forest fire prevention patrols and protects. The multi-rotor unmanned aerial vehicle is strong in controllability, can take off and land vertically and hover, is mainly suitable for task types with low altitude, low speed and vertical take off and land and hover requirements, and is used for executing accurate rechecking of suspected fire sources and monitoring fire scene information in real time.
Automatic machine base: the unmanned aerial vehicle hangar has the characteristics of quick response, flexibility, maneuverability, automatic lifting operation and automatic charging/changing in the whole process, high-strength continuous operation, simplicity in use and maintenance, field real-time control, real-time data return, remote control, shock absorption fixation and the like, and can fully protect the unmanned aerial vehicle in severe environments in remote areas.
Edge computing chip/box: the edge computing chip/box can provide core capabilities such as network, computing, storage, application and the like at one side close to a data source, can provide near-end services, and is used for carrying algorithms such as picture recognition and the like to realize functions such as screening, intercepting, recognizing, splicing and the like of data.
The communication satellite realizes global communication coverage and is used for timely and effectively transmitting information such as alarms and coordinates.
Furthermore, a plurality of fixed point watchtowers are distributed in the patrol area, watchers are configured on the watchtowers, and the watchers transmit the acquired data to a data center.
Further, be equipped with the unmanned aerial vehicle mounting cloud platform that is used for taking off and land of compound wing unmanned aerial vehicle and many rotor unmanned aerial vehicle on the automatic hangar.
Furthermore, the multi-rotor unmanned aerial vehicle transmits videos in real time through microwaves and an automatic hangar, the automatic hangar forwards the videos to a data center, the data center manually sends commands through a network to control the multi-rotor unmanned aerial vehicle to hover, and the shooting angle of the holder is adjusted to return video acquisition data in real time and provide the video acquisition data for the data center to be rechecked.
Furthermore, double spectrum cloud platforms or edge computing equipment are carried on the composite wing unmanned aerial vehicle, a smoke and fire point double identification algorithm and a smoke and fire secondary judgment algorithm are built in the double spectrum cloud platforms or the edge computing equipment, and the smoke and fire point double identification algorithm and the smoke and fire secondary judgment algorithm adopt the existing mature technical scheme; the smoke and fire point dual recognition algorithm is used for timely discovering the fire and utilizing visible light to clearly confirm the fire, the smoke and fire secondary judgment algorithm carries out secondary judgment on the fire and filters interference, the occurrence of false alarm is reduced, and accurate fire alarm is realized.
Further, the composite wing unmanned aerial vehicle goes to the nearest hangar to be charged or the battery is replaced under the condition of insufficient endurance.
A large-scale forest fire prevention patrol early warning method based on an unmanned aerial vehicle comprises the following steps:
specifically, the model can be established through machine learning by entering the number of facility devices, the position distribution, the unmanned aerial vehicle performance and the parameter information of the mounting device. The model is used for generating and adjusting the flight route of the unmanned aerial vehicle, and the patrol scanning of the designated area is covered under limited resources, so that the planning of the unmanned aerial vehicle dispatching route is completed, and the planned unmanned aerial vehicle route can be ensured to cover the forest patrol area;
the main contents of route planning include: unmanned aerial vehicle model selection, takeoff hangar configuration, landing hangar configuration, flight trajectory planning, flight height setting, flight time configuration, mounting equipment configuration and the like;
step 3, the composite wing unmanned aerial vehicle cruises and collects image data in a long distance according to a cruising line in a 5G base station coverage area, specifically, the composite wing unmanned aerial vehicle shoots a forest through a mounted double-spectrum holder device (not only can shoot a conventional visible light image, but also can shoot an infrared thermal imaging image) in a flight task executing process, and visible light and infrared thermal imaging video data are stored in an onboard storage device;
step 4, carrying out image contrast analysis and identification on the fire point by extracting image features through a computer vision technology by the composite wing unmanned aerial vehicle;
specifically, image contrast analysis is performed for identifying fire points by extracting image features (including color features, texture features, shape features, local feature points, and the like) through a computer vision technique;
in addition, the situation that misjudgment often occurs to the collected images in the daily patrol process is solved, the image identification accuracy is improved by introducing and relearning the data of misreport and missed report in a mode of marking labels through an AI deep learning algorithm, and therefore the robustness of the system is improved;
step 5, judging whether a fire is formed according to the alarm threshold; if yes, sending alarm and position information to the data center through the satellite link and executing the step 6; otherwise, executing step 3;
step 6, the data center judges whether the suspected fire point position is in the coverage range of the automatic hangar; if so, the composite wing unmanned aerial vehicle continues the original route cruise task and executes the step 7; otherwise, the composite wing unmanned aerial vehicle flies to the nearest 5G base station to upload the cruise data in time, continues the original air route cruise task after the cruise data are uploaded, and executes the step 8;
7, dispatching a multi-rotor unmanned aerial vehicle in an automatic hangar by the data center to perform low altitude patrol corresponding to a suspected fire point and returning video data in real time;
and 9, uploading all cruise data and finishing the cruise after the composite wing unmanned aerial vehicle finishes the cruise task.
Further, the cruise task formulated in the step 2 comprises unmanned aerial vehicle model selection, takeoff hangar configuration, landing hangar configuration, flight trajectory planning, flight height setting, flight time configuration and mounting equipment configuration.
Further, the image features extracted in step 4 by the computer vision technology include color features, texture features, shape features, local feature points and the like.
Further, in step 7, the multi-rotor unmanned aerial vehicle transmits videos to an automatic hangar in real time through microwaves, and forwards the videos to the data center through the automatic hangar.
Further, in step 7, the multi-rotor unmanned aerial vehicle is manually controlled to hover by sending commands through the network, the shooting angle of the holder is adjusted, and the video acquisition data is returned in real time and provided to the data center for rechecking.
The following is a detailed description of the specific principles of the present invention:
in order to realize the technical scheme of the invention, the problem of how to build coverage in a large-scale forest by the flight control data transmission network of the unmanned aerial vehicle is solved firstly, and then the endurance of the unmanned aerial vehicle is increased, so that the unmanned aerial vehicle can carry out large-scale flight patrol. Position and state information need to be reported in real time in the flight patrol process of the unmanned aerial vehicle, warning information also needs to be sent in real time after forest fires are found, and disaster scene pictures are transmitted to an information center in time. The full-range coverage of the network can be achieved through satellite transmission, but the transmission cost is high, the transmission bandwidth is small, and the wireless communication network is only suitable for sending simple text and voice data, while the 5G network is large in uploading bandwidth, high in transmission speed, small in network delay, but small in coverage range and large in base station power consumption, so that the 5G network is built on the forest edge, equipment such as an automatic unmanned aerial vehicle library and a network bridge is used as a network relay, the full-range state and alarm data real-time transmission is achieved through combination of a communication satellite, and video data are transmitted rapidly in time. Except for serving as network transmission relay equipment, the automatic unmanned aerial vehicle hangar can perform rapid unmanned aerial vehicle charging/battery replacing operation, and the range extension of the unmanned aerial vehicle is realized. Specifically, the method comprises the following aspects:
configuring a flight plan: according to actual unmanned aerial vehicle manufacturer, the model is accessed into an unmanned aerial vehicle flight control platform through an interface in an unmanned aerial vehicle forest fire prevention emergency command system, the automatic take-off and landing of the unmanned aerial vehicle is called, the flight function of the route is planned, a timing task is configured, the starting position and the flight track of each unmanned aerial vehicle are automatically/manually planned according to the near-ground low-altitude flight scanning range of the unmanned aerial vehicle, and the planned unmanned aerial vehicle route can cover the forest patrol area.
The main contents comprise: unmanned aerial vehicle model selection, takeoff hangar configuration, landing hangar configuration, flight trajectory planning, flight altitude setting, flight time configuration, mounting equipment configuration and the like.
Flight patrol data acquisition: the unmanned aerial vehicle shoots the forest through the mounted double-spectrum holder equipment (not only can shoot conventional visible light images, but also can shoot infrared thermal imaging images) in the process of executing a flight task, and visible light and infrared thermal imaging video data are stored in the onboard storage equipment.
And (3) analyzing mounting equipment data: the unmanned aerial vehicle collects and stores video data and simultaneously conducts screenshot analysis through a mounted/integrated edge computing device/chip or a double-optical-spectrum cloud deck, fire point, smoke point and fire source feature library matching analysis is conducted on the captured picture through a built-in AI optical image recognition neural network algorithm, and fire point matching degree is calculated. And meanwhile, judging whether an abnormal image which is obviously higher than the ambient temperature or the temperature of which exceeds an alarm threshold value exists or not by combining infrared thermal imaging data. If the abnormal situation is found, the altitude is automatically reduced to hover around the ignition point, and whether misjudgment is carried out or not is further checked. And finally, judging whether smoke point or fire point information exists according to the matching degree and infrared spectrum information, and calculating the position information of the suspected fire point by combining carried GPS/Beidou positioning data.
Sending early warning information: in the patrol process, after discovering and identifying suspected fire source information, the edge computing equipment carried by the unmanned aerial vehicle immediately transmits alarm information through a satellite narrow band by an integrated satellite communication chip, and the main content comprises information such as alarm type, alarm grade, longitude and latitude.
Accurate rechecking of suspected fire points: the cruise unmanned aerial vehicle sends and reports an emergency and asks for help or increased vigilance the back and judges whether the early warning place reconnoiters unmanned aerial vehicle coverage in the automatic hangar according to the facility distribution condition of unmanned aerial vehicle local storage, if not then calculate through the algorithm and adjust set flight route, go to the position of 5G basic stations nearest apart from, hover or hover in the signal range after receiving the 5G signal, utilize integrated 5G chip to upload the collection data that cruises to command center and carry out artifical recheck. And returning to continuously execute the established flight plan after the transmission is finished, and if the endurance is insufficient, going to the nearest hangar to charge or replace the battery.
If the early warning place is within the coverage range of the automatic hangar, the set route is continuously executed for patrol, the system automatically calls a reconnaissance unmanned aerial vehicle in the unmanned hangar closest to the fire to perform surrounding flight reconnaissance and recheck on the early warning position, the video is transmitted to the unmanned hangar through microwave implementation and is forwarded to a background server through the hangar, the unmanned aerial vehicle is manually controlled to hover by sending a command through a network, the shooting angle of a cloud deck is adjusted, and the video acquisition data is returned in real time and provided to a command center for rechecking. After rechecking, the remote control one-key return flight returns to the automatic unmanned aerial vehicle warehouse for charging standby.
Manual identification processing: and the operator on duty at the operation center manually checks the suspected fire point according to the image sent by the unmanned aerial vehicle, and if the suspected fire point is judged by mistake, the information identifier is saved and serves as a subsequent machine learning material. And if the fire is determined, executing specified operation organization personnel in the emergency plan or extinguishing the fire through equipment according to the size of the fire.
By adopting the technical scheme, the high-quality network coverage and the nearby data calculation processing of the forest edge are realized by constructing the 5G base station and mounting the edge calculation equipment on the forest edge; the automatic unmanned aerial vehicle hangar is deployed in a forest to realize network bridging, information receiving, data processing, and automatic dispatching of an unmanned aerial vehicle to execute an accurate patrol observation task under a specific condition, so that the functions of unmanned aerial vehicle endurance and the like are guaranteed; deploying multiple unmanned planes, wherein long-endurance unmanned planes come and go to all hangars in a forest edge 5G network coverage range, and realizing 24-hour uninterrupted all-dimensional forest coverage fire prevention inspection; the communication satellite is in full coverage, so that the forest belly non-mobile network coverage area can be ensured to transmit data such as fire alarm early warning coordinates and the like through the satellite narrow band. The system is not influenced by landform and landform, realizes high-frequency 24-hour uninterrupted automatic fire prevention and patrol of forests in a large range, recognizes and warns, can remotely control the unmanned aerial vehicle to enter the periphery of a fire scene to reconnoiter when forest fires occur, and checks the fire scene state in real time. The method is suitable for scenes such as large-scale forest fire prevention automatic patrol and protection of complex terrains, emergency dispatching and commanding of forest fires and the like.
The invention takes full-automatic patrol of the unmanned aerial vehicle and automatic identification of the real-time remote sensing image as a technical means for forest fire prevention, and has the advantages of low cost and high efficiency. Can break away from artificial participation, reduce personnel's potential safety hazard, realize the smog to forest fire, flame automatic identification and carry out early warning processing. Meanwhile, the unmanned aerial vehicle is used for carrying out remote sensing image data acquisition and comparison to find geological disaster risks such as geological settlement, mountain cracks and the like and carrying out forest geological natural disaster warning prevention by replacing the unmanned aerial vehicle mounting and the algorithm pool algorithm. The method has good practicability, can reduce forest fires and geological disasters, reduce economic loss and casualties, and create considerable value for the society.
Claims (9)
1. The method comprises the steps that a large-scale forest fire prevention patrol early warning method is carried out based on an unmanned aerial vehicle, an adopted system comprises a composite wing unmanned aerial vehicle, a multi-rotor unmanned aerial vehicle and 5G base stations, the 5G base stations are arranged on a boundary line of a patrol area at intervals, the composite wing unmanned aerial vehicle travels between the 5G base stations and cruises the patrol area at a long distance, the multi-rotor unmanned aerial vehicle is used for low-altitude fixed-point reconnaissance in the patrol area, a plurality of automatic hangars are dispersedly arranged in the patrol area, the multi-rotor unmanned aerial vehicle is dispersedly arranged in the automatic hangars and reconnaissance is carried out on the coverage area of the automatic hangars according to instructions, the multi-rotor unmanned aerial vehicle is in communication connection with the automatic hangars, and the automatic hangars are in communication connection with a data center; the automatic hangar is used for charging and cruising of the compound wing unmanned aerial vehicle and the multi-rotor unmanned aerial vehicle, edge computing equipment is configured on the 5G base station, the edge computing equipment computes and processes cruising data of the compound wing unmanned aerial vehicle nearby and transmits the cruising data to the data center through the 5G base station, a satellite communication module is carried on the compound wing unmanned aerial vehicle to communicate with a satellite covering the cruising area, and the warning condition is sent to the data center in time through the satellite; the method is characterized in that: the method comprises the following steps:
step 1, 5G base stations are configured at intervals on the boundary of a patrol area, and a plurality of automatic hangars are dispersedly configured in the patrol area;
step 2, formulating a cruise line based on the positions of the 5G base station and the automatic hangar to form a cruise task;
step 3, the composite wing unmanned aerial vehicle cruises and acquires image data in a long distance according to the cruise line in the coverage area of the 5G base station;
step 4, carrying out image contrast analysis and identification on the fire point by extracting image features through a computer vision technology by the composite wing unmanned aerial vehicle;
step 5, judging whether a fire is formed according to the alarm threshold; if yes, sending alarm and position information to the data center through the satellite link and executing the step 6; otherwise, executing step 3;
step 6, the data center judges whether the suspected fire point position is in the coverage range of the automatic hangar; if so, the composite wing unmanned aerial vehicle continues the original route cruise task and executes the step 7; otherwise, the composite wing unmanned aerial vehicle flies to the nearest 5G base station to upload the cruise data in time, continues the original air route cruise task after the cruise data are uploaded, and executes the step 8;
7, dispatching a multi-rotor unmanned aerial vehicle in an automatic hangar by the data center to perform low altitude patrol corresponding to a suspected fire point and returning video data in real time;
step 8, the data center manually judges whether the composite wing unmanned aerial vehicle is mistakenly reported or not based on the uploaded cruise data of the composite wing unmanned aerial vehicle or the video data of the multi-rotor unmanned aerial vehicle; if so, controlling the multi-rotor unmanned aerial vehicle to return to the hangar; otherwise, executing a fire emergency plan to carry out fire remedy;
and 9, completing the cruise task by the composite wing unmanned aerial vehicle, uploading all cruise data and finishing the cruise.
2. The unmanned aerial vehicle-based large-scale forest fire prevention patrol early warning method according to claim 1, which is characterized in that: a plurality of fixed point watchtowers are distributed in the patrol area, watchers are configured on the watchtowers, and the watchers transmit acquired data to a data center.
3. The unmanned aerial vehicle-based large-scale forest fire prevention patrol early warning method according to claim 1, which is characterized in that: be equipped with the unmanned aerial vehicle carry cloud platform that is used for taking off and land of compound wing unmanned aerial vehicle and many rotor unmanned aerial vehicle on the automatic hangar.
4. The unmanned aerial vehicle-based large-scale forest fire prevention patrol early warning method according to claim 1, which is characterized in that: many rotor unmanned aerial vehicle pass through microwave and automatic hangar real-time transmission video, and the automatic hangar forwards to data center, and the many rotor unmanned aerial vehicle of data center manual work send the order through the network and transfer control to hover, and the adjustment cloud platform is shot the angle and is provided data center with real-time passback video acquisition data and recheck.
5. The unmanned aerial vehicle-based large-scale forest fire prevention patrol early warning method according to claim 1, which is characterized in that: carry on two spectrum cloud platforms or edge computing equipment on the compound wing unmanned aerial vehicle, two spectrum cloud platforms or edge computing equipment embeds smog fire double identification algorithm and firework secondary judgement algorithm, and the clear affirmation of smog fire double identification algorithm is used for in time discovering the condition of a fire and utilizing visible light to the condition of a fire, and firework secondary judgement algorithm carries out secondary judgement to the condition of a fire and filters the interference.
6. The unmanned aerial vehicle-based large-scale forest fire prevention patrol early warning method according to claim 1, which is characterized in that: and the composite wing unmanned aerial vehicle goes to the nearest hangar to be charged or the battery is replaced under the condition of insufficient endurance.
7. The unmanned aerial vehicle-based large-scale forest fire prevention patrol early warning method according to claim 1, which is characterized in that: the cruise tasks formulated in the step 2 comprise unmanned aerial vehicle model selection, takeoff hangar configuration, landing hangar configuration, flight trajectory planning, flight height setting, flight time configuration and mounting equipment configuration.
8. The unmanned aerial vehicle-based large-scale forest fire prevention patrol early warning method according to claim 1, which is characterized in that: and 4, extracting image features through a computer vision technology to obtain color features, texture features, shape features and local feature points.
9. The unmanned aerial vehicle-based large-scale forest fire prevention patrol early warning method according to claim 1, which is characterized in that: and 7, transmitting the videos to an automatic hangar in real time through microwaves by the multi-rotor unmanned aerial vehicle, forwarding the videos to the data center through the automatic hangar, sending commands to control the multi-rotor unmanned aerial vehicle to hover by a person in the data center through a network, adjusting the shooting angle of a cloud deck, and returning the video acquisition data in real time to the data center for rechecking.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010118233.7A CN111325943B (en) | 2020-02-26 | 2020-02-26 | Method and system for carrying out large-range forest fire prevention patrol early warning based on unmanned aerial vehicle |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010118233.7A CN111325943B (en) | 2020-02-26 | 2020-02-26 | Method and system for carrying out large-range forest fire prevention patrol early warning based on unmanned aerial vehicle |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111325943A CN111325943A (en) | 2020-06-23 |
| CN111325943B true CN111325943B (en) | 2021-10-08 |
Family
ID=71171103
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010118233.7A Active CN111325943B (en) | 2020-02-26 | 2020-02-26 | Method and system for carrying out large-range forest fire prevention patrol early warning based on unmanned aerial vehicle |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111325943B (en) |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112286558A (en) * | 2020-11-09 | 2021-01-29 | 蚌埠依爱消防电子有限责任公司 | Method for real-time updating analysis program of acquisition equipment |
| CN112444824A (en) * | 2020-11-19 | 2021-03-05 | 温州虎穴科技有限公司 | Special positioning system for unmanned aerial vehicle |
| CN112887915A (en) * | 2021-01-26 | 2021-06-01 | 荔波县黄江河国家湿地公园管理站 | Forest fire prevention intelligent terminal communication method based on Beidou short message |
| CN112947561B (en) * | 2021-02-09 | 2022-03-08 | 北京三快在线科技有限公司 | Unmanned aerial vehicle exception handling system, method and device |
| CN113103944A (en) * | 2021-04-02 | 2021-07-13 | 重庆万重山智能科技有限公司 | Trailer and forest fire monitoring system based on unmanned aerial vehicle |
| CN113534829B (en) * | 2021-06-11 | 2024-04-05 | 南京邮电大学 | Unmanned aerial vehicle daily patrol detecting system based on edge calculation |
| CN113673151B (en) * | 2021-08-06 | 2023-11-07 | 浙江水利水电学院 | Method for optimizing simulated wild fire extinguishing |
| CN113361504B (en) * | 2021-08-10 | 2021-11-16 | 南京邮电大学 | Edge group intelligent method based on unmanned aerial vehicle cooperative networking |
| CN114758465B (en) * | 2022-03-30 | 2024-03-15 | 南京林业大学 | Forest protection method based on unmanned cluster cooperative intelligent technology |
| CN115661245B (en) * | 2022-10-24 | 2023-04-18 | 东北林业大学 | Large-scale live wire instantaneous positioning method based on unmanned aerial vehicle |
| CN116013018B (en) * | 2023-01-19 | 2023-09-29 | 广东智联空天科技有限公司 | Forest fire prevention early warning analysis method and system based on unmanned aerial vehicle detection |
| CN117388521A (en) * | 2023-12-13 | 2024-01-12 | 华海智汇技术有限公司 | Water conservancy unmanned aerial vehicle flow measurement method and system |
| CN117592759B (en) * | 2024-01-18 | 2024-04-26 | 国家林业和草原局西北调查规划院 | Unmanned aerial vehicle-based forest and grass resource gridding patrol method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2689809A1 (en) * | 2012-07-24 | 2014-01-29 | The Boeing Company | Wildfire arrest and prevention system |
| CN107481465A (en) * | 2017-08-21 | 2017-12-15 | 昆明理工大学 | A kind of aerial unmanned plane infrared monitoring method for early warning of forest adaptive cruise |
| CN108712638A (en) * | 2018-07-30 | 2018-10-26 | 张会珍 | A kind of burning straw supervisory systems and method |
| CN109903505A (en) * | 2017-12-08 | 2019-06-18 | 中电科特种飞机系统工程有限公司 | A kind of fire monitoring system, method and medium |
| CN110459028A (en) * | 2019-06-28 | 2019-11-15 | 山西砺剑无人机科技有限公司 | A kind of forest fire early-warning system |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104802962B (en) * | 2015-03-27 | 2017-08-04 | 徐州飞梦电子科技有限公司 | A kind of water life-saving system and method |
| CN205543960U (en) * | 2016-02-15 | 2016-08-31 | 西南交通大学 | High efficiency unmanned aerial vehicle crowd power patrol inspection system |
| CN106530570A (en) * | 2016-11-04 | 2017-03-22 | 江苏金米智能科技有限责任公司 | Forest fire-fighting fire-extinguishing system based on unmanned aerial vehicle |
| US9973261B1 (en) * | 2016-12-28 | 2018-05-15 | Echostar Technologies Llc | Rapidly-deployable, drone-based wireless communications systems and methods for the operation thereof |
| CN109117749A (en) * | 2018-07-23 | 2019-01-01 | 福建中海油应急抢维修有限责任公司 | A kind of abnormal object monitoring and managing method and system based on unmanned plane inspection image |
| CN110047241A (en) * | 2019-04-27 | 2019-07-23 | 刘秀萍 | A kind of forest fire unmanned plane cruise monitoring system |
-
2020
- 2020-02-26 CN CN202010118233.7A patent/CN111325943B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2689809A1 (en) * | 2012-07-24 | 2014-01-29 | The Boeing Company | Wildfire arrest and prevention system |
| CN107481465A (en) * | 2017-08-21 | 2017-12-15 | 昆明理工大学 | A kind of aerial unmanned plane infrared monitoring method for early warning of forest adaptive cruise |
| CN109903505A (en) * | 2017-12-08 | 2019-06-18 | 中电科特种飞机系统工程有限公司 | A kind of fire monitoring system, method and medium |
| CN108712638A (en) * | 2018-07-30 | 2018-10-26 | 张会珍 | A kind of burning straw supervisory systems and method |
| CN110459028A (en) * | 2019-06-28 | 2019-11-15 | 山西砺剑无人机科技有限公司 | A kind of forest fire early-warning system |
Non-Patent Citations (1)
| Title |
|---|
| 森林防火无人机系统的研究;焦振田;《中国优秀硕士学位论文全文数据库》;20190815(第08期);第8页、11-13页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111325943A (en) | 2020-06-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111325943B (en) | Method and system for carrying out large-range forest fire prevention patrol early warning based on unmanned aerial vehicle | |
| CN111580425A (en) | System and method suitable for forest fire danger monitoring | |
| CN106054928B (en) | A kind of full region fire generation measuring method based on unmanned plane network | |
| CN106210627A (en) | A kind of unmanned plane fire dispatch system | |
| CN110109480A (en) | A kind of unmanned plane inspection search and rescue system and rescue method based on Multi-sensor Fusion | |
| CN109787679A (en) | Police infrared arrest system and method based on multi-rotor unmanned aerial vehicle | |
| CN101968913B (en) | Flame tracing method for forest fire area | |
| WO2020235834A1 (en) | Airfield bird strike system using robot drone | |
| CN112068539A (en) | Unmanned aerial vehicle automatic driving inspection method for blades of wind turbine generator | |
| CN109709984A (en) | A kind of base station type unmanned plane Atmosphere Environment Monitoring System Bases | |
| CN107481465A (en) | A kind of aerial unmanned plane infrared monitoring method for early warning of forest adaptive cruise | |
| CN108447482B (en) | Unmanned aerial vehicle voice communication control system | |
| WO2020000790A1 (en) | Vertical mine shaft detection method and system | |
| CN112162565A (en) | Uninterrupted autonomous tower inspection method based on multi-machine cooperative operation | |
| CN111529995A (en) | Fire extinguishing method and system based on unmanned aerial vehicle inspection | |
| CN110647170A (en) | Navigation mark inspection device and method based on unmanned aerial vehicle | |
| CN205440876U (en) | Novel four rotor unmanned aerial vehicle | |
| CN112327929A (en) | Unmanned aerial vehicle aviation flight management system | |
| CN115379306A (en) | Aircraft relay communication-based field disaster monitoring system and method | |
| CN112423256A (en) | Unmanned aerial vehicle control system and method for executing transportation, signal transfer and remote tasks | |
| CN114020000A (en) | Unmanned aerial vehicle control system for hidden fire inspection of coal mine surface | |
| CN117592759B (en) | Unmanned aerial vehicle-based forest and grass resource gridding patrol method | |
| CN114283548A (en) | Fire continuous monitoring method and system for unmanned aerial vehicle | |
| CN112750303A (en) | Highway patrol is alert with unmanned aerial vehicle system | |
| CN106327921A (en) | Undercarriage safety monitoring method based on course line and visible data fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |