CN113050698A - Method and device for searching target object, unmanned aerial vehicle equipment and storage medium - Google Patents
Method and device for searching target object, unmanned aerial vehicle equipment and storage medium Download PDFInfo
- Publication number
- CN113050698A CN113050698A CN202110336461.6A CN202110336461A CN113050698A CN 113050698 A CN113050698 A CN 113050698A CN 202110336461 A CN202110336461 A CN 202110336461A CN 113050698 A CN113050698 A CN 113050698A
- Authority
- CN
- China
- Prior art keywords
- compared
- image
- target object
- unmanned aerial
- aerial vehicle
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000004590 computer program Methods 0.000 claims description 6
- 238000010276 construction Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/12—Target-seeking control
Landscapes
- Engineering & Computer Science (AREA)
- Aviation & Aerospace Engineering (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Automation & Control Theory (AREA)
- Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a method and a device for searching a target object, unmanned aerial vehicle equipment and a storage medium. The method comprises the steps of constructing a background image library of a target area and storing the background image library in the unmanned aerial vehicle, wherein the background image library comprises all features of the target area and first space coordinates which correspond to the background images one by one; receiving a search instruction, executing a flight task according to a preset flight path, shooting an image to be compared of the target area, and acquiring a second space coordinate of the image to be compared; and when the first space coordinate is the same as the second space coordinate, comparing the image to be compared with the background image, and determining whether a target object exists in the image to be compared. By the method, lost objects in a large open field can be quickly and effectively searched.
Description
Technical Field
The embodiment of the invention relates to the technical field of unmanned aerial vehicles, in particular to a method and a device for searching a target object, unmanned aerial vehicle equipment and a storage medium.
Background
In recent years, unmanned aerial vehicles have wide application in various fields, large-area search can be realized in specific environments and specific occasions through the unmanned aerial vehicles, and the rapid positioning target is facilitated.
In the prior art, the target object searching in the wide area such as playground, sports meeting, grassland and the like can only search specific target objects such as automobiles, people, bicycles and the like, so that the target object searching is too limited to be suitable for searching all target objects.
Therefore, it is an urgent need to provide a searching method suitable for all targets.
Disclosure of Invention
The embodiment of the invention provides a method and a device for searching a target object, unmanned aerial vehicle equipment and a storage medium, which can realize the search of any target object.
In a first aspect, an embodiment of the present invention provides a method for searching for a target object, including:
constructing a background image library of a target area and storing the background image library in the unmanned aerial vehicle, wherein the background image library comprises all characteristics of the target area and first space coordinates which are in one-to-one correspondence with the background images;
receiving a search instruction, executing a flight task according to a preset flight path, shooting an image to be compared of the target area, and acquiring a second space coordinate of the image to be compared;
and when the first space coordinate is the same as the second space coordinate, comparing the image to be compared with the background image, and determining whether a target object exists in the image to be compared.
In a second aspect, an embodiment of the present invention further provides an apparatus for searching for a target object, including:
the construction module is used for constructing a background image library of a target area and storing the background image library in the unmanned aerial vehicle, wherein the background image library comprises all features of the target area and first space coordinates which are in one-to-one correspondence with the background images;
the acquisition module is used for receiving a search instruction, executing a flight task according to a preset flight path, shooting an image to be compared of the target area and acquiring a second space coordinate of the image to be compared;
and the comparison module is used for comparing the image to be compared with the background image when the first space coordinate is the same as the second space coordinate, and determining whether a target object exists in the image to be compared.
In a third aspect, an embodiment of the present invention further provides an unmanned aerial vehicle device, including:
one or more processors;
storage means for storing one or more programs;
the one or more programs are executed by the one or more processors, so that the one or more processors implement the method for searching for a target object according to any embodiment of the present invention.
In a fourth aspect, the embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the method for searching for a target object according to any of the embodiments of the present invention.
The embodiment of the invention provides a method and a device for searching a target object, unmanned aerial vehicle equipment and a storage medium, wherein a background image library of a target area is firstly established and stored in an unmanned aerial vehicle, wherein the background image library comprises all characteristics of the target area and first space coordinates which are in one-to-one correspondence with the background image; then receiving a search instruction, executing a flight task according to a preset flight path, shooting an image to be compared in a target area, and acquiring a second space coordinate of the image to be compared; and finally, when the first space coordinate is the same as the second space coordinate, comparing the image to be compared with the background image, and determining whether a target object exists in the image to be compared. By means of the technical scheme, quick and effective search for lost objects in a large-scale open field can be achieved.
Drawings
Fig. 1 is a schematic flowchart illustrating a method for searching a target object according to an embodiment of the present invention;
fig. 2 is a flowchart illustrating a method for searching for a target object according to a second embodiment of the present invention;
FIG. 3a is a flowchart illustrating an exemplary process before searching for a target object according to a second embodiment of the present invention;
FIG. 3b is a flowchart illustrating an exemplary searching process of a method for searching for a target object according to a second embodiment of the present invention;
fig. 4 is a schematic structural diagram of an apparatus for searching a target object according to a third embodiment of the present invention;
fig. 5 is a schematic structural diagram of an unmanned aerial vehicle device according to a fourth embodiment of the present invention.
Detailed Description
Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present invention are shown in the drawings, it should be understood that the present invention may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and the embodiments of the present invention are illustrative only and are not intended to limit the scope of the present invention.
It should be understood that the various steps recited in the method embodiments of the present invention may be performed in a different order and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the invention is not limited in this respect.
The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.
It should be noted that the terms "first", "second", and the like in the present invention are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in the present invention are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that reference to "one or more" unless the context clearly dictates otherwise.
The names of messages or information exchanged between devices in the embodiments of the present invention are for illustrative purposes only, and are not intended to limit the scope of the messages or information.
Example one
After articles are lost in large open places with wide areas, such as playgrounds, sports halls, squares and other areas, if people want to search for the lost articles in the places with the lost articles, a large amount of manpower, material resources and time are consumed, and the search result is not accurate enough due to human factors in a manual search mode. Therefore, the method for searching the target object provided by the embodiment of the invention can effectively search lost objects in a large-scale open field by utilizing the characteristics of aerial photography of the unmanned aerial vehicle and combining the vision technology.
Fig. 1 is a schematic flow chart of a method for searching for a target object according to an embodiment of the present invention, which is applicable to a situation of searching for a target object, such as a situation of searching for a target in a large open space. The method may be performed by a device for searching for a target, where the device may be implemented by software and/or hardware and is generally integrated on a drone device, and the drone in this embodiment may be a drone with a Real-time kinematic (RTK) carrier-phase differential centimeter-level positioning function for positioning centimeter-level position coordinates.
As shown in fig. 1, a method for searching for a target object according to an embodiment of the present invention includes the following steps:
s110, constructing a background image library of a target area and storing the background image library in the unmanned aerial vehicle, wherein the background image library comprises all features of the target area and first space coordinates which are in one-to-one correspondence with the background images.
The target area can be an area where articles are lost, and the target area can be a large open area such as a grassland, a playground and a football field. The selection of the target area is not particularly limited herein.
In this embodiment, the background image library may be a database of background images, and the background image may be a picture or a video of a shot target area. Here, the method of constructing the background image library of the target region is not particularly limited. Exemplarily, the spatial coordinates corresponding to all the features and the background images in the target area can be obtained through a panoramic image of the target area; preferably, the unmanned aerial vehicle can execute the aerial photography task according to the flight path to acquire all the features, the background image and the space coordinates corresponding to the background image in the target area, and the flight path of the unmanned aerial vehicle executing the task of constructing the background image library of the target area can be stored in the unmanned aerial vehicle or in the cloud, so that the unmanned aerial vehicle executing the search instruction can acquire the space coordinates conveniently. In addition, all features may include morphological features and positional features of all items within the target area.
It should be noted that the construction of the target area background image library requires shooting in a situation where no person is flowing in the target area. For example, when the target area is a golf course, the target area background image library may be constructed at any time before the golf course is not open to the outside, and constructing the target area background image library at that time may ensure that the background image library includes only the features of the stationary objects within the target area and may ensure that no people are flowing in the acquired background image.
The number of the background images in the background image library of the target area may be a fixed value, for example, if the background images are photos or videos obtained by the unmanned aerial vehicle through aerial photography in the target area, the number of the background images may be calculated by an application program corresponding to the unmanned aerial vehicle according to the area of the target area. It is understood that the number of background images taken must be sufficient to enable the background images to cover the entire target area.
In this embodiment, the first spatial coordinate may be a two-dimensional coordinate or a three-dimensional coordinate, and each background image has a spatial coordinate corresponding to the background image.
S120, receiving a search instruction, executing a flight task according to a preset flight path, shooting an image to be compared of the target area, and acquiring a second space coordinate of the image to be compared.
Wherein, search the instruction can be for instructing unmanned aerial vehicle to search the instruction of target object, and search the instruction and can be sent by unmanned aerial vehicle's main control unit or the application that unmanned aerial vehicle corresponds, can trigger the transmission of searching the instruction by the user through the button on the application that unmanned aerial vehicle corresponds, can also trigger the transmission by the user through unmanned aerial vehicle's main control unit and search the instruction. In addition, the sending of the search command may also be triggered in any other manner, which is not described herein.
It should be noted that the search instruction needs to be sent when no other person flows in the target area or only a stationary object exists in the target area, for example, the search instruction may be sent after the basketball is closed if the target area is a basketball hall.
The preset flight path can be a path of the unmanned aerial vehicle flying in the target area, and the flight path can be obtained by automatically calculating the number of the images to be compared according to the area of the target area and the number of the images to be compared, which are required to be shot, by an application program corresponding to the unmanned aerial vehicle. In this embodiment, the preset flight path may be determined by an application corresponding to the drone before receiving the search instruction.
Further, the preset flight path is determined by an application program corresponding to the unmanned aerial vehicle according to a trajectory area input by a user, and the trajectory area is formed by a trajectory line provided by the user on the application program.
For example, the determining of the flight path may include: the method comprises the steps that a user opens an application program corresponding to the unmanned aerial vehicle on a mobile phone, then the user can open a map in the application program, a line segment of a target area corresponding to the map is drawn out on the map displayed on a mobile phone screen by using fingers to serve as a track line, an area formed by sequentially connecting a plurality of track lines is a track area, and the track area represents a corresponding area on the map where the target area is located; after the track area is determined, the application program can automatically calculate the actual area of the track area, namely the area of the target area, and can determine a preset flight path according to the actual area of the track area, the shape of the track area and the number of images to be compared, and the application program can ensure that the shot images to be compared can cover the whole target area by performing flight shooting according to the preset flight path. Of course, the flight path of the unmanned aerial vehicle executing the task of constructing the background image library of the target area may also be acquired as the preset flight path of the unmanned aerial vehicle executing the search instruction.
It can be understood that the shape of the track area is the actual shape of the target area, that is, if the target area is a rectangular playground, the user may check out a rectangular area on the map of the application as the track area.
In this embodiment, the manner of determining the track area is not limited to the above-mentioned manner, and the user may determine the track area in any other feasible manner, which is not limited herein.
Wherein, the flight task can be the task that the image is compared for unmanned aerial vehicle shoots. The images to be compared can be photos or videos obtained by flying and shooting according to a preset flight path in a set place after the unmanned aerial vehicle receives a search instruction, the images to be compared can be the first images to be compared or a plurality of images to be compared after the unmanned aerial vehicle receives the search instruction, and the number of the images to be compared can be the same as that of the background images.
Optionally, the first spatial coordinate of each background image may be used as a shooting position of each image to be compared, that is, the second spatial coordinate, and the position relationships correspond to one another. The second spatial coordinate may be a two-dimensional coordinate or a three-dimensional coordinate, where no specific limitation is imposed on the manner of obtaining the second spatial coordinate, and the second spatial coordinate may be obtained, for example, by a positioning system on the unmanned aerial vehicle.
Further, if the image to be compared is a photo, the determination method of the second spatial coordinate of the image to be compared may be: determining total flight time according to the distance of a preset flight path and a preset flight speed; and determining the time corresponding to the image to be compared according to the total flight time and the number of the background images.
Specifically, the determination method of the second spatial coordinate of the image to be compared may be: determining the ratio of the distance of the preset flight path to the preset flight speed as the total flight time; determining the ratio of the total flying time to the background image as the time corresponding to the image to be compared; and when the time corresponding to the shot images to be compared arrives, determining the flying position of the unmanned aerial vehicle as a second space coordinate corresponding to the majority of images to be compared.
The distance of the preset flight path can be understood as the total length of the preset flight path; the total time of flight can be understood as the time required for flying according to a preset flight path and completing a flight task.
In this embodiment, if the to-be-compared image is a photo and the number of the to-be-compared photos is multiple, the unmanned aerial vehicle may shoot at a time corresponding to the shooting of each photo, for example, one to-be-compared photo may be shot at intervals of one second, that is, the time for shooting the first to-be-compared photo is 10: 01: 01 seconds, the time for taking the second to-be-compared picture is 10: 01: 02 seconds, and so on.
Optionally, if the image to be compared is a video, the determining manner of the second spatial coordinate of the image to be compared may be: and determining a second space coordinate of the image to be compared according to the preset flight path, the preset flight speed, the number of the background images and the time length for shooting the video. It can be understood that the video is obtained by shooting the unmanned aerial vehicle hovering in the air, and the duration of the video is the duration of hovering of the unmanned aerial vehicle.
Further, the determination method of the second spatial coordinate of the image to be compared may further be: determining a distance interval value for shooting two connected images to be compared according to the preset distance of the flight path and the number of the background images, and determining a second space coordinate corresponding to the images to be compared according to the distance interval value and the flight position record. It is understood that the above-mentioned manner of determining the second spatial coordinate may be applied to the case where the image to be compared is a photo or a video.
Specifically, the determination method of the second spatial coordinate of the image to be compared may further be: determining the ratio of the distance of the preset flight path to the number of the background images as a distance interval value for shooting two adjacent images to be compared; determining the shooting position of the image to be compared according to the distance interval value and the flight position record; and when the distance interval value is reached, determining the flying position of the unmanned aerial vehicle as a second space coordinate corresponding to the image to be compared.
When the images to be compared are photos and the number of the images to be compared is multiple, for example, the distance interval value may be 1 meter apart, that is, the unmanned aerial vehicle takes one photo to be compared every 1 meter of flight, and a position coordinate may be obtained every other meter on the flight position record as a second space coordinate. It should be noted that the determination method of the second spatial coordinate mentioned in the embodiment of the present invention is not limited to the determination method of the second spatial coordinate, and the second spatial coordinate corresponding to the image to be compared may also be determined in any other manner, which is not limited herein.
S130, when the first space coordinate is the same as the second space coordinate, comparing the image to be compared with the background image, and determining whether a target object exists in the image to be compared.
The target object can be a lost object, and whether the target object exists in the target area can be determined by comparing the image to be compared with the background image, which has the same coordinates.
Specifically, the manner of comparing the image to be compared with the background image and determining whether a target object exists in the image to be compared may include:
in the first mode, after a to-be-compared image is obtained by shooting, the to-be-compared image can be compared with a background image in real time, and the similarity of the two images is determined. And the second space coordinate corresponding to the image to be compared is the same as the first space coordinate corresponding to the background image.
It can be understood that, by the first method, after the first image to be compared is obtained, the image is compared with the background image corresponding to the first spatial coordinate in the background image library, which is the same as the second spatial coordinate corresponding to the image to be compared, so as to determine whether the target object exists in the image to be compared. The first mode realizes real-time shooting and real-time comparison, and after the comparison determines the target object, the unmanned aerial vehicle can be controlled to stop flying and shooting.
And secondly, after the images to be compared are completely shot, the images to be compared with the background images in the same number are shot, and then each image to be compared is compared with the background image to determine whether the target object exists.
The two images can be compared only when the second spatial coordinate corresponding to the image to be compared is the same as the first spatial coordinate corresponding to the background image.
Whether a target object exists in the target area or not, namely the lost object exists, can be determined through comparison in any one of the two modes, and the specific mode for comparison can be selected according to actual conditions, and is not limited specifically here. In addition, if the user wants to stop searching for the target object when comparing in the first mode, the unmanned aerial vehicle can be controlled to stop executing the task through the remote controller of the unmanned aerial vehicle, and the unmanned aerial vehicle can also be controlled to stop executing the task through the end button in the application program corresponding to the unmanned aerial vehicle.
It should be noted that, in this embodiment, a specific manner of comparing the image to be compared with the background image is not limited, and may be any manner capable of determining similarity between two images, that is, the manner may effectively compare two images to determine whether there is a difference between the two images. It is understood that comparing the image to be compared with the background image may be performed by a main controller in the drone.
Furthermore, after the existence of the target object is determined, the area where the target object is located can be determined according to the shooting position of the image to be compared including the target object, namely the second space coordinate, and the specific position of the target object in the target area can be further determined by combining a global positioning system in the unmanned aerial vehicle.
Further, if it is determined that the target object exists and the specific position of the target object is obtained, the unmanned aerial vehicle can be controlled to fly to the specific position of the target object through the remote controller of the unmanned aerial vehicle, and when the unmanned aerial vehicle flies to the position of the target object, the prompting message can be sent through an application program corresponding to the unmanned aerial vehicle. The reason for setting the prompt information is that in the case that a plurality of objects exist correspondingly, after the position of the first object is determined, the prompt information is sent to the user, and whether the next object needs to be searched continuously is confirmed according to the instruction of the user.
The method for searching the target object provided by the embodiment of the invention comprises the steps of firstly, constructing a background image library of a target area and storing the background image library in the unmanned aerial vehicle, wherein the background image library comprises all characteristics of the target area and first space coordinates which are in one-to-one correspondence with the background images; then receiving a search instruction, executing a flight task according to a preset flight path, shooting an image to be compared in a target area, and acquiring a second space coordinate of the image to be compared; and finally, when the first space coordinate is the same as the second space coordinate, comparing the image to be compared with the background image, and determining whether a target object exists in the image to be compared. The method can realize quick and effective search of lost objects in large-scale open fields.
Example two
Fig. 2 is a schematic flow chart of a method for searching for a target object according to a second embodiment of the present invention, which is optimized based on the above embodiments. Specifically, before the constructing a target area background image library and storing the target area background image library in the unmanned aerial vehicle, the method further includes: acquiring the area of a target area and attribute information of the unmanned aerial vehicle; and determining the number of background images according to the area and the attribute information. Further, this embodiment also compares the image to be compared with the background image, and determines whether there is a target object in the image to be compared, which is further embodied as: determining the structural similarity between the image to be compared and the background image; determining whether a target object exists in the images to be compared based on the structural similarity; if the target object is determined to exist, marking the image to be compared with the target object, or/and sending a second space coordinate of the image to be compared with the target object to a ground terminal, and stopping shooting and returning the unmanned aerial vehicle. Please refer to the first embodiment for a detailed description of the present embodiment.
As shown in fig. 2, a method for searching for a target object according to a second embodiment of the present invention includes the following steps:
s210, acquiring the area of a target area and attribute information of the unmanned aerial vehicle;
this step is to establish background image storehouse through the mode of unmanned aerial vehicle aerial photograph.
Wherein, unmanned aerial vehicle's attribute information can include unmanned aerial vehicle's inherent attribute, and is exemplary, can include unmanned aerial vehicle's model, unmanned aerial vehicle's size, unmanned aerial vehicle flying speed etc.. In this embodiment, the attribute information of the drone may include a preset flying height, a camera focal length, and a camera pixel.
The preset flying height can be a preset flying height of the unmanned aerial vehicle, can be set by a user through an application program corresponding to the unmanned aerial vehicle, and can be stored in a main controller of the unmanned aerial vehicle in advance; the camera focal length may be the focal length of the camera on the drone; the camera pixels may be pixels of a camera on the drone.
The area of the target area can be obtained from an application program corresponding to the unmanned aerial vehicle, and the area of the target area can be determined according to a track area selected by a user on a map of the application program.
And S220, determining the number of the background images according to the area and the attribute information.
In this step, unmanned aerial vehicle can calculate ground resolution according to predetermined flying height, camera focus and camera pixel, calculates the number of shooing of first group's photo according to ground resolution and area again. The area of the region included in a shot image to be compared can be determined according to the preset flying height, the camera focal length and the camera pixel.
Specifically, determining the number of background images according to the area and the attribute information includes: determining the ground resolution according to the preset flying height, the camera focal length and the camera pixel, wherein the ground resolution is the actual ground area represented by one pixel on the screen; and determining the quotient of the area and the ground resolution as the number of the background images.
The calculation formula of the ground resolution is as follows:
GSD＝Height*100f*CameralPixel/CameraFocalLength
where GSD represents ground resolution, Height represents preset fly Height, camerapixel represents camera pixels, and camerafocalcength represents camera focal length.
It can be understood that the larger the number of background images captured, the higher the resolution of the background images, and the smaller the range covered by the background images.
S230, constructing a background image library of the target area and storing the background image library in the unmanned aerial vehicle, wherein the background image library comprises all the characteristics of the target area and first space coordinates in one-to-one correspondence with the background images.
S240, receiving a search instruction, executing a flight task according to a preset flight path, shooting an image to be compared of the target area, and acquiring a second space coordinate of the image to be compared.
And S250, when the first space coordinate is the same as the second space coordinate, determining the structural similarity between the image to be compared and the background image.
The structural similarity can represent the similarity degree of the two photos, and the calculation formula of the structural similarity is as follows:
SSIM(x,y)＝[l(x,y)]α[c(x,y)]β[s(x,y)]γ
where SSIM (x, y) represents the structural similarity of two x, y photographs, l (x, y) represents the brightness comparison of two photographs, c (x, y) represents the contrast comparison of two photographs, s (x, y) represents the structural comparison of two photographs, and α, β, and γ represent constants greater than 0.
Wherein, the mathematical expressions of l (x, y), c (x, y) and s (x, y) are as follows:
wherein u isxAnd uyRespectively, mean values of x, y, σxAnd σyRespectively, the standard deviation of x and y, sigmaxyDenotes the covariance of x and y, c1、c2And c3Are each a constant.
In actual engineering calculations, it is generally assumed that α ═ β ═ γ ═ 1, and
the structural similarity formula can be simplified as follows:
the similarity structure of any two photos can be directly calculated through the simplified formula, and in this embodiment, the similarity structure value of the image to be compared and the background image taken each time is calculated through the simplified formula. The second space coordinate corresponding to the image to be compared is the same as the first space coordinate corresponding to the background image.
S260, determining whether a target object exists in the images to be compared based on the structural similarity.
The general structural similarity may be a value in a range of [0,1], in this step, if the structural similarity of the two images is in a set threshold range, it may be determined that the target object exists in the images to be compared, for example, the set threshold range may be [0.6,1], a larger value of the structural similarity indicates that the difference between the two images is smaller, and a larger value of the structural similarity indicates that the two images are the same.
If the structural similarity between the image to be compared with the second spatial coordinate and the background image with the first spatial coordinate is within the set threshold range, it can be determined that there is no difference between the two images, i.e., there is no target object in the image to be compared, and if the structural similarity between the image to be compared with the second spatial coordinate and the background image with the first spatial coordinate is not within the set threshold range, it can be determined that there is a certain degree of difference between the two images, i.e., there may be a target object in the image to be compared.
S270, if the target object is determined to exist, marking the image to be compared with the target object, or/and sending a second space coordinate of the image to be compared with the target object to a ground terminal, and stopping shooting and returning by the unmanned aerial vehicle.
In this step, after the presence of the target object is determined, the following three operations may be performed:
the first operation is to mark the images to be compared with the target object, and control the unmanned aerial vehicle to stop shooting and return to the home.
The image to be compared, in which the target object exists, may be marked in any manner, and for example, a mark code may be set for the image to be compared in which the target object exists.
The second operation is to send the second space coordinate corresponding to the image to be compared, where the target object exists, to the ground terminal, and control the unmanned aerial vehicle to stop shooting and return to the home.
Wherein, ground terminal can be the application that unmanned aerial vehicle corresponds, and unmanned aerial vehicle's main control unit can send second space coordinate to ground terminal.
And the third operation is to mark the image to be compared with the target object, send the second space coordinate corresponding to the image to be compared with the target object to the ground terminal, and control the unmanned aerial vehicle to stop shooting and return to the home. The three operation modes may be selected according to user requirements, and it is understood that other operations may be taken after the target object is determined to exist, which is not limited herein.
The method for searching the target object provided by the second embodiment of the invention embodies the process of determining whether the target object exists or not. According to the method, whether the target object exists can be determined more efficiently by comparing and screening the two images shot at the same space coordinate, and an object identification feature library does not need to be established in advance like the prior art, so that the calculated amount is reduced.
The embodiment of the invention provides a specific implementation mode on the basis of the technical scheme of each embodiment.
As a specific implementation manner of the present invention, for example, fig. 3a is a flowchart illustrating an example of a method for searching for a target object according to a second embodiment of the present invention, as shown in fig. 3a, before the target object is searched, that is, before the second shooting instruction is received, the following steps need to be executed:
step 1, setting a target area.
Namely, the user marks out the track area on the application program corresponding to the unmanned aerial vehicle.
Step 2, executing flight mission
And 3, photographing and generating a picture set.
Namely, a background image library of the target area is constructed.
Fig. 3b is a flowchart illustrating an exemplary searching process of a method for searching for a target object according to a second embodiment of the present invention, where the exemplary searching process of the target object shown in fig. 3b may include the following steps:
step 1, importing a preset flight task.
I.e. to introduce a preset flight path.
And 2, executing a preset flight task.
Namely receiving a searching instruction, and executing a flight task according to a preset flight path.
And 3, photographing and generating a new picture.
And shooting the image to be compared in the target area, and acquiring a second space coordinate of the image to be compared.
And 4, comparing and analyzing the new picture and the old picture.
When the first space coordinate is the same as the second space coordinate, the image to be compared is compared with the background image.
And 5, judging whether the new picture has the target object or not.
Namely, whether a target object exists in the images to be compared is determined.
If not, returning to execute the step 3, namely continuing flying to the next shooting position to shoot a new image to be compared; if yes, go to step 6.
And 6, flying the unmanned aerial vehicle to a specific position of the target object to hover.
And 7, judging whether to continue the task.
I.e. whether to continue the flight photography for the target object.
If yes, returning to execute the step 3; if not, the task is ended.
EXAMPLE III
Fig. 4 is a schematic structural diagram of an apparatus for searching for a target object according to a third embodiment of the present invention, which may be suitable for searching for a target object in a large open field, where the apparatus may be implemented by software and/or hardware and is generally integrated on a drone device.
As shown in fig. 4, the apparatus for searching for a target object includes: a construction module 410, an acquisition module 420, and an alignment module 430.
The construction module 410 is configured to construct a background image library of a target area and store the background image library in the unmanned aerial vehicle, where the background image library includes all features of the target area and first spatial coordinates corresponding to the background images one to one.
The obtaining module 420 is configured to receive a search instruction, execute a flight mission according to a preset flight path, capture an image to be compared in the target area, and obtain a second spatial coordinate of the image to be compared.
A comparing module 430, configured to compare the to-be-compared image with the background image when the first spatial coordinate is the same as the second spatial coordinate, and determine whether a target object exists in the to-be-compared image.
In this embodiment, the device first constructs a background image library of a target area through a construction module 410 and stores the background image library in the unmanned aerial vehicle, wherein the background image library comprises all features of the target area and first space coordinates in one-to-one correspondence with the background images; then, receiving a search instruction through the obtaining module 420, executing a flight task according to a preset flight path, shooting an image to be compared in a target area, and obtaining a second space coordinate of the image to be compared; finally, when the first spatial coordinate is the same as the second spatial coordinate, the comparison module 430 compares the image to be compared with the background image to determine whether a target object exists in the image to be compared.
The embodiment provides a device for searching a target object, which can effectively determine whether the target object exists by comparing a first group of photos with photos to be compared. In addition, the device can be applied to searching for any target object, and has a wide application range.
Further, the apparatus for searching for a target object further comprises a determining module, wherein the determining module is configured to: acquiring the area of a target area and attribute information of the unmanned aerial vehicle; and determining the number of background images according to the area and the attribute information.
On the basis of the above optimization, the fixed number determination module includes a first determination unit configured to: determining the ground resolution according to the preset flying height, the camera focal length and the camera pixel, wherein the ground resolution is the actual ground area represented by one pixel on the screen; and determining the quotient of the area and the ground resolution as the number of the background images.
Based on the technical scheme, the preset flight path is determined by an application program corresponding to the unmanned aerial vehicle according to a track area input by a user, and the track area is formed by a track line provided by the user on the application program.
Further, the apparatus further includes a second spatial coordinate determination module, specifically configured to: determining the ratio of the distance of the preset flight path to the preset flight speed as the total flight time; determining the ratio of the total flight time to the number of the background images as the time corresponding to shooting the images to be compared; and when the time corresponding to the shot images to be compared arrives, determining the flying position of the unmanned aerial vehicle as a second space coordinate corresponding to the images to be compared.
Further, the second spatial coordinate determination module is further configured to: determining the ratio of the distance of the preset flight path to the number of the background images as a distance interval value for shooting two adjacent images to be compared; determining the shooting position of the image to be compared according to the distance interval value and the flight position record; and when the distance interval value is reached, determining the flying position of the unmanned aerial vehicle as a second space coordinate corresponding to the image to be compared.
Further, the alignment module 430 is specifically configured to: determining the structural similarity between the image to be compared and the background image; determining whether a target object exists in the images to be compared based on the structural similarity; if the target object is determined to exist, marking the image to be compared with the target object, or/and sending a second space coordinate of the image to be compared with the target object to a ground terminal, and stopping shooting and returning the unmanned aerial vehicle.
The device for searching the target object can execute the method for searching the target object provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Example four
Fig. 5 is a schematic structural diagram of an unmanned aerial vehicle device according to a fourth embodiment of the present invention. As shown in fig. 5, the unmanned aerial vehicle provided by the fourth embodiment of the present invention includes: one or more processors 51 and storage 52; the processor 51 in the drone device may be one or more, and fig. 5 takes one processor 51 as an example; storage 52 is used to store one or more programs; the one or more programs are executed by the one or more processors 51, so that the one or more processors 51 implement the method for searching for a target object according to any one of the embodiments of the present invention.
The drone device may further include: an input device 53 and an output device 54.
The processor 51, the storage device 52, the input device 53 and the output device 54 in the drone device may be connected by a bus or other means, as exemplified by the bus connection in fig. 5.
The storage device 52 in the drone apparatus serves as a computer-readable storage medium, and can be used to store one or more programs, which may be software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the method for searching for a target provided in one or two embodiments of the present invention (for example, the modules in the apparatus for searching for a target shown in fig. 4 include a building module 410, an obtaining module 420, and a comparing module 430). The processor 51 executes various functional applications and data processing of the drone device by running software programs, instructions and modules stored in the storage device 52, that is, implements the method for searching for a target object in the above method embodiment.
The storage device 52 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created from use of the drone device, and the like. Further, the storage 52 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the storage 52 may further include memory located remotely from the processor 51, which may be connected to the device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 53 may be used to receive input numeric or character information and generate key signal inputs relating to user settings and function control of the drone apparatus. The output device 54 may include a display device such as a display screen.
And, when the one or more programs comprised by the above-mentioned drone device are executed by the one or more processors 51, the programs perform the following operations:
constructing a background image library of a target area and storing the background image library in the unmanned aerial vehicle, wherein the background image library comprises all characteristics of the target area and first space coordinates which are in one-to-one correspondence with the background images;
receiving a search instruction, executing a flight task according to a preset flight path, shooting an image to be compared of the target area, and acquiring a second space coordinate of the image to be compared;
and when the first space coordinate is the same as the second space coordinate, comparing the image to be compared with the background image, and determining whether a target object exists in the image to be compared.
EXAMPLE five
An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program is used for executing a method for searching for a target object when executed by a processor, and the method includes:
constructing a background image library of a target area and storing the background image library in the unmanned aerial vehicle, wherein the background image library comprises all characteristics of the target area and first space coordinates which are in one-to-one correspondence with the background images;
receiving a search instruction, executing a flight task according to a preset flight path, shooting an image to be compared of the target area, and acquiring a second space coordinate of the image to be compared;
and when the first space coordinate is the same as the second space coordinate, comparing the image to be compared with the background image, and determining whether a target object exists in the image to be compared.
Optionally, the program may be further configured to perform a method for searching for a target object according to any embodiment of the present invention when executed by a processor.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM), a flash Memory, an optical fiber, a portable CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. A computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take a variety of forms, including, but not limited to: an electromagnetic signal, an optical signal, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, Radio Frequency (RF), etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A method of searching for a target object, performed by an unmanned aerial vehicle, comprising:
constructing a background image library of a target area and storing the background image library in the unmanned aerial vehicle, wherein the background image library comprises all characteristics of the target area and first space coordinates which are in one-to-one correspondence with the background images;
receiving a search instruction, executing a flight task according to a preset flight path, shooting an image to be compared of the target area, and acquiring a second space coordinate of the image to be compared;
and when the first space coordinate is the same as the second space coordinate, comparing the image to be compared with the background image, and determining whether a target object exists in the image to be compared.
2. The method of claim 1, further comprising, before the constructing and storing the target area background image library in the drone:
acquiring the area of a target area and attribute information of the unmanned aerial vehicle;
and determining the number of background images according to the area and the attribute information.
3. The method of claim 2, wherein the attribute information comprises: the method comprises the following steps of presetting flying height, camera focal length and camera pixels; correspondingly, the determining the number of the background images according to the area and the attribute information includes:
determining the ground resolution according to the preset flying height, the camera focal length and the camera pixel, wherein the ground resolution is the actual ground area represented by one pixel on the screen;
and determining the quotient of the area and the ground resolution as the number of the background images.
4. The method of claim 1, wherein the predetermined flight path is determined by an application program corresponding to the drone according to a trajectory area input by a user, the trajectory area being formed by a trajectory line provided by the user on the application program.
5. The method of claim 1, wherein the second spatial coordinates of the images to be compared are determined by:
determining the ratio of the distance of the preset flight path to the preset flight speed as the total flight time;
determining the ratio of the total flight time to the number of the background images as the time corresponding to shooting the images to be compared;
and when the time corresponding to the shot images to be compared arrives, determining the flying position of the unmanned aerial vehicle as a second space coordinate corresponding to the images to be compared.
6. The method of claim 1, wherein the second spatial coordinates of the images to be compared are determined by:
determining the ratio of the distance of the preset flight path to the number of the background images as a distance interval value for shooting two adjacent images to be compared;
determining the shooting position of the image to be compared according to the distance interval value and the flight position record;
and when the distance interval value is reached, determining the flying position of the unmanned aerial vehicle as a second space coordinate corresponding to the image to be compared.
7. The method of claim 1, wherein the comparing the image to be compared with the background image to determine whether a target object exists in the image to be compared comprises:
determining the structural similarity between the image to be compared and the background image;
determining whether a target object exists in the images to be compared based on the structural similarity;
if the target object is determined to exist, marking the image to be compared with the target object, or/and sending a second space coordinate of the image to be compared with the target object to a ground terminal, and stopping shooting and returning the unmanned aerial vehicle.
8. An apparatus for searching for a target object, comprising:
the construction module is used for constructing a background image library of a target area and storing the background image library in the unmanned aerial vehicle, wherein the background image library comprises all features of the target area and first space coordinates which are in one-to-one correspondence with the background images;
the acquisition module is used for receiving a search instruction, executing a flight task according to a preset flight path, shooting an image to be compared of the target area and acquiring a second space coordinate of the image to be compared;
and the comparison module is used for comparing the image to be compared with the background image when the first space coordinate is the same as the second space coordinate, and determining whether a target object exists in the image to be compared.
9. An unmanned aerial vehicle device comprising:
one or more processors;
storage means for storing one or more programs;
the one or more programs are executable by the one or more processors to cause the one or more processors to perform the method of searching for a target of any of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method of searching for an object according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110336461.6A CN113050698B (en) | 2021-03-29 | 2021-03-29 | Method and device for searching target object, unmanned aerial vehicle equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110336461.6A CN113050698B (en) | 2021-03-29 | 2021-03-29 | Method and device for searching target object, unmanned aerial vehicle equipment and storage medium |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410223697.2A Division CN118092468A (en) | 2021-03-29 | Method and device for searching target object, unmanned aerial vehicle equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113050698A true CN113050698A (en) | 2021-06-29 |
| CN113050698B CN113050698B (en) | 2024-03-01 |
Family
ID=76516229
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110336461.6A Active CN113050698B (en) | 2021-03-29 | 2021-03-29 | Method and device for searching target object, unmanned aerial vehicle equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113050698B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004246075A (en) * | 2003-02-13 | 2004-09-02 | Xing Inc | Background video supply system of music playback device, music playback device and server |
| US20090161962A1 (en) * | 2007-12-20 | 2009-06-25 | Gallagher Andrew C | Grouping images by location |
| CN105430322A (en) * | 2016-01-22 | 2016-03-23 | 深圳市华玺科技有限公司 | Method and system for accessing video conference by unmanned aerial vehicle |
| CN106713701A (en) * | 2016-12-20 | 2017-05-24 | 华中科技大学 | Cluster motion data acquisition method and system based on image processing technology |
| CN108564787A (en) * | 2018-05-31 | 2018-09-21 | 北京理工大学 | Traffic observation procedure, system and equipment based on Floating Car method |
| CN208298351U (en) * | 2018-05-30 | 2018-12-28 | 安徽中天保安服务集团有限公司 | A kind of radar monitoring early warning system based on unmanned plane |
| CN109283943A (en) * | 2018-11-20 | 2019-01-29 | 深圳市道通智能航空技术有限公司 | Method for searching and search device when a kind of unmanned plane lost contact |
| CN109782791A (en) * | 2017-11-14 | 2019-05-21 | 长城汽车股份有限公司 | A kind of landing method and device of unmanned plane |
| CN110658852A (en) * | 2019-09-16 | 2020-01-07 | 苏州米龙信息科技有限公司 | Intelligent target searching method and system for unmanned aerial vehicle |
| CN112097770A (en) * | 2020-08-05 | 2020-12-18 | 中国人民解放军军事科学院国防科技创新研究院 | Multi-unmanned aerial vehicle collaborative full coverage path planning method and device, storage medium and terminal |
-
2021
- 2021-03-29 CN CN202110336461.6A patent/CN113050698B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004246075A (en) * | 2003-02-13 | 2004-09-02 | Xing Inc | Background video supply system of music playback device, music playback device and server |
| US20090161962A1 (en) * | 2007-12-20 | 2009-06-25 | Gallagher Andrew C | Grouping images by location |
| CN105430322A (en) * | 2016-01-22 | 2016-03-23 | 深圳市华玺科技有限公司 | Method and system for accessing video conference by unmanned aerial vehicle |
| CN106713701A (en) * | 2016-12-20 | 2017-05-24 | 华中科技大学 | Cluster motion data acquisition method and system based on image processing technology |
| CN109782791A (en) * | 2017-11-14 | 2019-05-21 | 长城汽车股份有限公司 | A kind of landing method and device of unmanned plane |
| CN208298351U (en) * | 2018-05-30 | 2018-12-28 | 安徽中天保安服务集团有限公司 | A kind of radar monitoring early warning system based on unmanned plane |
| CN108564787A (en) * | 2018-05-31 | 2018-09-21 | 北京理工大学 | Traffic observation procedure, system and equipment based on Floating Car method |
| CN109283943A (en) * | 2018-11-20 | 2019-01-29 | 深圳市道通智能航空技术有限公司 | Method for searching and search device when a kind of unmanned plane lost contact |
| CN110658852A (en) * | 2019-09-16 | 2020-01-07 | 苏州米龙信息科技有限公司 | Intelligent target searching method and system for unmanned aerial vehicle |
| CN112097770A (en) * | 2020-08-05 | 2020-12-18 | 中国人民解放军军事科学院国防科技创新研究院 | Multi-unmanned aerial vehicle collaborative full coverage path planning method and device, storage medium and terminal |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113050698B (en) | 2024-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200012854A1 (en) | Processing method for augmented reality scene, terminal device, system, and computer storage medium | |
| CN102647449B (en) | Based on the intelligent photographic method of cloud service, device and mobile terminal | |
| EP3766044B1 (en) | Three-dimensional environment modeling based on a multicamera convolver system | |
| CN107067794B (en) | Indoor vehicle positioning and navigation system and method based on video image processing | |
| WO2022036980A1 (en) | Pose determination method and apparatus, electronic device, storage medium, and program | |
| CN109241820B (en) | Unmanned aerial vehicle autonomous shooting method based on space exploration | |
| CN106406343B (en) | Control method, device and system of unmanned aerial vehicle | |
| RU2741443C1 (en) | Method and device for sampling points selection for surveying and mapping, control terminal and data storage medium | |
| CN103442436A (en) | Indoor positioning terminal, network, system and method | |
| WO2017133147A1 (en) | Live-action map generation method, pushing method and device for same | |
| CN107710736A (en) | Aid in the method and system of user's capture images or video | |
| CN106228218A (en) | The intelligent control method of a kind of destination object based on movement and system | |
| WO2020098431A1 (en) | Method and device for establishing map model | |
| CN112815923A (en) | Visual positioning method and device | |
| CN114943773A (en) | Camera calibration method, device, equipment and storage medium | |
| CN110555876A (en) | Method and apparatus for determining position | |
| WO2015154383A1 (en) | Photographing method and photographing terminal | |
| WO2023273415A1 (en) | Positioning method and apparatus based on unmanned aerial vehicle, storage medium, electronic device, and product | |
| KR101038940B1 (en) | System and method for managing image information using object extracted from image | |
| US20210357620A1 (en) | System, moving object, and information processing apparatus | |
| CN113063421A (en) | Navigation method and related device, mobile terminal and computer readable storage medium | |
| CN115407355B (en) | Library position map verification method and device and terminal equipment | |
| CN113050698B (en) | Method and device for searching target object, unmanned aerial vehicle equipment and storage medium | |
| CN114596537B (en) | Regional people stream data determination method, device, equipment and storage medium | |
| CN115988322A (en) | Method and device for generating panoramic image, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |