CN117456134A - Unmanned aerial vehicle VR display method and device, electronic equipment and storage medium - Google Patents
Unmanned aerial vehicle VR display method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN117456134A CN117456134A CN202311374147.2A CN202311374147A CN117456134A CN 117456134 A CN117456134 A CN 117456134A CN 202311374147 A CN202311374147 A CN 202311374147A CN 117456134 A CN117456134 A CN 117456134A
- Authority
- CN
- China
- Prior art keywords
- lasting
- target
- live
- action image
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 230000002045 lasting effect Effects 0.000 claims abstract description 458
- 238000001514 detection method Methods 0.000 claims description 160
- 230000003287 optical effect Effects 0.000 claims description 37
- 238000004590 computer program Methods 0.000 claims description 12
- 238000013507 mapping Methods 0.000 claims description 6
- 230000008030 elimination Effects 0.000 claims description 4
- 238000003379 elimination reaction Methods 0.000 claims description 4
- 230000000149 penetrating effect Effects 0.000 description 38
- 230000003068 static effect Effects 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000009966 trimming Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/95—Computational photography systems, e.g. light-field imaging systems
- H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Computer Hardware Design (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Computer Graphics (AREA)
- Human Computer Interaction (AREA)
- Signal Processing (AREA)
- Image Processing (AREA)
Abstract
The invention provides an unmanned aerial vehicle VR display method, which comprises the following steps: acquiring a first live-action image obtained by shooting a target unmanned aerial vehicle in a target shooting area; detecting whether a first lasting target is contained in the first live-action image; if the first live-action image contains a first lasting target, acquiring a second live-action image obtained by shooting the reference unmanned aerial vehicle in a target shooting area; eliminating a first lasting target in the first live-action image based on the second live-action image to obtain a target live-action image; and after the target live-action image is subjected to VR image processing, obtaining a VR display image of the target unmanned aerial vehicle, and displaying the VR display image through VR display equipment. And eliminating the lasting target in the first real image by using the second real image shot by the reference unmanned aerial vehicle to obtain a target real image without the lasting target for VR image processing, so as to obtain a VR display image without lasting, and further avoid lasting lenses in VR images.
Description
Technical Field
The invention relates to the technical field of virtual reality, in particular to an unmanned aerial Vehicle (VR) display method and device, electronic equipment and a storage medium.
Background
Among existing unmanned aerial vehicle technologies, VR (Virtual Reality) technology is increasingly used. The combination of drones with VR enables users to explore the world from the perspective of the drone, particularly in aerial photography and detection. However, some technical problems still exist in the existing unmanned aerial vehicle VR technology, one of which is that the picture is worn. The image wearing refers to unwanted elements appearing in the image due to the structure or human factors of the unmanned aerial vehicle in the shooting process, and the elements can destroy the aesthetic feeling of the image and interfere the watching experience of audiences. In the unmanned aerial vehicle shooting process, the upper penetrating part possibly comprises unmanned aerial vehicle structures such as a tripod head support, a rotor wing and a lifting support of the unmanned aerial vehicle. The existing mode of eliminating the upper is mainly through designing a new unmanned aerial vehicle structure, however, the upper penetrating condition cannot be completely eradicated, and mainly because the unmanned aerial vehicle is used as a carrier of a camera, when the shooting angle of the camera is overlarge or the installation position of the camera is unreasonable, the condition of picture upper penetrating still exists. Therefore, the existing unmanned aerial vehicle VR technology still has the problem of a lasting target, which affects the viewing experience of the audience to a certain extent. .
Disclosure of Invention
The embodiment of the invention provides an unmanned aerial vehicle VR display method, which aims to solve the problem that a VR picture is worn in the VR display process of an unmanned aerial vehicle. The first real image shot by the target unmanned aerial vehicle is subjected to wall penetrating detection, when the first real image contains a first wall penetrating target, a second real image shot by the reference unmanned aerial vehicle is acquired, the same wall penetrating structure cannot appear in the second real image by utilizing the difference of the structures of the reference unmanned aerial vehicle and the target unmanned aerial vehicle, the wall penetrating target in the first real image is eliminated by utilizing the second real image, the target real image without the wall penetrating target is obtained, the VR image processing is performed by utilizing the target real image without the wall penetrating target, and the VR display image without the wall penetrating is obtained for display, so that the wall penetrating lens can be avoided in a VR picture, and the VR viewing experience of a user is improved.
In a first aspect, an embodiment of the present invention provides a VR display method of an unmanned aerial vehicle, where the method includes:
acquiring a first live-action image obtained by shooting a target unmanned aerial vehicle in a target shooting area;
detecting whether a first lasting target is contained in the first live-action image or not, wherein the first lasting target is determined based on a lasting structure of the target man-machine;
If the first live-action image comprises a first lasting target, acquiring a second live-action image obtained by shooting a reference unmanned aerial vehicle in the target shooting area, wherein the reference unmanned aerial vehicle and the target unmanned aerial vehicle have different unmanned aerial vehicle structures;
eliminating the first lasting target in the first live-action image based on the second live-action image to obtain a target live-action image;
and after the target live-action image is subjected to VR image processing, obtaining a VR display image of the target unmanned aerial vehicle, and displaying the VR display image through VR display equipment.
Optionally, the first live-action image includes color information, optical flow information and depth information, and the detecting whether the first live-action image includes a lasting target includes:
determining a foreground object in the first live-action image based on the color information;
determining the distance information from the foreground target to the target unmanned aerial vehicle according to the depth information of the foreground target;
determining the motion type of the foreground object based on the optical flow information of the foreground object;
determining whether the foreground object is a first lasting object or not based on the distance information of the foreground object and the motion type of the foreground object;
If the foreground object is a first lasting object, determining that the first live-action image comprises the first lasting object.
Optionally, the obtaining a second live-action image obtained by shooting the reference unmanned aerial vehicle in the target shooting area includes:
determining the position information of the first live-action image, and matching candidate live-action images in the same position according to the position information of the first live-action image;
if the non-lasting real image exists in the candidate real images, selecting the non-lasting real image as a second real image;
if the candidate live-action image does not have the non-lasting live-action image, determining a second live-action image in the candidate live-action image according to the first lasting objective.
Optionally, the candidate live-action image includes a second lasting target, and the determining, according to the first lasting target, the second live-action image in the candidate live-action image includes:
determining the lasting type corresponding to the first lasting target according to the motion type of the foreground target, and determining a detection frame of the first lasting target in the first live-action image;
determining a second live-action image in the candidate live-action image based on the lasting type of the first lasting target and the detection frame of the first lasting target, wherein the second lasting target of the second live-action image and the first lasting target of the first live-action image have different lasting types and detection frame positions.
Optionally, the determining, based on the lasting type of the first lasting target and the image position of the first lasting target, a second live-action image in the candidate live-action images includes:
determining a first lasting feature of the first live-action image based on the lasting type of the first lasting target and a detection frame of the first lasting target;
determining a second lasting characteristic of the candidate live-action image based on the lasting type of the second lasting target and a detection frame of the second lasting target;
determining the lasting similarity between the first live-action image and the candidate live-action image according to the characteristic similarity between the first lasting characteristic and the second lasting characteristic;
and determining one candidate live-action image with the smallest lasting similarity as a second live-action image in the candidate live-action images.
Optionally, the determining, based on the lasting type of the first lasting target and the detection frame of the first lasting target, a second live-action image in the candidate live-action images includes:
calculating the cross-over ratio between the detection frame of the first lasting target and the detection frame of the second lasting target;
determining the detection frame of the second lasting target with the cross ratio of zero as a candidate detection frame;
Calculating the minimum distance between the detection frame of the first lasting target and the candidate detection frame and the center point distance;
determining a weighted distance between the detection frame of the first lasting target and the candidate detection frame based on the minimum distance and the center point distance;
selecting a candidate detection frame with the largest weighted distance from the candidate detection frames as a target detection frame, and determining the candidate live-action image corresponding to the target detection frame as a second live-action image.
Optionally, the eliminating the first lasting target in the first live-action image based on the second live-action image to obtain a target live-action image includes:
mapping the region where the first lasting target is located into the second live-action image to obtain a target replacement image for replacing the first lasting target;
and in the first live-action image, replacing and eliminating the first lasting target through the target replacement image to obtain a target live-action image.
In a second aspect, an embodiment of the present invention further provides an unmanned aerial vehicle VR display apparatus, where the unmanned aerial vehicle VR display apparatus includes:
the first acquisition module is used for acquiring a first live-action image obtained by shooting the target unmanned aerial vehicle in the target shooting area;
The detection module is used for detecting whether the first live-action image comprises a first lasting target or not, and the first lasting target is determined based on a lasting structure of the target man-machine;
the second acquisition module is used for acquiring a second real image obtained by shooting the reference unmanned aerial vehicle in the target shooting area if the first real image contains a first lasting target, wherein the reference unmanned aerial vehicle and the target unmanned aerial vehicle have different lasting structures;
the elimination module is used for eliminating the first lasting target in the first live-action image based on the second live-action image to obtain a target live-action image;
and the processing module is used for obtaining the VR display image of the target unmanned aerial vehicle after the VR image processing is carried out on the target live-action image, and displaying the VR display image through VR display equipment.
In a third aspect, an embodiment of the present invention provides an electronic device, including: the system comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein the steps in the VR display method of the unmanned aerial vehicle are realized when the processor executes the computer program.
In a fourth aspect, an embodiment of the present invention provides a computer readable storage medium, where a computer program is stored on the computer readable storage medium, where the computer program when executed by a processor implements steps in the VR display method of the unmanned aerial vehicle provided by the embodiment of the present invention.
In the embodiment of the invention, a first live-action image obtained by shooting a target unmanned aerial vehicle in a target shooting area is obtained; detecting whether a first lasting target is contained in the first live-action image or not, wherein the first lasting target is determined based on a lasting structure of the target man-machine; if the first live-action image comprises a first lasting target, acquiring a second live-action image obtained by shooting a reference unmanned aerial vehicle in the target shooting area, wherein the reference unmanned aerial vehicle and the target unmanned aerial vehicle have different unmanned aerial vehicle structures; eliminating the first lasting target in the first live-action image based on the second live-action image to obtain a target live-action image; and after the target live-action image is subjected to VR image processing, obtaining a VR display image of the target unmanned aerial vehicle, and displaying the VR display image through VR display equipment. The first real image shot by the target unmanned aerial vehicle is subjected to wall penetrating detection, when the first real image contains a first wall penetrating target, a second real image shot by the reference unmanned aerial vehicle is acquired, the same wall penetrating structure cannot appear in the second real image by utilizing the difference of the structures of the reference unmanned aerial vehicle and the target unmanned aerial vehicle, the wall penetrating target in the first real image is eliminated by utilizing the second real image, the target real image without the wall penetrating target is obtained, the VR image processing is performed by utilizing the target real image without the wall penetrating target, and the VR display image without the wall penetrating is obtained for display, so that the wall penetrating lens can be avoided in a VR picture, and the VR viewing experience of a user is improved.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a method flowchart of an unmanned aerial vehicle VR display method provided in an embodiment of the present invention;
fig. 2 is a schematic structural diagram of an VR display device of an unmanned aerial vehicle according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
As shown in fig. 1, fig. 1 is a method flowchart of an unmanned aerial vehicle VR display method provided in an embodiment of the present invention. The VR display method of the unmanned aerial vehicle comprises the following steps:
101. and acquiring a first live-action image obtained by shooting the target unmanned aerial vehicle in the target shooting area.
In the embodiment of the invention, the target unmanned aerial vehicle may be an unmanned aerial vehicle for providing a live-action image to perform VR image processing, a rotatable cradle head is provided on the target unmanned aerial vehicle, a camera capable of shooting is provided on the rotatable cradle head, and a user can control a shooting angle of the camera by controlling rotation of the cradle head. In one possible embodiment, a flight route and shooting parameters may be set for the target unmanned aerial vehicle in advance, the target unmanned aerial vehicle may fly according to the set flight route, and shooting is performed through the set shooting parameters in the flight process, where the shooting parameters may include a shooting frame rate, a shooting resolution, a shooting angle, a shooting focusing parameter, and the like.
The target shooting area may be a user-selected flyable shooting area. The camera may be a binocular camera, and the first live-action image may include color information, depth information, and optical flow information. The color information may be RGB three-channel color information, the depth information may be acquired according to a dual-target camera, and the optical flow information may be obtained by performing optical flow estimation on the first live-action image and the last frame of live-action image.
102. Detecting whether the first live-action image contains a first lasting target.
In the embodiment of the invention, the first lasting objective is determined based on a lasting structure of the objective man-machine. The unmanned aerial vehicle's of above-mentioned target wears group's structure can be structures such as cloud platform support, rotor, lifting support, and at unmanned aerial vehicle flight shooting in-process, for the fixed setting of camera, unmanned aerial vehicle of different structures has different and wears group's structure.
And carrying out target detection on the first live-action image through the target detection model, and further judging whether the first live-action image contains the first lasting target or not. Specifically, sample wall penetrating images of a certain number of different unmanned aerial vehicles can be collected, wall penetrating targets in the sample wall penetrating images are marked, marking frames of the wall penetrating targets are obtained to serve as wall penetrating labels, each sample wall penetrating image corresponds to a group of wall penetrating labels, and the group of wall penetrating labels comprise at least one wall penetrating label. And constructing a training data set based on the sample lasting image and the lasting label corresponding to the sample lasting image. And constructing a model to be trained based on the deep convolutional neural network, wherein the input of the model to be trained is a sample lasting image, and the input of the model to be trained is a predicted frame of a lasting target in the sample lasting image. And performing supervised training on the model to be trained through a training data set, obtaining a target detection model after training is completed, wherein the input of the target detection model is a first live-action image, the output of the target detection model is a detection frame or a blank of a lasting target in the first live-action image, if the output is blank, the first live-action image is represented that the lasting target is not contained, and if the output is the detection frame of the lasting target, the output is represented that the lasting target is not contained.
Specifically, in the training process, a sample lasting image is input into a model to be trained for processing, a predicted lasting target of the sample lasting image is obtained, loss calculation is carried out on the predicted lasting target of the sample lasting image and a lasting label of the sample lasting image through a loss function, a loss value between the predicted lasting target of the sample lasting image and the lasting label of the sample lasting image is obtained, a counter propagation algorithm is adopted to carry out parameter adjustment on the model to be trained according to the loss value, the parameter adjustment process is iterated until the iteration times reach the preset times, training is stopped, a trained target detection model is obtained, a first live-action image is input into the target detection model for processing, an output result is obtained, and if the output result is null, the first lasting target is determined to exist in the first live-action image. The first lasting destination may be one or more.
Optionally, the first live-action image includes color information, optical flow information and depth information, and the step of detecting whether the first live-action image includes a lasting target specifically includes: determining a foreground object in the first live-action image based on the color information; determining the distance information from the foreground target to the target unmanned aerial vehicle according to the depth information of the foreground target; determining the motion type of the foreground object based on the optical flow information of the foreground object; determining whether the foreground object is a first lasting object or not based on the distance information of the foreground object and the motion type of the foreground object; if the foreground object is a first lasting object, determining that the first live-action image comprises the first lasting object.
In the embodiment of the present invention, the color information may be RGB three-channel color information, the depth information may be acquired according to a dual-target camera, and the optical flow information may be obtained by performing optical flow estimation on the first live-action image and the last frame live-action image. The depth information is used for representing the distance between the corresponding pixel point and the camera, and the optical flow information is used for representing the variation of the pixel point in the process from the last frame of live-action image to the first live-action image.
Specifically, the first live-action image includes an RGB image, a depth image, and an optical flow image, the RGB image corresponds to color information, the depth image corresponds to depth information, and the optical flow image corresponds to optical flow information.
The foreground detection algorithm can detect the foreground target of the RGB image, if the foreground target is not detected, the first live-action image can be determined to be a non-lasting image, then VR image processing is carried out on the first live-action image, a VR display image corresponding to the first live-action image is obtained, and the VR display image corresponding to the first live-action image is displayed through VR display equipment. The VR display device may be a wearable VR device, such as VR glasses or a VR head display. If the foreground object is detected, an image area or a detection frame of the foreground object may be acquired.
After the existence of the foreground object in the first live-action image is detected, the foreground object can be mapped into the depth image to obtain depth information of the foreground object, the distance between the foreground object and the camera is determined according to the depth information of the foreground object, and as the camera is arranged on the unmanned aerial vehicle, the distance from the foreground object to the camera can be determined as the distance from the foreground object to the target unmanned aerial vehicle, and when the distance from the foreground object to the target unmanned aerial vehicle is smaller than the preset distance, the foreground object can be determined as the structure on the target unmanned aerial vehicle.
After detecting that a foreground object exists in the first live-action image, mapping the foreground object into an optical flow image to obtain optical flow information of the foreground object, determining a motion type of the foreground object according to the optical flow information of the foreground object, determining that the motion type of the foreground object is a dynamic type if the average optical flow of the foreground object is greater than a preset first optical flow threshold value, and determining that the motion type of the foreground object is a static type if the average optical flow of the foreground object is less than a preset second optical flow threshold value, wherein the first optical flow threshold value is greater than the second optical flow threshold value. It should be noted that, in the flight process of the unmanned aerial vehicle, the static type of foreground object may be a static structure such as a cradle head support, a lifting support, and the like, and the dynamic type of foreground object may be a dynamic structure such as a rotor wing.
After determining the distance from the foreground object to the target unmanned aerial vehicle and the motion type of the foreground object, whether the corresponding foreground object is the first lasting object or not can be determined. Specifically, if the distance from the foreground object to the target unmanned aerial vehicle is smaller than the preset distance, the foreground object can be determined to be close to the target unmanned aerial vehicle, the foreground object can be determined to be a lasting object, and if the distance from the foreground object to the target unmanned aerial vehicle is larger than the preset distance, the foreground object can be determined to be a non-lasting object. After the lasting target is determined, the motion type of the lasting target can be determined according to the optical flow information of the lasting target, if the average optical flow of the lasting target is greater than a preset first optical flow threshold value, the motion type of the lasting target can be determined to be a dynamic type, and if the average optical flow of the lasting target is less than a preset second optical flow threshold value, the motion type of the lasting target can be determined to be a static type, wherein the first optical flow threshold value is greater than the second optical flow threshold value. After the foreground object is determined to be the lasting object and the motion type of the foreground object is determined, the foreground object is determined to be the first lasting object.
The first lasting target is determined through the color information, the optical flow information and the depth information included in the first live-action image, so that the accuracy of the lasting target detection can be improved.
103. And if the first live-action image comprises the first lasting target, acquiring a second live-action image obtained by shooting the reference unmanned aerial vehicle in the target shooting area.
In the embodiment of the invention, the reference unmanned aerial vehicle and the target unmanned aerial vehicle have different unmanned aerial vehicle structures. Specifically, above-mentioned reference unmanned aerial vehicle and target unmanned aerial vehicle have different and wear group's structure, say target unmanned aerial vehicle's rotor setting in organism top, reference unmanned aerial vehicle's rotor setting in the below of organism, target unmanned aerial vehicle's cloud deck support sets up in the top of organism, reference unmanned aerial vehicle's cloud deck support sets up in the below of organism, target unmanned aerial vehicle's lifting support sets up the left and right sides at the organism, reference unmanned aerial vehicle's lifting support sets up in the front and back both sides of organism, like this, when carrying out the same shooting angle, avoid same wearing group's structure to shelter from the picture for first live-action image appears the same target of wearing group with second live-action image. The reference drone may be one or more.
The reference unmanned aerial vehicle can shoot through the same flight route and shooting parameters of the target unmanned aerial vehicle to obtain candidate live-action images, and the candidate live-action images are stored in a corresponding candidate live-action image library. When the first lasting target appears in the first live-action image, the second live-action image with the same position can be matched in the candidate live-action image library. Because the reference unmanned aerial vehicle and the target unmanned aerial vehicle have different structures, the first upper penetrating target in the first live-action image cannot appear in the second live-action image with high probability.
Optionally, the step of acquiring the second live-action image obtained by shooting the reference unmanned aerial vehicle in the target shooting area specifically includes: determining the position information of a first live-action image, and matching candidate live-action images in the same position according to the position information of the first live-action image; if the non-lasting-effect image exists in the candidate real-effect images, selecting the non-lasting-effect image as a second real-effect image; if the candidate live-action image does not have the non-lasting live-action image, determining a second live-action image in the candidate live-action image according to the first lasting objective.
In the embodiment of the present invention, the position information of the first live-action image may be longitude and latitude information when the first live-action image is shot, and a plurality of candidate live-action images with the same or closest longitude and latitude information are matched in the candidate live-action images according to the longitude and latitude information of the first live-action image.
After the candidate live-action images with the same or closest longitude and latitude information are matched, judging whether each candidate live-action image is a lasting live-action image or not, specifically, carrying out lasting detection on the trimming live-action image by the lasting detection method of the first live-action image, if the second lasting target exists in the candidate live-action image, determining that the candidate live-action image is a lasting live-action image, and if the second lasting target does not exist in the candidate live-action image, determining that the candidate live-action image is a non-lasting live-action image. If a plurality of non-lasting real images exist in the candidate real images, calculating the similarity between the first real image and the plurality of non-lasting real images, and selecting the non-lasting real image with the largest similarity as a second real image.
If the matched candidate live-action image is a non-lasting live-action image, the candidate live-action image can be directly determined to be a second live-action image. If the matched candidate live-action images are all the lasting-action images, a second live-action image can be determined in the lasting-action images according to the first lasting-action targets, specifically, the lasting-action similarity between the first lasting-action targets and the second lasting-action targets in the lasting-action images can be calculated, and the lasting-action image with the smallest lasting-action similarity in the lasting-action images is determined as the second live-action image.
According to the position information of the first live-action image, the candidate live-action images at the same position are matched, and under the condition that the non-lasting live-action image does not exist in the candidate live-action images, the second live-action image is determined in the candidate live-action images according to the first lasting objective, so that the accuracy of the second live-action image can be further improved.
Optionally, the candidate live-action image includes a second lasting target, and the step of determining the second live-action image in the candidate live-action image according to the first lasting target specifically includes: determining the lasting type corresponding to the first lasting target according to the motion type of the foreground target, and determining a detection frame of the first lasting target in the first live-action image; based on the lasting type of the first lasting target and the detection frame of the first lasting target, determining a second real image in the candidate real image, wherein the second lasting target of the second real image and the first lasting target of the first real image have different lasting types and detection frame positions.
In the embodiment of the invention, under the condition that a non-lasting-effect image does not exist in the candidate real-effect image, the lasting type of the first lasting-effect object can be determined according to the motion type of the foreground object in the first real-effect image, the lasting-effect type can comprise a static type and a dynamic type, the first lasting-effect object of the static type can be a lasting-effect object corresponding to static structures such as a tripod head bracket, a lifting bracket and the like, and the first lasting-effect object of the dynamic type can be a lasting-effect object corresponding to a rotor wing to-be-dynamic structure.
The detection frame of the first lasting target can be obtained from the result of lasting detection on the first live-action image. According to the lasting type of the first lasting through object, determining a second lasting through object of different lasting through types with the first lasting through object culture in the candidate live-action images, and determining the candidate live-action image corresponding to the second lasting through object farthest from the first lasting through object detection frame as the second live-action image.
The second lasting target of the second live-action image and the first lasting target of the first live-action image have different lasting types and detection frame positions, so that the lasting target does not exist in the corresponding image area when the first lasting image is mapped to the second live-action image.
Optionally, the step of determining the second live-action image in the candidate live-action images based on the lasting type of the first lasting target and the image position of the first lasting target specifically includes: determining a first lasting feature of the first live-action image based on the lasting type of the first lasting target and the detection frame of the first lasting target; determining a second lasting characteristic of the candidate live-action image based on a lasting type of a second lasting target and a detection frame of the second lasting target; determining the lasting similarity between the first live-action image and the candidate live-action image according to the characteristic similarity between the first lasting characteristic and the second lasting characteristic; and determining one candidate live-action image with the smallest lasting similarity as a second live-action image in the candidate live-action images.
In the embodiment of the invention, the first lasting feature includes a lasting type of the first lasting target, a position of the detection frame, and a feature corresponding to the confidence of the detection frame, and the second lasting feature includes a lasting type of the second lasting target, a position of the detection frame, and a feature corresponding to the confidence of the detection frame. The detection frame of the first lasting target may be represented by (X1, Y1, W1, H1, R1), wherein (X1, Y1) represents the center coordinates of the detection frame of the first lasting target, W1 represents the width of the detection frame of the first lasting target, H1 represents the height of the detection frame of the first lasting target, and R1 represents the confidence of the detection frame of the first lasting target. The lasting type of the first lasting target and the detection frame of the first lasting target can be subjected to coding processing to obtain lasting characteristics of the first lasting target, and the lasting characteristics of the first lasting target are determined to be first lasting characteristics of the first live-action image. Similarly, the detection frame of the second lasting target may be represented by (X2, Y2, W2, H2, R2), where (X2, Y2) represents the center coordinates of the detection frame of the second lasting target, W2 represents the width of the detection frame of the second lasting target, H2 represents the height of the detection frame of the second lasting target, and R2 represents the confidence of the detection frame of the second lasting target. The lasting type of the second lasting target and the detection frame of the second lasting target can be subjected to coding processing to obtain lasting characteristics of the second lasting target, and the lasting characteristics of the second lasting target are determined to be the second lasting characteristics of the candidate live-action image.
After the first lasting feature of the first live-action image and the second lasting feature of the candidate live-action image are obtained, feature similarity between the first lasting feature and the second lasting feature can be calculated, wherein the feature similarity can be cosine similarity or Euclidean distance similarity. After the feature similarity between the first lasting feature and the second lasting feature is obtained, the feature similarity between the first lasting feature and the second lasting feature is determined to be the lasting similarity between the first live-action image and the second live-action image, and the candidate live-action image with the minimum lasting similarity is determined to be the second live-action image.
By determining the candidate live-action image with the smallest lasting similarity as the second live-action image, the second lasting target which is the same as the first lasting target in the second live-action image can be further avoided.
Optionally, the step of determining the second live-action image in the candidate live-action image based on the lasting type of the first lasting target and the detection frame of the first lasting target specifically includes: calculating the cross-over ratio between the detection frame of the first lasting target and the detection frame of the second lasting target; determining a detection frame of a second lasting target with the cross ratio of zero as a candidate detection frame; calculating the minimum distance between the detection frame of the first lasting target and the candidate detection frame and the distance between the center points; determining a weighted distance between the detection frame of the first lasting target and the candidate detection frame based on the minimum distance and the center point distance; selecting a candidate detection frame with the largest weighted distance from the candidate detection frames as a target detection frame, and determining a candidate live-action image corresponding to the target detection frame as a second live-action image.
In the embodiment of the invention, the first live-action image and the second live-action image have the same size resolution, the detection frame of the first lasting target can be mapped into the candidate live-action image, or the detection frame of the second lasting target is mapped into the first live-action image, the intersection ratio between the detection frame of the first lasting target and the detection frame of the second lasting target is calculated in the same image coordinate, the bigger the intersection ratio is, the closer the positions of the first lasting target and the second lasting target are, the closer the size is, the smaller the intersection ratio is, the more the positions of the first lasting target and the second lasting target are, the larger the size difference is, and the zero the intersection ratio is, the first lasting target and the second lasting target do not have a task intersection area.
And determining the detection frame of the second lasting target with the zero crossing ratio as a candidate detection frame according to the crossing ratio between the detection frame of the first lasting target and the detection frame of the second lasting target. If the second lasting target detection frame with the zero crossing ratio is not available, the second lasting target detection frame with the minimum crossing ratio is determined as the target detection frame, and the candidate live-action image corresponding to the target detection frame is determined as the second live-action image. If a plurality of detection frames of the second lasting target with the zero cross-over ratio exist, determining the detection frames of the second lasting target with the zero cross-over ratio as candidate detection frames, and calculating the minimum distance and the center point distance between the detection frames of the first lasting target and the candidate detection frames; and determining the weighted distance between the detection frame of the first lasting target and the candidate detection frame based on the minimum distance and the center point distance. The weighted distance may be d= [ ud 1 +(1-u)d 2 ]Wherein D is the weighted distance, D 1 D, the minimum distance between the detection frame of the first lasting target and the candidate detection frame is d 2 For the center point distance between the detection frame of the first lasting target and the candidate detection frame, u is 1/(|s) 1 -s 2 |+1),s 1 The detection frame area s of the detection frame which is the first upper penetrating target 2 The detection frame area is the candidate detection frame.
Because the candidate live-action image corresponding to the candidate detection frame with the largest weighted distance is determined as the second live-action image, the second lasting target which is the same as the first lasting target can be further avoided from appearing in the second live-action image.
104. And eliminating the first lasting target in the first live-action image based on the second live-action image to obtain a target live-action image.
In the embodiment of the invention, if the second lasting e target does not exist in the second live-action image, the second live-action image can be determined to be the target live-action image. If the second lasting target exists in the second live-action image, the corresponding region image in the first live-action image can be replaced by the region image where the non-second lasting target in the second live-action image is located, so that the first lasting target in the first live-action image is replaced by the region image where the non-second lasting target in the second live-action image is located, and the first live-action image without the first lasting target is obtained. After the corresponding region image in the first live-action image is replaced by the region image of the non-second lasting target in the second live-action image, smoothing processing can be carried out on the first live-action image, and the target live-action image is obtained.
Optionally, the step of eliminating the first lasting target in the first live-action image based on the second live-action image to obtain the target live-action image specifically includes: mapping the region where the first lasting target is located into a second live-action image to obtain a target replacement image for replacing the first lasting target; and in the first live-action image, replacing and eliminating the first lasting target through the target replacement image to obtain a target live-action image.
In the embodiment of the invention, the area where the first lasting target is located can be determined according to the position of the detection frame of the first lasting target, or the area where the first lasting target is located can be determined in the first live-action image through a target segmentation algorithm. After the region where the first lasting target is located is obtained, the region where the first lasting target is located can be mapped into a second live-action image, a region image corresponding to the region where the first lasting target is located is obtained in the second live-action image, the region image corresponding to the region where the first lasting target is located is extracted from the second live-action image, and the extracted region image is used as a target replacement image for replacing the first lasting target.
After the target replacement image for replacing the first lasting through target is obtained, replacing the image of the corresponding area in the first live-action image through the target replacement image, so that the purpose of replacing and eliminating the first lasting through target is achieved, and the target live-action image which does not contain the first lasting through target is obtained. Because the types and positions of the second lasting target in the second live-action image and the first lasting image in the first live-action image are different, the second lasting image is not included in the target replacement image obtained through the mapping of the first lasting target, so that the first lasting target is not included in the target live-action image, and the second lasting target is not introduced.
105. And after the target live-action image is subjected to VR image processing, obtaining a VR display image of the target unmanned aerial vehicle, and displaying the VR display image through VR display equipment.
In the embodiment of the invention, after the target live-action image is obtained, the target live-action image does not contain the first lasting target and does not introduce the second lasting target, so that the target live-action image can be subjected to VR image processing to obtain the VR display image without lasting pictures. And the VR display image of the target unmanned aerial vehicle is displayed through VR display equipment, so that a lasting target cannot appear, and the watching experience of a user is improved.
In the embodiment of the invention, a first live-action image obtained by shooting a target unmanned aerial vehicle in a target shooting area is obtained; detecting whether a first lasting target is contained in the first live-action image or not, wherein the first lasting target is determined based on a lasting structure of the target man-machine; if the first live-action image comprises a first lasting target, acquiring a second live-action image obtained by shooting a reference unmanned aerial vehicle in the target shooting area, wherein the reference unmanned aerial vehicle and the target unmanned aerial vehicle have different unmanned aerial vehicle structures; eliminating the first lasting target in the first live-action image based on the second live-action image to obtain a target live-action image; and after the target live-action image is subjected to VR image processing, obtaining a VR display image of the target unmanned aerial vehicle, and displaying the VR display image through VR display equipment. The first real image shot by the target unmanned aerial vehicle is subjected to wall penetrating detection, when the first real image contains a first wall penetrating target, a second real image shot by the reference unmanned aerial vehicle is acquired, the same wall penetrating structure cannot appear in the second real image by utilizing the difference of the structures of the reference unmanned aerial vehicle and the target unmanned aerial vehicle, the wall penetrating target in the first real image is eliminated by utilizing the second real image, the target real image without the wall penetrating target is obtained, the VR image processing is performed by utilizing the target real image without the wall penetrating target, and the VR display image without the wall penetrating is obtained for display, so that the wall penetrating lens can be avoided in a VR picture, and the VR viewing experience of a user is improved.
As shown in fig. 2, an embodiment of the present invention provides an unmanned aerial vehicle VR display device, which includes:
a first obtaining module 201, configured to obtain a first live-action image obtained by shooting a target unmanned aerial vehicle in a target shooting area;
the detection module 202 is configured to detect whether the first live-action image includes a first lasting target, where the first lasting target is determined based on a lasting structure of the target man-machine;
the second obtaining module 203 is configured to obtain a second live-action image obtained by shooting the reference unmanned aerial vehicle in the target shooting area if the first live-action image includes a first lasting target, where the reference unmanned aerial vehicle and the target unmanned aerial vehicle have different lasting structures;
the elimination module 204 is configured to eliminate the first lasting target in the first live-action image based on the second live-action image, so as to obtain a target live-action image;
and the processing module 205 is configured to obtain a VR display image of the target unmanned aerial vehicle after performing VR image processing on the target live-action image, and display the VR display image through a VR display device.
Optionally, the first live-action image includes color information, optical flow information, and depth information, and the detection module 202 is further configured to determine a foreground object in the first live-action image based on the color information; determining the distance information from the foreground target to the target unmanned aerial vehicle according to the depth information of the foreground target; determining the motion type of the foreground object based on the optical flow information of the foreground object; determining whether the foreground object is a first lasting object or not based on the distance information of the foreground object and the motion type of the foreground object; if the foreground object is a first lasting object, determining that the first live-action image comprises the first lasting object.
Optionally, the second obtaining module 203 is further configured to determine location information of the first live-action image, and match candidate live-action images in the same location according to the location information of the first live-action image; if the non-lasting real image exists in the candidate real images, selecting the non-lasting real image as a second real image; if the candidate live-action image does not have the non-lasting live-action image, determining a second live-action image in the candidate live-action image according to the first lasting objective.
Optionally, the candidate live-action image includes a second lasting target, and the second obtaining module 203 is further configured to determine a lasting type corresponding to the first lasting target according to a motion type of the foreground target, and determine a detection frame of the first lasting target in the first live-action image; determining a second live-action image in the candidate live-action image based on the lasting type of the first lasting target and the detection frame of the first lasting target, wherein the second lasting target of the second live-action image and the first lasting target of the first live-action image have different lasting types and detection frame positions.
Optionally, the second obtaining module 203 is further configured to determine a first lasting feature of the first live-action image based on a lasting type of the first lasting target and a detection frame of the first lasting target; determining a second lasting characteristic of the candidate live-action image based on the lasting type of the second lasting target and a detection frame of the second lasting target; determining the lasting similarity between the first live-action image and the candidate live-action image according to the characteristic similarity between the first lasting characteristic and the second lasting characteristic; and determining one candidate live-action image with the smallest lasting similarity as a second live-action image in the candidate live-action images.
Optionally, the second obtaining module 203 is further configured to calculate an intersection ratio between the detection frame of the first lasting target and the detection frame of the second lasting target; determining the detection frame of the second lasting target with the cross ratio of zero as a candidate detection frame; calculating the minimum distance between the detection frame of the first lasting target and the candidate detection frame and the center point distance; determining a weighted distance between the detection frame of the first lasting target and the candidate detection frame based on the minimum distance and the center point distance; selecting a candidate detection frame with the largest weighted distance from the candidate detection frames as a target detection frame, and determining the candidate live-action image corresponding to the target detection frame as a second live-action image.
Optionally, the elimination module 204 is further configured to map the area where the first lasting target is located to the second live-action image, so as to obtain a target replacement image for replacing the first lasting target; and in the first live-action image, replacing and eliminating the first lasting target through the target replacement image to obtain a target live-action image.
It should be noted that, the unmanned aerial vehicle VR display device provided by the embodiment of the present invention may be applied to devices such as a shooting device, a smart phone, a computer, and a server, which may perform unmanned aerial vehicle VR display.
The unmanned aerial vehicle VR display device provided by the embodiment of the invention can realize all the processes realized by the unmanned aerial vehicle VR display method in the method embodiment, and can achieve the same beneficial effects. In order to avoid repetition, a description thereof is omitted.
Referring to fig. 3, fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, as shown in fig. 3, including: a memory 302, a processor 301, and a computer program stored on the memory 302 and executable on the processor 301 for a method of VR display of a drone, wherein:
the processor 301 is configured to call a computer program stored in the memory 302, and perform the following steps:
Acquiring a first live-action image obtained by shooting a target unmanned aerial vehicle in a target shooting area;
detecting whether a first lasting target is contained in the first live-action image or not, wherein the first lasting target is determined based on a lasting structure of the target man-machine;
if the first live-action image comprises a first lasting target, acquiring a second live-action image obtained by shooting a reference unmanned aerial vehicle in the target shooting area, wherein the reference unmanned aerial vehicle and the target unmanned aerial vehicle have different unmanned aerial vehicle structures;
eliminating the first lasting target in the first live-action image based on the second live-action image to obtain a target live-action image;
and after the target live-action image is subjected to VR image processing, obtaining a VR display image of the target unmanned aerial vehicle, and displaying the VR display image through VR display equipment.
Optionally, the first live-action image includes color information, optical flow information, and depth information, and the detecting, performed by the processor 301, whether the first live-action image includes a lasting target includes:
determining a foreground object in the first live-action image based on the color information;
Determining the distance information from the foreground target to the target unmanned aerial vehicle according to the depth information of the foreground target;
determining the motion type of the foreground object based on the optical flow information of the foreground object;
determining whether the foreground object is a first lasting object or not based on the distance information of the foreground object and the motion type of the foreground object;
if the foreground object is a first lasting object, determining that the first live-action image comprises the first lasting object.
Optionally, the acquiring, performed by the processor 301, the second live-action image obtained by shooting, by the reference unmanned aerial vehicle, in the target shooting area includes:
determining the position information of the first live-action image, and matching candidate live-action images in the same position according to the position information of the first live-action image;
if the non-lasting real image exists in the candidate real images, selecting the non-lasting real image as a second real image;
if the candidate live-action image does not have the non-lasting live-action image, determining a second live-action image in the candidate live-action image according to the first lasting objective.
Optionally, the candidate live-action image includes a second lasting target, and the determining, by the processor 301, the second live-action image from the candidate live-action image according to the first lasting target includes:
Determining the lasting type corresponding to the first lasting target according to the motion type of the foreground target, and determining a detection frame of the first lasting target in the first live-action image;
determining a second live-action image in the candidate live-action image based on the lasting type of the first lasting target and the detection frame of the first lasting target, wherein the second lasting target of the second live-action image and the first lasting target of the first live-action image have different lasting types and detection frame positions.
Optionally, the determining, by the processor 301, the second live-action image from the candidate live-action images based on the lasting type of the first lasting target and the image position of the first lasting target includes:
determining a first lasting feature of the first live-action image based on the lasting type of the first lasting target and a detection frame of the first lasting target;
determining a second lasting characteristic of the candidate live-action image based on the lasting type of the second lasting target and a detection frame of the second lasting target;
determining the lasting similarity between the first live-action image and the candidate live-action image according to the characteristic similarity between the first lasting characteristic and the second lasting characteristic;
And determining one candidate live-action image with the smallest lasting similarity as a second live-action image in the candidate live-action images.
Optionally, the determining, by the processor 301, the second live-action image in the candidate live-action image based on the lasting type of the first lasting target and the detection frame of the first lasting target includes:
calculating the cross-over ratio between the detection frame of the first lasting target and the detection frame of the second lasting target;
determining the detection frame of the second lasting target with the cross ratio of zero as a candidate detection frame;
calculating the minimum distance between the detection frame of the first lasting target and the candidate detection frame and the center point distance;
determining a weighted distance between the detection frame of the first lasting target and the candidate detection frame based on the minimum distance and the center point distance;
selecting a candidate detection frame with the largest weighted distance from the candidate detection frames as a target detection frame, and determining the candidate live-action image corresponding to the target detection frame as a second live-action image.
Optionally, the removing, by the processor 301, the first lasting target in the first live-action image based on the second live-action image, to obtain a target live-action image includes:
Mapping the region where the first lasting target is located into the second live-action image to obtain a target replacement image for replacing the first lasting target;
and in the first live-action image, replacing and eliminating the first lasting target through the target replacement image to obtain a target live-action image.
It should be noted that, the electronic device provided by the embodiment of the invention can be applied to devices such as a smart phone, a computer, a server and the like which can perform the VR display method of the unmanned aerial vehicle.
The electronic equipment provided by the embodiment of the invention can realize each process realized by the VR display method of the unmanned aerial vehicle in the embodiment of the method, and can achieve the same beneficial effects. In order to avoid repetition, a description thereof is omitted.
The embodiment of the invention also provides a computer readable storage medium, and a computer program is stored on the computer readable storage medium, and when the computer program is executed by a processor, the method for displaying the VR of the unmanned aerial vehicle or the method for displaying the VR of the unmanned aerial vehicle at the application end provided by the embodiment of the invention is realized, and the same technical effects can be achieved, so that repetition is avoided, and no redundant description is provided here.
Those skilled in the art will appreciate that the processes implementing all or part of the methods of the above embodiments may be implemented by a computer program for instructing relevant hardware, and the program may be stored in a computer readable storage medium, and the program may include the processes of the embodiments of the methods as above when executed. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a random access Memory (RandomAccess Memory, RAM) or the like.
The foregoing disclosure is illustrative of the present invention and is not to be construed as limiting the scope of the invention, which is defined by the appended claims.
Claims (10)
1. An unmanned aerial vehicle VR display method, comprising the steps of:
acquiring a first live-action image obtained by shooting a target unmanned aerial vehicle in a target shooting area;
detecting whether a first lasting target is contained in the first live-action image or not, wherein the first lasting target is determined based on a lasting structure of the target man-machine;
if the first live-action image comprises a first lasting target, acquiring a second live-action image obtained by shooting a reference unmanned aerial vehicle in the target shooting area, wherein the reference unmanned aerial vehicle and the target unmanned aerial vehicle have different unmanned aerial vehicle structures;
eliminating the first lasting target in the first live-action image based on the second live-action image to obtain a target live-action image;
and after the target live-action image is subjected to VR image processing, obtaining a VR display image of the target unmanned aerial vehicle, and displaying the VR display image through VR display equipment.
2. The method of claim 1, wherein the first live-action image includes color information, optical flow information, and depth information, and the detecting whether the first live-action image includes a lasting target includes:
determining a foreground object in the first live-action image based on the color information;
determining the distance information from the foreground target to the target unmanned aerial vehicle according to the depth information of the foreground target;
determining the motion type of the foreground object based on the optical flow information of the foreground object;
determining whether the foreground object is a first lasting object or not based on the distance information of the foreground object and the motion type of the foreground object;
if the foreground object is a first lasting object, determining that the first live-action image comprises the first lasting object.
3. The method of claim 2, wherein the obtaining the second live-action image obtained by shooting the reference unmanned aerial vehicle in the target shooting area comprises:
determining the position information of the first live-action image, and matching candidate live-action images in the same position according to the position information of the first live-action image;
If the non-lasting real image exists in the candidate real images, selecting the non-lasting real image as a second real image;
if the candidate live-action image does not have the non-lasting live-action image, determining a second live-action image in the candidate live-action image according to the first lasting objective.
4. The method of claim 3, wherein the candidate live-action image includes a second lasting objective, and wherein determining the second live-action image from the candidate live-action image based on the first lasting objective includes:
determining the lasting type corresponding to the first lasting target according to the motion type of the foreground target, and determining a detection frame of the first lasting target in the first live-action image;
determining a second live-action image in the candidate live-action image based on the lasting type of the first lasting target and the detection frame of the first lasting target, wherein the second lasting target of the second live-action image and the first lasting target of the first live-action image have different lasting types and detection frame positions.
5. The method of claim 4, wherein the determining a second live-action image from the candidate live-action images based on the lasting type of the first lasting destination and the image location of the first lasting destination comprises:
Determining a first lasting feature of the first live-action image based on the lasting type of the first lasting target and a detection frame of the first lasting target;
determining a second lasting characteristic of the candidate live-action image based on the lasting type of the second lasting target and a detection frame of the second lasting target;
determining the lasting similarity between the first live-action image and the candidate live-action image according to the characteristic similarity between the first lasting characteristic and the second lasting characteristic;
and determining one candidate live-action image with the smallest lasting similarity as a second live-action image in the candidate live-action images.
6. The method of claim 4, wherein the determining a second live-action image from the candidate live-action images based on the lasting type of the first lasting destination and the detection box of the first lasting destination comprises:
calculating the cross-over ratio between the detection frame of the first lasting target and the detection frame of the second lasting target;
determining the detection frame of the second lasting target with the cross ratio of zero as a candidate detection frame;
calculating the minimum distance between the detection frame of the first lasting target and the candidate detection frame and the center point distance;
Determining a weighted distance between the detection frame of the first lasting target and the candidate detection frame based on the minimum distance and the center point distance;
selecting a candidate detection frame with the largest weighted distance from the candidate detection frames as a target detection frame, and determining the candidate live-action image corresponding to the target detection frame as a second live-action image.
7. The method of any of claims 1-6, wherein the eliminating the first lasting target in the first live-action image based on the second live-action image to obtain a target live-action image comprises:
mapping the region where the first lasting target is located into the second live-action image to obtain a target replacement image for replacing the first lasting target;
and in the first live-action image, replacing and eliminating the first lasting target through the target replacement image to obtain a target live-action image.
8. Unmanned aerial vehicle VR display device, its characterized in that, unmanned aerial vehicle VR display device includes:
the first acquisition module is used for acquiring a first live-action image obtained by shooting the target unmanned aerial vehicle in the target shooting area;
The detection module is used for detecting whether the first live-action image comprises a first lasting target or not, and the first lasting target is determined based on a lasting structure of the target man-machine;
the second acquisition module is used for acquiring a second real image obtained by shooting the reference unmanned aerial vehicle in the target shooting area if the first real image contains a first lasting target, wherein the reference unmanned aerial vehicle and the target unmanned aerial vehicle have different lasting structures;
the elimination module is used for eliminating the first lasting target in the first live-action image based on the second live-action image to obtain a target live-action image;
and the processing module is used for obtaining the VR display image of the target unmanned aerial vehicle after the VR image processing is carried out on the target live-action image, and displaying the VR display image through VR display equipment.
9. An electronic device, comprising: memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps in the unmanned aerial vehicle VR display method of any one of claims 1 to 7 when the computer program is executed.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps in the unmanned aerial vehicle VR display method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311374147.2A CN117456134A (en) | 2023-10-23 | 2023-10-23 | Unmanned aerial vehicle VR display method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311374147.2A CN117456134A (en) | 2023-10-23 | 2023-10-23 | Unmanned aerial vehicle VR display method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117456134A true CN117456134A (en) | 2024-01-26 |
Family
ID=89594064
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311374147.2A Pending CN117456134A (en) | 2023-10-23 | 2023-10-23 | Unmanned aerial vehicle VR display method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117456134A (en) |
-
2023
- 2023-10-23 CN CN202311374147.2A patent/CN117456134A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109345556B (en) | Neural network foreground separation for mixed reality | |
| EP3576017B1 (en) | Method and system for determining pose of object in image, and storage medium | |
| US11443454B2 (en) | Method for estimating the pose of a camera in the frame of reference of a three-dimensional scene, device, augmented reality system and computer program therefor | |
| GB2583676A (en) | Augmenting detected regions in image or video data | |
| WO2022156640A1 (en) | Gaze correction method and apparatus for image, electronic device, computer-readable storage medium, and computer program product | |
| EP2874097A2 (en) | Automatic scene parsing | |
| WO2023093217A1 (en) | Data labeling method and apparatus, and computer device, storage medium and program | |
| WO2023056544A1 (en) | Object and camera localization system and localization method for mapping of the real world | |
| CN112492388A (en) | Video processing method, device, equipment and storage medium | |
| KR100560464B1 (en) | Multi-view display system with viewpoint adaptation | |
| CN112207821B (en) | Target searching method of visual robot and robot | |
| CN110941996A (en) | Target and track augmented reality method and system based on generation of countermeasure network | |
| US20230214967A1 (en) | Image inpainting based on multiple image transformations | |
| CN112562056A (en) | Control method, device, medium and equipment for virtual light in virtual studio | |
| CN114511661A (en) | Image rendering method and device, electronic equipment and storage medium | |
| CN104700384B (en) | Display systems and methods of exhibiting based on augmented reality | |
| CN107274477B (en) | Background modeling method based on three-dimensional space surface layer | |
| Ahn et al. | Implement of an automated unmanned recording system for tracking objects on mobile phones by image processing method | |
| JP2023038203A (en) | Display device, display method, program and image projection system | |
| CN113988957B (en) | Automatic image scoring method and system based on element recognition | |
| CN117456134A (en) | Unmanned aerial vehicle VR display method and device, electronic equipment and storage medium | |
| WO2023086398A1 (en) | 3d rendering networks based on refractive neural radiance fields | |
| CA3172195A1 (en) | Object and camera localization system and localization method for mapping of the real world | |
| CN111258408B (en) | Object boundary determining method and device for man-machine interaction | |
| US20230005213A1 (en) | Imaging apparatus, imaging method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |