CN108597036B - Virtual reality environment danger sensing method and device - Google Patents
Virtual reality environment danger sensing method and device Download PDFInfo
- Publication number
- CN108597036B CN108597036B CN201810412419.6A CN201810412419A CN108597036B CN 108597036 B CN108597036 B CN 108597036B CN 201810412419 A CN201810412419 A CN 201810412419A CN 108597036 B CN108597036 B CN 108597036B
- Authority
- CN
- China
- Prior art keywords
- frame
- current frame
- world coordinates
- feature points
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a virtual reality environment danger sensing method and device. The method comprises the following steps: extracting feature points in each frame of image collected by the VR camera in real time; for the second frame and the later frame, matching the characteristic points in the current frame and the previous frame, and calculating a motion vector when the VR camera collects the current frame image relative to the previous frame image according to each pair of matching points; according to the calculated motion vector, calculating the world coordinates of each feature point successfully matched on the current frame image and the previous frame image; detecting whether an attention object exists in the current frame image or not according to all feature points of the world coordinates calculated in the current frame image, if so, calculating the distance between the attention object and the current VR camera, and if the Euclidean distance is smaller than a preset threshold value, sending danger reminding to a user. The invention can sense danger and send out a prompt in the VR environment.
Description
Technical Field
The present invention relates to the field of VR (Virtual Reality) technology. In particular to a VR environmental risk perception method and a VR environmental risk perception device.
Background
The current situation in the use process of the VR product is analyzed to know that: the user does not know whether the self motion can cause danger because the real environment that the user is located can not be perceived when using the VR product, so the user dares to move freely with great care in the motion process.
With the continuous improvement of the performance of a computing chip and the continuous optimization and improvement of a rendering algorithm, the reality of a VR product picture is continuously enhanced, and in such a background, a user cannot experience the immersion feeling all over the body. There is an urgent need for a method that can sense the real environment and prompt.
In the current VR application, a real picture is simply shot and mapped through a camera, and the real environment itself is lack of accurate perception.
Disclosure of Invention
The invention provides a VR environment danger sensing method and a VR environment danger sensing device, which are used for realizing danger sensing and reminding in a VR environment.
The technical scheme of the invention is realized as follows:
a virtual reality, VR, environmental hazard awareness method, the method comprising:
extracting feature points in each frame of image acquired by the VR camera in real time;
for each frame image of a second frame and a later frame acquired by the VR camera, matching the feature points in the current frame image and the feature points in the previous frame image, and calculating a motion vector when the VR camera acquires the current frame image relative to the previous frame image according to the positions of each pair of matching points in the two frame images; according to the calculated motion vector, calculating the world coordinates of each feature point successfully matched on the current frame image and the previous frame image;
detecting whether an attention object exists in the current frame image according to all feature points of which world coordinates are calculated in the current frame image, if so, calculating the current world coordinates of the VR camera according to the initial world coordinates of the VR camera and the motion vector of the VR camera relative to the previous frame image when acquiring each frame image, calculating the distance between the attention object and the current VR camera according to the world coordinates of the attention object in the current frame and the current world coordinates of the VR camera, and if the distance is smaller than a preset threshold value, sending danger reminding to a user.
After the calculating the world coordinates of each feature point successfully matched on the current frame image and the previous frame image, the method further comprises:
the world coordinates of the calculated feature points are put into a local map description library, and meanwhile, frame identifications corresponding to the feature points are recorded in the local map description library;
after matching the feature points in the current frame image and the previous frame image, and calculating the motion vector when the VR camera collects the current frame image relative to the motion vector when the VR camera collects the previous frame image according to the positions of each pair of matching points in the two frame images, the method further comprises the following steps:
judging whether the current frame meets one of the following key frame judgment conditions:
first, the total number of key frames in the key frame set is less than a first threshold value;
secondly, the number of the feature points of the current frame successfully matched with the previous frame image/the total number of the feature points extracted from the current frame is less than a second threshold value;
if yes, determining the current frame as a key frame, putting the frame identification of the current frame into a key frame set, and then executing the action of calculating a motion vector when the VR camera collects the current frame image relative to the motion vector when the VR camera collects the previous frame image according to the positions of each pair of matching points in the two frame images; otherwise, determining the current frame as a non-key frame, discarding the current frame, and directly switching to the next frame.
When the current frame is determined to be the key frame, the calculating of the world coordinates of each feature point successfully matched on the current frame image and the previous frame image further comprises:
respectively matching all feature points of the current frame with the calculated world coordinates with all feature points of each key frame in the key frame set, if the matching rate exceeds a preset third threshold value, considering that the current frame is redundant, not adding the current frame into the key frame set, not updating a local map description library with the feature points of the current frame with the calculated world coordinates, and switching to the next frame, wherein if the world coordinates of the two feature points are the same, the two feature points are matched.
After the calculating the world coordinates of each feature point successfully matched on the current frame image and the previous frame image, the method further comprises:
forming a bag-of-words BOW vector by all feature points of the current frame with the calculated world coordinates, respectively matching the BOW vector of the current frame with the BOW vector of each key frame in the key frame set, if the BOW vector of the current frame is successfully matched with one key frame, considering that the current frame is successfully relocated, namely considering that the position where the VR camera collects the current frame is the same as the position where the key frame is successfully collected, discarding the feature points originally extracted from the current frame, finding the world coordinates of all the feature points corresponding to the key frame which is successfully matched in a local map description library, adding the frame identifier of the current frame in a frame identifier list of the world coordinates of each found feature point, and not putting the current frame in the key frame set.
The key frame determination condition further includes:
the time consumed by the last repositioning process of the current frame is longer than a preset fifth threshold.
The method further comprises:
when a preset closed loop detection period comes, respectively calculating the distance between the BOW vector of the latest key frame and the BOW vector of each key frame associated with the latest key frame for the key frames in the key frame set, and taking the associated key frame with the minimum distance as a candidate closed loop frame of the latest key frame, wherein when the world coordinates of at least one pair of feature points in the two key frames are the same, the two key frames are considered to be associated;
calculating a motion vector when the VR camera acquires the latest key frame relative to the acquisition of the candidate loopback frame according to the world coordinates of the feature points on the latest key frame and the candidate loopback frame, matching the feature points of the two frames by taking the candidate loopback frame as the previous frame of the latest key frame, recalculating the world coordinates of all the feature points which are successfully matched according to the calculated motion vector, updating the world coordinates of all the feature points of the latest key frame in a local map description library according to the calculated world coordinates of all the feature points which are successfully matched, and adding frame identifications corresponding to all the feature points which are successfully matched in the local map description library: frame identification of the latest key frame.
The extraction of the feature points in the image is as follows: and extracting the characteristic FAST characteristic points in the image by an accelerated segmentation test.
The sending of the danger reminder to the user comprises:
copying preset single-channel reminding audio data into a left channel and a right channel, and respectively carrying out FFT (fast Fourier transform) on the reminding audio data of the two channels to obtain frequency domain reminding audio data of the left channel and the right channel;
determining a sounding position of the reminding audio according to the world coordinates of the attention object and the world coordinates of the VR camera, wherein the sounding position is represented by HRTF standard spatial position parameters, the sounding position of the reminding audio is located on a straight line connecting line between the VR camera and the attention object, and the distance between the sounding position of the reminding audio and the VR camera is preset;
reading corresponding HRTF conversion data from an HRTF standard database according to the sounding position of the reminding audio, carrying out FFT conversion on the HRTF conversion data to obtain frequency domain HRTF conversion data, multiplying the frequency domain reminding audio data of the left and right channels with the frequency domain HRTF conversion data respectively to obtain frequency domain reminding space audio data of the left and right channels, carrying out IFFT conversion on the frequency domain reminding space audio data of the left and right channels respectively to obtain time domain reminding space audio data of the left and right channels, and playing the time domain reminding space audio data of the left and right channels to a user through the left and right channels respectively.
The sending of the danger reminder to the user comprises:
superposing a contour formed by all feature points of known world coordinates of the object of interest on the 3-dimensional VR image in which the object of interest is detected; alternatively, the first and second electrodes may be,
displaying prompting text information on the 2-dimensional image in an overlapping mode, wherein the prompting text information comprises: distance information of the VR camera from the object of interest; alternatively, the first and second electrodes may be,
and displaying the 3-dimensional VR image without the background information, namely deleting the background image except the outline formed by all feature points with known world coordinates on the VR image with the detected attention so as to display only the VR camera, the user and the attention, wherein when the attention is displayed, the color of the attention is displayed in a progressive manner according to the distance between the attention and the VR camera from far to near.
A virtual reality, VR, environmental hazard awareness apparatus, the apparatus comprising:
the characteristic extraction and calculation module is used for extracting characteristic points in each frame of image acquired by the VR camera in real time; for each frame image of a second frame and a later frame acquired by the VR camera, matching the feature points in the current frame image and the feature points in the previous frame image, and calculating a motion vector when the VR camera acquires the current frame image relative to the previous frame image according to the positions of each pair of matching points in the two frame images; according to the calculated motion vector, calculating the world coordinates of each feature point successfully matched on the current frame image and the previous frame image;
and the danger detection module is used for detecting whether an attention object exists in the current frame image according to all the feature points of which the world coordinates are calculated in the current frame image, if so, calculating the current world coordinates of the VR camera according to the initial world coordinates of the VR camera and the motion vector of the VR camera relative to the previous frame image when acquiring each frame image, calculating the distance between the attention object and the current VR camera according to the world coordinates of the attention object in the current frame and the current world coordinates of the VR camera, and if the distance is smaller than a preset threshold value, sending a danger prompt to a user.
The feature extraction and calculation module is further used for calculating the world coordinates of each feature point successfully matched on the current frame image and the previous frame image,
the world coordinates of the calculated feature points are put into a local map description library, and meanwhile, frame identifications corresponding to the feature points are recorded in the local map description library;
and after the characteristic extraction and calculation module matches the characteristic points in the current frame image and the previous frame image, the characteristic extraction and calculation module further uses the motion vector when the VR camera collects the current frame image relative to the motion vector when the previous frame image is collected according to the positions of each pair of matching points in the two frame images,
judging whether the current frame meets one of the following key frame judgment conditions:
first, the total number of key frames in the key frame set is less than a first threshold value;
secondly, the number of the feature points of the current frame successfully matched with the previous frame image/the total number of the feature points extracted from the current frame is less than a second threshold value;
if yes, determining the current frame as a key frame, putting the frame identification of the current frame into a key frame set, and then executing the action of calculating a motion vector when the VR camera collects the current frame image relative to the motion vector when the VR camera collects the previous frame image according to the positions of each pair of matching points in the two frame images; otherwise, determining the current frame as a non-key frame, discarding the current frame, and directly switching to the next frame.
When the current frame is determined to be the key frame, the characteristic extraction and calculation module is further used after calculating the world coordinates of each characteristic point successfully matched on the current frame image and the previous frame image,
respectively matching all feature points of the current frame with the calculated world coordinates with all feature points of each key frame in the key frame set, if the matching rate exceeds a preset third threshold value, considering that the current frame is redundant, not adding the current frame into the key frame set, not updating a local map description library with the feature points of the current frame with the calculated world coordinates, and switching to the next frame, wherein if the world coordinates of the two feature points are the same, the two feature points are matched.
The feature extraction and calculation module is further used for calculating the world coordinates of each feature point successfully matched on the current frame image and the previous frame image,
forming a bag-of-words BOW vector by all feature points of the current frame with the calculated world coordinates, respectively matching the BOW vector of the current frame with the BOW vector of each key frame in the key frame set, if the BOW vector of the current frame is successfully matched with one key frame, considering that the current frame is successfully relocated, namely considering that the position where the VR camera collects the current frame is the same as the position where the key frame is successfully collected, discarding the feature points originally extracted from the current frame, finding the world coordinates of all the feature points corresponding to the key frame which is successfully matched in a local map description library, adding the frame identifier of the current frame in a frame identifier list of the world coordinates of each found feature point, and not putting the current frame in the key frame set.
The key frame determination condition that the feature extraction and calculation module determines whether the current frame satisfies further includes:
the time consumed by the last repositioning process of the current frame is longer than a preset fifth threshold.
The feature extraction and computation module is further configured to,
when a preset closed loop detection period comes, respectively calculating the distance between the BOW vector of the latest key frame and the BOW vector of each key frame associated with the latest key frame for the key frames in the key frame set, and taking the associated key frame with the minimum distance as a candidate closed loop frame of the latest key frame, wherein when the world coordinates of at least one pair of feature points in the two key frames are the same, the two key frames are considered to be associated;
calculating a motion vector when the VR camera acquires the latest key frame relative to the acquisition of the candidate loopback frame according to the world coordinates of the feature points on the latest key frame and the candidate loopback frame, matching the feature points of the two frames by taking the candidate loopback frame as the previous frame of the latest key frame, recalculating the world coordinates of all the feature points which are successfully matched according to the calculated motion vector, updating the world coordinates of all the feature points of the latest key frame in a local map description library according to the calculated world coordinates of all the feature points which are successfully matched, and adding frame identifications corresponding to all the feature points which are successfully matched in the local map description library: frame identification of the latest key frame.
The feature extraction and calculation module extracts feature points in the image as follows: and extracting the characteristic FAST characteristic points in the image by an accelerated segmentation test.
The danger detection module sends out danger to remind the user including:
copying preset single-channel reminding audio data into a left channel and a right channel, and respectively carrying out FFT (fast Fourier transform) on the reminding audio data of the two channels to obtain frequency domain reminding audio data of the left channel and the right channel;
determining a sounding position of the reminding audio according to the world coordinates of the attention object and the world coordinates of the VR camera, wherein the sounding position is represented by HRTF standard spatial position parameters, the sounding position of the reminding audio is located on a straight line connecting line between the VR camera and the attention object, and the distance between the sounding position of the reminding audio and the VR camera is preset;
reading corresponding HRTF conversion data from an HRTF standard database according to the sounding position of the reminding audio, carrying out FFT conversion on the HRTF conversion data to obtain frequency domain HRTF conversion data, multiplying the frequency domain reminding audio data of the left and right channels with the frequency domain HRTF conversion data respectively to obtain frequency domain reminding space audio data of the left and right channels, carrying out IFFT conversion on the frequency domain reminding space audio data of the left and right channels respectively to obtain time domain reminding space audio data of the left and right channels, and playing the time domain reminding space audio data of the left and right channels to a user through the left and right channels respectively.
The danger detection module sends out danger to remind the user including:
superposing a contour formed by all feature points of known world coordinates of the object of interest on the 3-dimensional VR image in which the object of interest is detected; alternatively, the first and second electrodes may be,
displaying prompting text information on the 2-dimensional image in an overlapping mode, wherein the prompting text information comprises: distance information of the VR camera from the object of interest; alternatively, the first and second electrodes may be,
and displaying the 3-dimensional VR image without the background information, namely deleting the background image except the outline formed by all feature points with known world coordinates on the VR image with the detected attention so as to display only the VR camera, the user and the attention, wherein when the attention is displayed, the color of the attention is displayed in a progressive manner according to the distance between the attention and the VR camera from far to near.
The invention realizes danger perception and reminding in the VR environment.
Drawings
Fig. 1 is a flowchart of a VR environmental risk sensing method according to an embodiment of the present invention;
FIG. 2 is a flowchart of a VR environmental risk sensing method according to another embodiment of the present invention;
FIG. 3 is an exemplary diagram of an application of the present invention for performing a danger-reminding using a 3-dimensional image;
fig. 4 is a schematic structural diagram of a VR environmental risk sensing device according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
Fig. 1 is a flowchart of a VR environmental risk sensing method according to an embodiment of the present invention, which includes the following specific steps:
step 101: and extracting the characteristic points in each frame of image acquired by the VR camera in real time.
Step 102: and for each frame image of the second frame and the subsequent frame acquired by the VR camera, matching the feature points in the current frame image and the previous frame image.
Step 103: and calculating a motion vector when the VR camera collects the current frame image relative to the previous frame image according to the positions of each pair of matching points in the two frame images.
Step 104: and according to the calculated motion vector, calculating the world coordinates of each feature point successfully matched on the current frame image and the previous frame image.
Step 105: detecting whether an attention object exists in the current frame image according to all feature points of which world coordinates are calculated in the current frame image, if so, calculating the current world coordinates of the VR camera according to the initial world coordinates of the VR camera and the motion vector of the VR camera relative to the previous frame image when acquiring each frame image, calculating the distance between the attention object and the current VR camera according to the world coordinates of the attention object in the current frame and the current world coordinates of the VR camera, and if the distance is smaller than a preset threshold value, sending danger reminding to a user.
Fig. 2 is a flowchart of a VR environmental risk sensing method according to another embodiment of the present invention, which includes the following specific steps:
step 201: and the VR camera collects images in real time.
Step 202: for each frame of image collected by the VR camera, FAST (Features from accessed Segment Test, feature acquisition by Accelerated segmentation Test) feature points in the image are extracted.
FAST feature points refer to pixels in an image that have a gray value that is brighter or darker than its surroundings. For example: presetting a first threshold, comparing the gray value of each point in the image with the gray values of a preset number of surrounding preset points, and if the absolute value of the gray difference between the gray value of each point and the gray value of each surrounding point is greater than the first threshold, determining that the current point is a FAST characteristic point.
Step 203: and when a second frame image is acquired, matching the FAST characteristic points in the second frame image and the FAST characteristic points in the first frame image, and calculating a translation value and a rotation value of the VR camera when acquiring the second frame image relative to the first frame image according to the positions of each pair of matching points in the two frame images and the epipolar geometry principle.
The matching mode of the FAST characteristic points belongs to the mature technology, and the invention is not repeated.
Step 204: and calculating the world coordinate of each FAST characteristic point successfully matched in a world coordinate system according to the calculated translation value and rotation value by the principle of triangulation, and calculating and storing the initial world coordinate of the VR camera.
The origin of the world coordinate system is actually the initial optical center of the VR camera, the X, Y axis is the axis parallel to the horizontal and vertical sides of the VR camera lens, respectively, and the Z axis is the axis perpendicular to the VR camera lens.
Step 205: putting the world coordinates of the successfully matched FAST feature points on the images of the first frame and the second frame into a local map description library, and simultaneously recording the frame identification corresponding to each successfully matched FAST feature point in the local map description library: frame identifications of the first frame and the second frame.
For example: m FAST feature points are extracted from a first frame image, n FAST feature points are extracted from a second frame image, the matched FAST feature points on the first frame image and the second frame image have p (p is less than or equal to m, p is less than or equal to n) pairs, and the three-dimensional world coordinates of each pair of mutually matched FAST feature points are the same, so that the three-dimensional world coordinates of the p FAST feature points successfully matched are put into a local map description library, wherein the p FAST feature points successfully matched correspond to the first frame and the second frame at the same time.
Step 206: and for each image of the third frame and the later frame, matching the FAST characteristic points in the current frame image and the FAST characteristic points in the previous frame image, and calculating a translation value and a rotation value when the VR camera collects the current frame image relative to the previous frame image according to the positions of each pair of matching points in the two frame images by using an epipolar geometry principle.
Step 207: and calculating the world coordinates of each FAST feature point which is successfully matched on the current frame image and the previous frame image but does not exist in the local map description library by means of triangulation according to the calculated translation value and rotation value.
In the FAST feature points successfully matched between the current frame image and the previous frame image, the world coordinates of some FAST feature points may be calculated when the previous frame (or a frame further ahead) image is processed, that is, the world coordinates of the FAST feature points are already stored in the local map description library, and the world coordinates of the FAST feature points are not calculated repeatedly.
Step 208: the world coordinates of the FAST feature points obtained through calculation are put into a local map description library, and meanwhile, frame identifications corresponding to the FAST feature points are recorded in the local map description library: frame identifications of the current frame and the previous frame.
For each FAST feature point which is successfully matched between the current frame image and the previous frame image and already exists in the local map description library, a current frame identifier needs to be added to a frame identifier list corresponding to each FAST feature point in the local map description library.
Step 209: according to all FAST characteristic points of world coordinates calculated in a current frame (including a first frame and a later frame) image stored in a local map description library, whether an object of interest (such as an edge or an obstacle) exists in the current frame image or not is detected, if yes, according to an initial world coordinate of a VR camera and a translation value and a rotation value of the VR camera relative to the previous frame image when the VR camera collects each frame image, the current world coordinate of the VR camera is calculated, according to the world coordinate of the object of interest in the current frame and the current world coordinate of the VR camera, the Euclidean distance between the object of interest and the current VR camera is calculated, and if the Euclidean distance is smaller than a preset threshold value, it is determined that a prompt needs to be sent to a user.
When calculating the euclidean distance between the attention object and the current VR camera, the euclidean distance between the closest point of the attention object to the VR camera and the VR camera may be used as a criterion.
Among these, the concerns are: edges or obstacles can be detected by conventional image detection methods such as edge detection or obstacle detection.
In practical applications, consider: as images acquired by VR cameras are more and more, the total number of extracted FAST feature points will be more and more, and thus the number of world coordinates of FAST feature points stored in a local map description library will be more and more, and in order to save storage space, the following solutions are provided:
judging whether the current frame meets one of the following two conditions, if so, determining the current frame as a key frame, putting the frame identifier of the current frame into a key frame set, and continuously executing subsequent processing related to the current frame; otherwise, determining the current frame as a non-key frame, discarding the current frame, and directly switching to the next frame:
first, the total number of key frames in the key frame set is less than a first threshold value;
secondly, the number of FAST feature points successfully matched between the current frame and the previous frame image/the total number of FAST feature points extracted by the current frame is less than a second threshold value.
In order to save the storage space further, the invention further provides the following optimization scheme:
after the current frame is determined as the key frame, the method further comprises: and respectively matching the FAST characteristic points of the current frame with the FAST characteristic points of each key frame in the key frame set according to the world coordinates of the FAST characteristic points, if the matching rate exceeds a preset third threshold value, considering that the current frame is redundant, not performing subsequent processing on the current frame (namely, not adding the current frame into the key frame set, and not updating a local map description library according to the FAST characteristic points of the current frame), and directly transferring to the next frame. Here, the two FAST feature points match means that the world coordinates of the two FAST feature points are the same.
In addition, considering the estimation errors of translation and rotation, the invention proposes the following relocation procedure:
forming a BOW (Bag-of-Words) vector by all FAST feature points of the current frame, respectively matching the BOW vector of the current frame with the BOW vector of each key frame in the key frame set, if the matching with a key frame is successful, considering that the repositioning of the current frame is successful, namely considering that the position of a VR (virtual reality) camera when acquiring the current frame is the same as the position of the key frame which is successfully acquired, discarding the FAST feature points which are originally extracted from the current frame, directly taking all FAST feature points in the key frame which is successfully matched as the world feature points of the current frame, namely finding the world coordinates of all FAST feature points corresponding to the key frame which is successfully matched in a local map description library, and adding the frame identification of the current frame in the frame identification list of the world coordinates of each FAST feature point which is found, meanwhile, as the current frame is completely matched with the successfully matched key frame, the current frame is not put into the key frame set; if the current frame is not successfully matched with any key frame, no special processing is carried out.
Here, the two BOW vectors match means that the distance between the two BOW vectors is smaller than a preset fourth threshold.
In addition, when determining whether the current frame is a key frame, the following condition may also be adopted:
the time duration of the last repositioning process of the current frame is greater than a preset fifth threshold.
In order to further eliminate the estimation error of translation and rotation, the invention further proposes the following scheme:
presetting a closed loop detection period, when the closed loop detection period comes, respectively calculating the distance between the BOW vector of the latest key frame and the BOW vector of each key frame associated with the latest key frame for the key frames in the key frame set, and taking the associated key frame with the smallest distance as a candidate loop frame of the latest key frame, wherein when at least one pair of matched FAST characteristic points (namely the world coordinates of the pair of FAST characteristic points are the same) in the two key frames, the two key frames are considered to be associated;
according to the world coordinates of the FAST feature points on the latest key frame and the candidate loop frames, calculating a translation value and a rotation value when the VR camera acquires the latest key frame relative to the acquired candidate loop frames, taking the candidate loop frames as the previous frame of the latest key frame, matching the FAST feature points of the two frames, recalculating the world coordinates of all FAST feature points which are successfully matched according to the calculated translation value and rotation value, updating the world coordinate information of all FAST feature points of the latest key frame in a local map description library by the calculated world coordinate information of all FAST feature points which are successfully matched, and adding frame identifications corresponding to all FAST feature points which are successfully matched in the local map description library: frame identification of the latest key frame.
In the invention, the reminding to the user can adopt an audio mode or/and an image mode.
When the audio mode is adopted, the specific scheme can be as follows:
step 01: the preset single-channel reminding audio data is copied into a left channel and a right channel, and Fast Fourier Transform (FFT) conversion is respectively carried out on the reminding audio data of the two channels to obtain frequency domain reminding audio data of the left channel and the right channel.
Step 02: determining the sounding position of the reminding audio according to the world coordinates of the attention object (which can be represented by the world coordinates of the point closest to the VR camera in the attention object) and the world coordinates of the VR camera, wherein the sounding position is represented by HRTF (Head Related Transfer Function) standard spatial position parameters.
The sounding position of the reminding audio is located on a straight line connecting line between the VR camera and an attention object (which can be represented by a point closest to the VR camera), and the distance between the sounding position of the reminding audio and the VR camera can be preset.
Step 03: reading corresponding HRTF conversion data from an HRTF standard database according to the sounding position of the reminding audio, carrying out FFT conversion on the HRTF conversion data to obtain frequency domain HRTF conversion data, multiplying the frequency domain reminding audio data of the left and right channels with the frequency domain HRTF conversion data respectively to obtain frequency domain reminding space audio data of the left and right channels, carrying out IFFT conversion on the frequency domain reminding space audio data of the left and right channels respectively to obtain time domain reminding space audio data of the left and right channels, and playing the time domain reminding space audio data of the left and right channels to a user through the left and right channels respectively.
In addition, the user can be reminded by vibrating the motor.
When the image mode is adopted for reminding, the specific scheme can be as follows:
an outline made up of all FAST feature points showing known world coordinates of the object of interest is superimposed on a VR (3D) image in which the object of interest is detected. As shown in fig. 3, the left image is an original VR image without image reminding, the right image is an image with image reminding, and positions such as corners of walls and the like are shown in outline in the right image;
or, displaying prompting text information in an overlapping manner on the 2D image in which the attention object is detected, wherein the text information can be distance information of the VR camera from the attention object;
or, deleting the background image excluding the detected VR image of the background information, namely deleting the background image except the outline formed by the FAST characteristic points of all known world coordinates on the detected VR image of the attention object, so as to display only the VR camera, the user and the attention object, wherein when the attention object is displayed, the color of the attention object can be displayed in a gradually-changed mode according to the distance between the attention object and the VR camera from far to near, namely, the farther the attention object is from the VR camera, the higher the transparency of the attention object is, and vice versa, the lower the transparency is.
Fig. 4 is a schematic structural diagram of a VR environmental risk sensing apparatus according to an embodiment of the present invention, where the apparatus mainly includes: a feature extraction and calculation module 41 and a risk detection module 42, wherein:
the feature extraction and calculation module 41 is configured to extract, for each frame of image acquired by the VR camera in real time, a FAST feature point in the image; for each frame image of a second frame and a later frame acquired by the VR camera, matching the FAST characteristic points in the current frame image and the FAST characteristic points in the previous frame image, and calculating a translation value and a rotation value when the VR camera acquires the current frame image relative to the previous frame image according to the positions of each pair of matching points in the two frame images; and according to the calculated translation value and rotation value, calculating the world coordinates of each FAST feature point successfully matched on the current frame image and the previous frame image.
And a danger detection module 42, configured to detect whether an attention object exists in the current frame image according to the world coordinates of the FAST feature point in the current frame image calculated by the feature extraction and calculation module 41, if yes, calculate the current world coordinates of the VR camera according to the initial world coordinates of the VR camera and a translation value and a rotation value of the VR camera when acquiring each frame image relative to the previous frame image, calculate an euclidean distance between the attention object and the current VR camera according to the world coordinates of the attention object in the current frame and the current world coordinates of the VR camera, and if the euclidean distance is smaller than a preset threshold, send a danger prompt to the user.
In practical application, after the feature extraction and calculation module 41 calculates the world coordinates of each FAST feature point successfully matched on the current frame image and the previous frame image, the calculated world coordinates of the FAST feature points are further used for placing the world coordinates of the FAST feature points into the local map description library, and meanwhile, the frame identifications corresponding to each FAST feature point are recorded in the local map description library;
after the feature extraction and calculation module 41 matches the FAST feature points in the current frame image and the previous frame image, the feature extraction and calculation module is further configured to determine whether the current frame meets one of the following key frame determination conditions before calculating a translation value and a rotation value when the VR camera collects the current frame image relative to the previous frame image according to the positions of each pair of matching points in the two frame images:
first, the total number of key frames in the key frame set is less than a first threshold value;
secondly, the number of FAST characteristic points successfully matched between the current frame and the previous frame image/the total number of FAST characteristic points extracted by the current frame is less than a second threshold value;
if yes, determining the current frame as a key frame, putting the frame identification of the current frame into a key frame set, and then executing the action of calculating a translation value and a rotation value when the VR camera collects the current frame image relative to the previous frame image according to the positions of each pair of matching points in the two frame images; otherwise, determining the current frame as a non-key frame, discarding the current frame, and directly switching to the next frame.
In practical applications, when the current frame is determined as a key frame, the feature extraction and calculation module 41 is further configured to calculate world coordinates of each FAST feature point that is successfully matched between the current frame image and the previous frame image, and then respectively match all FAST feature points of the current frame whose world coordinates are calculated with all FAST feature points of each key frame in the key frame set whose world coordinates are calculated, if the matching rate exceeds a preset third threshold, the current frame is considered to be redundant, the current frame is not added to the key frame set, and the local map description library is not updated with the FAST feature points of the current frame whose world coordinates are calculated, and the next frame is switched to, where if the world coordinates of the two FAST feature points are the same, the two FAST feature points are matched.
In practical application, the feature extraction and calculation module 41 is further configured to calculate the world coordinates of each FAST feature point successfully matched on the current frame image and the previous frame image, configure all FAST feature points of the current frame image with the calculated world coordinates into a bag of words BOW vector, match the BOW vector of the current frame with the BOW vector of each key frame in the key frame set, if the match with a key frame is successful, consider that the repositioning of the current frame is successful, i.e., consider that the position where the VR camera collects the current frame is the same as the position where the VR camera collects the key frame successfully matched, discard the FAST feature points originally extracted from the current frame, find the world coordinates of all FAST feature points corresponding to the key frame successfully matched in the local map description library, add the frame identifier of the current frame in the frame identifier list of the world coordinates of each found FAST feature point, and the current frame is not put into the key frame set.
In practical applications, the key frame determination condition that the feature extraction and calculation module 41 determines whether the current frame satisfies further includes: the time consumed by the last repositioning process of the current frame is longer than a preset fifth threshold.
In practical applications, the feature extraction and calculation module 41 is further configured to, when a preset closed-loop detection period comes, calculate, for a key frame in the key frame set, a distance between a BOW vector of a latest key frame and a BOW vector of each key frame associated therewith, and use an associated key frame with a smallest distance as a candidate loop frame of the latest key frame, where, when world coordinates of at least one pair of FAST feature points in two key frames are the same, the two key frames are considered to be associated; according to the world coordinates of the FAST feature points on the latest key frame and the candidate loop frames, calculating a translation value and a rotation value when the VR camera acquires the latest key frame relative to the acquired candidate loop frames, taking the candidate loop frames as the previous frame of the latest key frame, matching the FAST feature points of the two frames, recalculating the world coordinates of all FAST feature points which are successfully matched according to the calculated translation value and rotation value, updating the world coordinates of all FAST feature points of the latest key frame in a local map description library by the calculated world coordinates of all FAST feature points which are successfully matched, and adding frame identifications corresponding to all FAST feature points which are successfully matched into the local map description library: frame identification of the latest key frame.
In practical applications, the danger detection module 42 issues the danger alert to the user including:
copying preset single-channel reminding audio data into a left channel and a right channel, and respectively carrying out FFT (fast Fourier transform) on the reminding audio data of the two channels to obtain frequency domain reminding audio data of the left channel and the right channel;
determining a sounding position of the reminding audio according to the world coordinates of the attention object and the world coordinates of the VR camera, wherein the sounding position is represented by HRTF standard spatial position parameters, the sounding position of the reminding audio is located on a straight line connecting line between the VR camera and the attention object, and the distance between the sounding position of the reminding audio and the VR camera is preset;
reading corresponding HRTF conversion data from an HRTF standard database according to the sounding position of the reminding audio, carrying out FFT conversion on the HRTF conversion data to obtain frequency domain HRTF conversion data, multiplying the frequency domain reminding audio data of the left and right channels with the frequency domain HRTF conversion data respectively to obtain frequency domain reminding space audio data of the left and right channels, carrying out IFFT conversion on the frequency domain reminding space audio data of the left and right channels respectively to obtain time domain reminding space audio data of the left and right channels, and playing the time domain reminding space audio data of the left and right channels to a user through the left and right channels respectively.
In practical applications, the sending of the danger prompt to the user by the danger detection module 41 includes:
superposing a contour consisting of all FAST characteristic points displaying known world coordinates of the attention object on the 3-dimensional VR image in which the attention object is detected; or, displaying a reminding text message on the 2-dimensional image in an overlapping manner, wherein the reminding text message comprises: distance information of the VR camera from the object of interest; or, displaying the 3-dimensional VR image without the background information, namely deleting the background image except the outline formed by the FAST characteristic points of all known world coordinates on the VR image with the detected attention so as to display only the VR camera, the user and the attention, wherein when displaying the attention, the color of the attention is displayed in a progressive color according to the distance between the attention and the VR camera from far to near.
The apparatus may be located in a VR device.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (18)
1. A Virtual Reality (VR) environmental risk perception method is characterized by comprising the following steps:
extracting feature points in each frame of image acquired by the VR camera in real time;
for each frame image of a second frame and a later frame acquired by the VR camera, matching the feature points in the current frame image and the feature points in the previous frame image, and calculating a motion vector when the VR camera acquires the current frame image relative to the previous frame image according to the positions of each pair of matching points in the two frame images; according to the calculated motion vector, calculating the world coordinates of each feature point successfully matched on the current frame image and the previous frame image, wherein if the current frame image is the second frame image, the initial world coordinates of the VR camera are calculated simultaneously according to the calculated motion vector;
detecting whether an attention object exists in the current frame image according to all feature points of which world coordinates are calculated in the current frame image, if so, calculating the current world coordinates of the VR camera according to the initial world coordinates of the VR camera and the motion vector of the VR camera relative to the previous frame image when acquiring each frame image, calculating the distance between the attention object and the current VR camera according to the world coordinates of the attention object in the current frame and the current world coordinates of the VR camera, and if the distance is smaller than a preset threshold value, sending danger reminding to a user.
2. The method of claim 1, wherein the calculating the world coordinates of each feature point successfully matched on the current frame image and the previous frame image further comprises:
the world coordinates of the calculated feature points are put into a local map description library, and meanwhile, frame identifications corresponding to the feature points are recorded in the local map description library;
after matching the feature points in the current frame image and the previous frame image, and calculating the motion vector when the VR camera collects the current frame image relative to the motion vector when the VR camera collects the previous frame image according to the positions of each pair of matching points in the two frame images, the method further comprises the following steps:
judging whether the current frame meets one of the following key frame judgment conditions:
first, the total number of key frames in the key frame set is less than a first threshold value;
secondly, the number of the feature points of the current frame successfully matched with the previous frame image/the total number of the feature points extracted from the current frame is less than a second threshold value;
if yes, determining the current frame as a key frame, putting the frame identification of the current frame into a key frame set, and then executing the action of calculating a motion vector when the VR camera collects the current frame image relative to the motion vector when the VR camera collects the previous frame image according to the positions of each pair of matching points in the two frame images; otherwise, determining the current frame as a non-key frame, discarding the current frame, and directly switching to the next frame.
3. The method of claim 2, wherein when the current frame is determined to be the key frame, said calculating the world coordinates of each feature point successfully matched on the current frame image and the previous frame image further comprises:
respectively matching all feature points of the current frame with the calculated world coordinates with all feature points of each key frame in the key frame set, if the matching rate exceeds a preset third threshold value, considering that the current frame is redundant, not adding the current frame into the key frame set, not updating a local map description library with the feature points of the current frame with the calculated world coordinates, and switching to the next frame, wherein if the world coordinates of the two feature points are the same, the two feature points are matched.
4. The method of claim 2, wherein the calculating the world coordinates of each feature point successfully matched on the current frame image and the previous frame image further comprises:
forming a bag-of-words BOW vector by all feature points of the current frame with the calculated world coordinates, respectively matching the BOW vector of the current frame with the BOW vector of each key frame in the key frame set, if the BOW vector of the current frame is successfully matched with one key frame, considering that the current frame is successfully relocated, namely considering that the position where the VR camera collects the current frame is the same as the position where the key frame is successfully collected, discarding the feature points originally extracted from the current frame, finding the world coordinates of all the feature points corresponding to the key frame which is successfully matched in a local map description library, adding the frame identifier of the current frame in a frame identifier list of the world coordinates of each found feature point, and not putting the current frame in the key frame set.
5. The method of claim 4, wherein the key frame decision condition further comprises:
the time consumed by the last repositioning process of the current frame is longer than a preset fifth threshold.
6. The method of claim 2, further comprising:
when a preset closed loop detection period comes, respectively calculating the distance between the BOW vector of the latest key frame and the BOW vector of each key frame associated with the latest key frame for the key frames in the key frame set, and taking the associated key frame with the minimum distance as a candidate closed loop frame of the latest key frame, wherein when the world coordinates of at least one pair of feature points in the two key frames are the same, the two key frames are considered to be associated;
calculating a motion vector when the VR camera acquires the latest key frame relative to the acquisition of the candidate loopback frame according to the world coordinates of the feature points on the latest key frame and the candidate loopback frame, matching the feature points of the two frames by taking the candidate loopback frame as the previous frame of the latest key frame, recalculating the world coordinates of all the feature points which are successfully matched according to the calculated motion vector, updating the world coordinates of all the feature points of the latest key frame in a local map description library according to the calculated world coordinates of all the feature points which are successfully matched, and adding frame identifications corresponding to all the feature points which are successfully matched in the local map description library: frame identification of the latest key frame.
7. The method of claim 1, wherein the extracting the feature points in the image is: and extracting the characteristic FAST characteristic points in the image by an accelerated segmentation test.
8. The method of claim 1, wherein said issuing a hazard reminder to a user comprises:
copying preset single-channel reminding audio data into a left channel and a right channel, and respectively carrying out FFT (fast Fourier transform) on the reminding audio data of the two channels to obtain frequency domain reminding audio data of the left channel and the right channel;
determining a sounding position of the reminding audio according to the world coordinates of the attention object and the world coordinates of the VR camera, wherein the sounding position is represented by HRTF standard spatial position parameters, the sounding position of the reminding audio is located on a straight line connecting line between the VR camera and the attention object, and the distance between the sounding position of the reminding audio and the VR camera is preset;
reading corresponding HRTF conversion data from an HRTF standard database according to the sounding position of the reminding audio, carrying out FFT conversion on the HRTF conversion data to obtain frequency domain HRTF conversion data, multiplying the frequency domain reminding audio data of the left and right channels with the frequency domain HRTF conversion data respectively to obtain frequency domain reminding space audio data of the left and right channels, carrying out IFFT conversion on the frequency domain reminding space audio data of the left and right channels respectively to obtain time domain reminding space audio data of the left and right channels, and playing the time domain reminding space audio data of the left and right channels to a user through the left and right channels respectively.
9. The method of claim 1, wherein said issuing a hazard reminder to a user comprises:
superposing a contour formed by all feature points of known world coordinates of the object of interest on the 3-dimensional VR image in which the object of interest is detected; alternatively, the first and second electrodes may be,
displaying prompting text information on the 2-dimensional image in an overlapping mode, wherein the prompting text information comprises: distance information of the VR camera from the object of interest; alternatively, the first and second electrodes may be,
and displaying the 3-dimensional VR image without the background information, namely deleting the background image except the outline formed by all feature points with known world coordinates on the VR image with the detected attention so as to display only the VR camera, the user and the attention, wherein when the attention is displayed, the color of the attention is displayed in a progressive manner according to the distance between the attention and the VR camera from far to near.
10. A virtual reality, VR, environmental hazard awareness apparatus, comprising:
the characteristic extraction and calculation module is used for extracting characteristic points in each frame of image acquired by the VR camera in real time; for each frame image of a second frame and a later frame acquired by the VR camera, matching the feature points in the current frame image and the feature points in the previous frame image, and calculating a motion vector when the VR camera acquires the current frame image relative to the previous frame image according to the positions of each pair of matching points in the two frame images; according to the calculated motion vector, calculating the world coordinates of each feature point successfully matched on the current frame image and the previous frame image, wherein if the current frame image is a second frame image collected by the VR camera, the initial world coordinates of the VR camera are calculated simultaneously according to the calculated motion vector;
and the danger detection module is used for detecting whether an attention object exists in the current frame image according to all the feature points of which the world coordinates are calculated in the current frame image, if so, calculating the current world coordinates of the VR camera according to the initial world coordinates of the VR camera and the motion vector of the VR camera relative to the previous frame image when acquiring each frame image, calculating the distance between the attention object and the current VR camera according to the world coordinates of the attention object in the current frame and the current world coordinates of the VR camera, and if the distance is smaller than a preset threshold value, sending a danger prompt to a user.
11. The apparatus of claim 10, wherein the feature extraction and calculation module further calculates the world coordinates of each feature point successfully matched between the current frame image and the previous frame image,
the world coordinates of the calculated feature points are put into a local map description library, and meanwhile, frame identifications corresponding to the feature points are recorded in the local map description library;
and after the characteristic extraction and calculation module matches the characteristic points in the current frame image and the previous frame image, the characteristic extraction and calculation module further uses the motion vector when the VR camera collects the current frame image relative to the motion vector when the previous frame image is collected according to the positions of each pair of matching points in the two frame images,
judging whether the current frame meets one of the following key frame judgment conditions:
first, the total number of key frames in the key frame set is less than a first threshold value;
secondly, the number of the feature points of the current frame successfully matched with the previous frame image/the total number of the feature points extracted from the current frame is less than a second threshold value;
if yes, determining the current frame as a key frame, putting the frame identification of the current frame into a key frame set, and then executing the action of calculating a motion vector when the VR camera collects the current frame image relative to the motion vector when the VR camera collects the previous frame image according to the positions of each pair of matching points in the two frame images; otherwise, determining the current frame as a non-key frame, discarding the current frame, and directly switching to the next frame.
12. The apparatus of claim 11, wherein when the current frame is determined as the key frame, the feature extraction and calculation module further calculates the world coordinates of each feature point successfully matched on the current frame image and the previous frame image,
respectively matching all feature points of the current frame with the calculated world coordinates with all feature points of each key frame in the key frame set, if the matching rate exceeds a preset third threshold value, considering that the current frame is redundant, not adding the current frame into the key frame set, not updating a local map description library with the feature points of the current frame with the calculated world coordinates, and switching to the next frame, wherein if the world coordinates of the two feature points are the same, the two feature points are matched.
13. The apparatus of claim 11, wherein the feature extraction and calculation module further calculates the world coordinates of each feature point successfully matched between the current frame image and the previous frame image,
forming a bag-of-words BOW vector by all feature points of the current frame with the calculated world coordinates, respectively matching the BOW vector of the current frame with the BOW vector of each key frame in the key frame set, if the BOW vector of the current frame is successfully matched with one key frame, considering that the current frame is successfully relocated, namely considering that the position where the VR camera collects the current frame is the same as the position where the key frame is successfully collected, discarding the feature points originally extracted from the current frame, finding the world coordinates of all the feature points corresponding to the key frame which is successfully matched in a local map description library, adding the frame identifier of the current frame in a frame identifier list of the world coordinates of each found feature point, and not putting the current frame in the key frame set.
14. The apparatus of claim 13, wherein the determining whether the current frame satisfies the key frame determination condition by the feature extraction and calculation module further comprises:
the time consumed by the last repositioning process of the current frame is longer than a preset fifth threshold.
15. The apparatus of claim 11, wherein the feature extraction and computation module is further configured to,
when a preset closed loop detection period comes, respectively calculating the distance between the BOW vector of the latest key frame and the BOW vector of each key frame associated with the latest key frame for the key frames in the key frame set, and taking the associated key frame with the minimum distance as a candidate closed loop frame of the latest key frame, wherein when the world coordinates of at least one pair of feature points in the two key frames are the same, the two key frames are considered to be associated;
calculating a motion vector when the VR camera acquires the latest key frame relative to the acquisition of the candidate loopback frame according to the world coordinates of the feature points on the latest key frame and the candidate loopback frame, matching the feature points of the two frames by taking the candidate loopback frame as the previous frame of the latest key frame, recalculating the world coordinates of all the feature points which are successfully matched according to the calculated motion vector, updating the world coordinates of all the feature points of the latest key frame in a local map description library according to the calculated world coordinates of all the feature points which are successfully matched, and adding frame identifications corresponding to all the feature points which are successfully matched in the local map description library: frame identification of the latest key frame.
16. The apparatus of claim 10, wherein the feature extraction and calculation module extracts the feature points in the image as: and extracting the characteristic FAST characteristic points in the image by an accelerated segmentation test.
17. The apparatus of claim 10, wherein the hazard detection module issuing a hazard reminder to a user comprises:
copying preset single-channel reminding audio data into a left channel and a right channel, and respectively carrying out FFT (fast Fourier transform) on the reminding audio data of the two channels to obtain frequency domain reminding audio data of the left channel and the right channel;
determining a sounding position of the reminding audio according to the world coordinates of the attention object and the world coordinates of the VR camera, wherein the sounding position is represented by HRTF standard spatial position parameters, the sounding position of the reminding audio is located on a straight line connecting line between the VR camera and the attention object, and the distance between the sounding position of the reminding audio and the VR camera is preset;
reading corresponding HRTF conversion data from an HRTF standard database according to the sounding position of the reminding audio, carrying out FFT conversion on the HRTF conversion data to obtain frequency domain HRTF conversion data, multiplying the frequency domain reminding audio data of the left and right channels with the frequency domain HRTF conversion data respectively to obtain frequency domain reminding space audio data of the left and right channels, carrying out IFFT conversion on the frequency domain reminding space audio data of the left and right channels respectively to obtain time domain reminding space audio data of the left and right channels, and playing the time domain reminding space audio data of the left and right channels to a user through the left and right channels respectively.
18. The apparatus of claim 10, wherein the hazard detection module issuing a hazard reminder to a user comprises:
superposing a contour formed by all feature points of known world coordinates of the object of interest on the 3-dimensional VR image in which the object of interest is detected; alternatively, the first and second electrodes may be,
displaying prompting text information on the 2-dimensional image in an overlapping mode, wherein the prompting text information comprises: distance information of the VR camera from the object of interest; alternatively, the first and second electrodes may be,
and displaying the 3-dimensional VR image without the background information, namely deleting the background image except the outline formed by all feature points with known world coordinates on the VR image with the detected attention so as to display only the VR camera, the user and the attention, wherein when the attention is displayed, the color of the attention is displayed in a progressive manner according to the distance between the attention and the VR camera from far to near.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810412419.6A CN108597036B (en) | 2018-05-03 | 2018-05-03 | Virtual reality environment danger sensing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810412419.6A CN108597036B (en) | 2018-05-03 | 2018-05-03 | Virtual reality environment danger sensing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108597036A CN108597036A (en) | 2018-09-28 |
| CN108597036B true CN108597036B (en) | 2022-04-12 |
Family
ID=63620609
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810412419.6A Active CN108597036B (en) | 2018-05-03 | 2018-05-03 | Virtual reality environment danger sensing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108597036B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020107487A1 (en) * | 2018-11-30 | 2020-06-04 | 深圳市大疆创新科技有限公司 | Image processing method and unmanned aerial vehicle |
| CN111105467B (en) * | 2019-12-16 | 2023-08-29 | 北京超图软件股份有限公司 | Image calibration method and device and electronic equipment |
| US11493764B2 (en) * | 2020-06-04 | 2022-11-08 | Htc Corporation | Method for dynamically displaying real-world scene, electronic device, and computer readable medium |
Family Cites Families (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006345445A (en) * | 2005-06-10 | 2006-12-21 | Sony Corp | Moving picture conversion apparatus and moving picture conversion method, and computer program |
| WO2008106680A2 (en) * | 2007-03-01 | 2008-09-04 | Jerry Mahabub | Audio spatialization and environment simulation |
| CN101391589A (en) * | 2008-10-30 | 2009-03-25 | 上海大学 | Vehicle intelligent alarming method and device |
| ITMI20121210A1 (en) * | 2012-07-11 | 2014-01-12 | Rai Radiotelevisione Italiana | A METHOD AND AN APPARATUS FOR THE EXTRACTION OF DESCRIPTORS FROM VIDEO CONTENT, PREFERABLY FOR SEARCH AND RETRIEVAL PURPOSE |
| US9754167B1 (en) * | 2014-04-17 | 2017-09-05 | Leap Motion, Inc. | Safety for wearable virtual reality devices via object detection and tracking |
| JP2016004493A (en) * | 2014-06-18 | 2016-01-12 | キヤノン株式会社 | Image processor and control method thereof |
| US10339389B2 (en) * | 2014-09-03 | 2019-07-02 | Sharp Laboratories Of America, Inc. | Methods and systems for vision-based motion estimation |
| CN105574552A (en) * | 2014-10-09 | 2016-05-11 | 东北大学 | Vehicle ranging and collision early warning method based on monocular vision |
| US10719939B2 (en) * | 2014-10-31 | 2020-07-21 | Fyusion, Inc. | Real-time mobile device capture and generation of AR/VR content |
| WO2017180990A1 (en) * | 2016-04-14 | 2017-10-19 | The Research Foundation For The State University Of New York | System and method for generating a progressive representation associated with surjectively mapped virtual and physical reality image data |
| CN107025668B (en) * | 2017-03-30 | 2020-08-18 | 华南理工大学 | Design method of visual odometer based on depth camera |
| CN107193279A (en) * | 2017-05-09 | 2017-09-22 | 复旦大学 | Robot localization and map structuring system based on monocular vision and IMU information |
-
2018
- 2018-05-03 CN CN201810412419.6A patent/CN108597036B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN108597036A (en) | 2018-09-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7282080B2 (en) | 3D bounding boxes from 2D images and point cloud data | |
| US10977818B2 (en) | Machine learning based model localization system | |
| EP2614487B1 (en) | Online reference generation and tracking for multi-user augmented reality | |
| US9269003B2 (en) | Diminished and mediated reality effects from reconstruction | |
| JP4198054B2 (en) | 3D video conferencing system | |
| US9412034B1 (en) | Occlusion handling for computer vision | |
| CN107392958B (en) | Method and device for determining object volume based on binocular stereo camera | |
| CN108597036B (en) | Virtual reality environment danger sensing method and device | |
| KR20150082379A (en) | Fast initialization for monocular visual slam | |
| US20200258309A1 (en) | Live in-camera overlays | |
| TWI744610B (en) | Scene reconstructing system, scene reconstructing method and non-transitory computer-readable medium | |
| WO2019021569A1 (en) | Information processing device, information processing method, and program | |
| US11282180B1 (en) | Object detection with position, pose, and shape estimation | |
| WO2018216341A1 (en) | Information processing device, information processing method, and program | |
| US11373411B1 (en) | Three-dimensional object estimation using two-dimensional annotations | |
| US20200211275A1 (en) | Information processing device, information processing method, and recording medium | |
| CN116740669B (en) | Multi-view image detection method, device, computer equipment and storage medium | |
| CN107358624B (en) | Monocular dense instant positioning and map reconstruction method | |
| KR102336284B1 (en) | Moving Object Detection Method and System with Single Camera | |
| US11417063B2 (en) | Determining a three-dimensional representation of a scene | |
| KR20150040194A (en) | Apparatus and method for displaying hologram using pupil track based on hybrid camera | |
| CN112598736A (en) | Map construction based visual positioning method and device | |
| KR102542363B1 (en) | Method for recognizing object in 3 dimentional space | |
| US20220230342A1 (en) | Information processing apparatus that estimates object depth, method therefor, and storage medium holding program therefor | |
| US20240144595A1 (en) | 3d scene reconstruction with additional scene attributes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |