WO2017107537A1

WO2017107537A1 - Virtual reality device and obstacle avoidance method

Info

Publication number: WO2017107537A1
Application number: PCT/CN2016/096987
Authority: WO
Inventors: 张超
Original assignee: 乐视控股（北京）有限公司; 乐视致新电子科技（天津）有限公司
Priority date: 2015-12-21
Filing date: 2016-08-26
Publication date: 2017-06-29
Also published as: CN105893928A

Abstract

Disclosed are a virtual reality device and an obstacle avoidance method. In the embodiments provided in the present application, an obstacle avoidance instruction is made according to a relative movement speed of a virtual reality device with respect to an obstacle, a distance between the virtual reality device and the obstacle at a second time point and the acceleration of the virtual reality device at the second time point, so as to lead a user of the virtual reality device to avoid the obstacle effectively, so that the experience satisfaction is high.

Description

Virtual reality device and obstacle avoidance method

This application claims priority to the Chinese Patent Application filed on Dec. 21, 2015, the application number is 201510976181.6, and the application name is “Virtual Reality Device and Obstacle Avoidance Method Provided by Virtual Reality Device”, the entire contents of which are incorporated by reference. Combined in this application.

Technical field

The present application relates to the field of communications, and in particular, to a virtual reality device and an obstacle avoidance method.

Background technique

A head mounted display is a device for displaying images and colors. Usually in the form of an eye mask or helmet, the display is placed close to the user's eyes, and the focal length is adjusted by the optical path to project the image to the eye at a close distance.

In the process of implementing the prior art, the inventors found that at least the following problems exist in the prior art:

In the process of using a head-mounted virtual reality device, such as an eye mask or a helmet, since the user's attention is mainly focused on the head-mounted display, the obstacle in the environment in which the user is located cannot be effectively perceived, and the user is easily safe. Hidden dangers.

One solution to this problem is to integrate a 3D camera in the head mounted device to detect the environment in which the user is located. However, on the one hand, holographic 3D cameras usually require more than 5 lenses, which brings about a significant increase in manufacturing costs for equipment manufacturing. On the other hand, the 3D camera only reports the image, but can not give the user the pre-judgment of avoiding obstacles, and the effectiveness of the user's potential safety hazard is insufficient.

Therefore, in the present application, an obstacle avoidance method for effectively solving the obstacle avoidance effectiveness of a head-mounted virtual reality device by a user and a virtual reality device having an effective obstacle avoidance function are provided.

Summary of the invention

The embodiment of the present application provides an obstacle avoidance method for guiding a user of a virtual reality device to effectively avoid obstacles and experience satisfactory satisfaction. Specifically, an obstacle avoidance method is applied to a virtual reality device, including the following steps:

At a first time point, acquiring a first image of the obstacle photographed by the first camera at the first angle and a second image of the obstacle photographed by the second camera at the second angle;

At a second time point, acquiring a third image of the obstacle photographed by the first camera at the third angle and a fourth image of the obstacle photographed by the second camera at the fourth angle;

Deriving, according to the first image, the second image, the third image, and the fourth image, a relative motion speed of the virtual reality device and the obstacle, and a distance between the virtual reality device and the obstacle at the second time point;

Acquiring the acceleration of the virtual reality device at the second time point;

The obstacle avoidance instruction is made according to the relative motion speed of the virtual reality device and the obstacle, the distance of the virtual reality device from the obstacle at the second time point, and the acceleration of the virtual reality device at the second time point.

The embodiment of the present application further provides a virtual reality device, including:

Binocular camera module for:

Calculation module for:

According to the first image, the second image, the third image and the fourth image formed by the obstacle in the binocular camera module, the relative motion speed of the virtual reality device and the obstacle is obtained, and the virtual reality device is at the second time point and the obstacle Distance of matter

Acceleration module for:

Acquiring, the acceleration of the virtual reality device at the second time point;

The computing module is further configured to:

The embodiment of the present application provides an electronic device, including the virtual reality device described in any of the foregoing embodiments.

The embodiment of the present application provides a non-transitory computer readable storage medium, wherein the non-transitory computer readable storage medium can store computer instructions, which can implement the obstacle avoidance method provided by the embodiments of the present application. Some or all of the steps in each implementation.

An embodiment of the present application provides an electronic device, including: one or more processors; and a memory; Wherein the memory stores instructions executable by the one or more processors, the instructions being arranged to perform any of the above-described obstacle avoidance methods of the present application.

An embodiment of the present application provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, The computer is caused to perform the above-mentioned obstacle avoidance method according to the embodiment of the present application.

The virtual reality device and the obstacle avoidance method provided by the embodiments of the present application have at least the following beneficial effects:

The user of the virtual reality device can be guided according to the relative motion speed of the virtual reality device and the obstacle, the distance of the virtual reality device from the obstacle at the second time point, and the acceleration of the virtual reality device at the second time point. Effective obstacle avoidance and good experience satisfaction.

DRAWINGS

The drawings described herein are intended to provide a further understanding of the present application, and are intended to be a part of this application. In the drawing:

FIG. 1 is a schematic flowchart of a method for avoiding obstacles in an embodiment of the present application.

2 is a schematic diagram of measuring the distance of a virtual reality device from an obstacle at a second point in time.

FIG. 3 is a schematic structural diagram of a virtual reality device according to an embodiment of the present disclosure;

FIG. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

detailed description

The technical solutions of the present application will be clearly and completely described in the following with reference to the specific embodiments of the present application and the corresponding drawings. It is apparent that the described embodiments are only a part of the embodiments of the present application, and not all of them. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present application without departing from the inventive scope are the scope of the present application.

FIG. 1 is a flowchart of a method for avoiding obstacles in an embodiment of the present application, which specifically includes the following steps:

S01: At a first time point, acquire a first image of the obstacle photographed by the first camera at the first angle and a second image of the obstacle photographed by the second camera at the second angle.

The virtual reality device includes, but is not limited to, an eye-catching device, a helmet, a glasses, and the like, and an electronic virtual reality device that integrates sound, light, electricity, and the like with a computer chip as a core, and integrates visual, acoustic, and tactile sensors. The virtual reality device of the embodiment of the present invention is configured with a binocular camera module, a calculation module, an acceleration module, and the like.

The binocular camera module can refer to an integrated camera formed by two cameras arranged in a position similar to a human body. Of course, the binocular camera module here includes, in addition to the two cameras, auxiliary components such as corresponding signal transmission lines.

When the user wears the virtual reality device, at the first time, the first camera in the binocular camera module captures the obstacle at a first angle to form a first image; the second camera in the binocular camera module takes a second angle An obstacle forms a second image.

It should be noted that the obstacle here can be regarded as a target object. In reality, it can be correspondingly a street light pole, a parked car, a person near the user, and the like. These target objects are presented in the background image in the foreground image in the first image and the second image. In practical applications, in order to speed up the speed of information processing, in the design of the computer-specific algorithm, some target objects that are obviously impossible to become obstacles can be pre-filtered out to prevent these target objects from adversely affecting the detection of obstacles. .

The terms "first" and "second" are used merely for the convenience of the description, and do not imply a certain order or a continuous relationship. For the first camera to capture an obstacle at a first angle to form a first image, the "first angle" herein refers to the manner in which the obstacle is imaged on the first camera. Different obstacles are imaged differently on the first camera, and the same obstacles are imaged differently on different cameras. The "first" and "second" herein refer only to the logical difference of this narrative.

S02: At a second time point, acquire a third image of the obstacle photographed by the first camera at the third angle and a fourth image of the obstacle photographed by the second camera at the fourth angle.

Similar to step S01, when the user wears the virtual reality device, at the second time, the first camera in the binocular camera module captures the obstacle at a third angle to form a third image; the second camera in the binocular camera module The obstacle is photographed at a fourth angle to form a fourth image.

Here, "first", "second", "third", "fourth" are similar to "first" and "second" above, which means that the narrative is logically different.

S03: Deriving the relative motion speed of the virtual reality device and the obstacle according to the first image, the second image, the third image, and the fourth image, and the distance between the virtual reality device and the obstacle at the second time point.

In an embodiment provided by the present application, the computing module of the virtual reality device is based on an obstacle in the double The third image and the fourth image formed by the camera module are derived, and the distance between the virtual reality device and the obstacle at the second time is obtained, which specifically includes:

a computing module of the virtual reality device calculates a parallax (L1-L2) of the obstacle in the third image and the fourth image;

Deriving the distance of the virtual reality device from the obstacle at the second time according to the formula d=bf/(L1-L2); wherein b is the distance between the optical center C1 of the first camera and the second camera optical center C2;

f is the focal length of the first camera and the second camera.

Please refer to FIG. 2. FIG. 2 is a schematic diagram of measuring the distance of the virtual reality device from the obstacle at the second time.

Specifically, the distance between the virtual reality device and the obstacle at the second time point can be derived from the position of the obstacle in the third image and the fourth image.

In Fig. 2, point P is assumed to be any target point on the obstacle. C1 and C2 are assumed to be the optical center of the first camera and the optical center of the second camera, respectively, and the distance between the optical center C1 and the optical center C2 is b. The focal lengths of the first camera and the second camera are both f. The projection point of the point P on the imaging plane of the first camera is P1, and the projection point of the point P on the imaging plane of the second camera is P2. Point P, the distance from the line connecting optical center C1 and optical center C2 is d. The passing center C1 is perpendicular to the imaging plane, and the foot is A1. The optical center C2 is perpendicular to the imaging plane, and the foot is A2. P is perpendicular to the imaging plane, and the foot is B. Set, A1P1 = L1, A2P2 = L2, P2B = a.

From the similar triangle relationship:

D-f/d=a/(a+L2);

D-f/d=(b-L1+L2+a)/(a+b+L2);

From the above two formulas can be calculated:

d=f(a+L2)/L2=bf/(L1-L2)

Thus, the distance d is related to b, f and L1-L2. L1-L2 is called the parallax of the point P on the imaging planes of the first camera and the second camera. The corresponding points of the two image responses have parallax only in the horizontal direction, and the coordinate values in the Y direction are equal. For a fixed binocular camera module, the values of the parameters b and f have been determined, and only the corresponding pixel parallax needs to be obtained for the image to obtain the distance of the obstacle.

Similarly, according to the above formula d=bf/(L1-L2), the distance between the virtual reality device and the obstacle can be derived at the first time point in the first image and the second image.

Further, in still another embodiment provided by the present application, the computing module of the virtual reality device is derived according to the first image, the second image, the third image, and the fourth image formed by the obstacle in the binocular camera module. The relative motion speed of the virtual reality device and the obstacle includes:

A computing module of the virtual reality device calculates a parallax of the obstacle in the first image and the second image;

Deriving the distance of the virtual reality device from the obstacle at the first time according to the formula d=bf/(L1-L2);

Calculating the relative relationship between the virtual reality device and the obstacle according to the distance of the virtual reality device from the obstacle at the first time point, the distance of the virtual reality device from the obstacle at the second time point, and the time difference between the first time point and the second time point Movement speed.

In order to distinguish, the distance between the virtual reality device and the obstacle at the first time point is set to d1, and the distance between the virtual reality device and the obstacle at the second time point is d2, and the time difference t between the two is known, thereby obtaining The average value of the relative motion speed of the virtual reality device and the obstacle is v=(d2-d1)/t. Here, when the time difference is short, the average value of the relative motion speed of the virtual reality device and the obstacle can be approximated as the second time point virtual The relative speed of movement of realistic equipment and obstacles.

In another achievable manner provided by the embodiment of the present application, the motion of the obstacle in the three-dimensional scene is considered to correspond to the projection of the obstacle in the two-dimensional image plane. The flow of this motion in the form of image plane brightness is called optical flow.

When calculating according to this algorithm, you can assume:

(1) The brightness between adjacent frames is constant;

(2) The frame taking time of adjacent video frames is continuous, or the motion of objects between adjacent frames is relatively small;

(3) Maintain spatial consistency; that is, pixels of the same sub-image have the same motion.

Correspondingly, the brightness between the first image and the third image is constant, and the brightness between the second image and the fourth image is constant;

The time interval between the first time point and the second time point is small, so that the obstacle has a small variation distance in the first image and the third image;

The obstacles in the first camera imaging process, the pixels constituting the first image have substantially the same motion, thereby forming a third image.

In still another embodiment provided by the present application, the computing module of the virtual reality device derives the virtuality according to the first image, the second image, the third image, and the fourth image formed by the obstacle in the binocular camera module. The relative movement speed of the actual equipment and obstacles, including:

Finding V _x and V according to the first image, the second image, the third image, and the fourth image formed by the obstacle in the binocular camera module, and the equation I _x *V _x +I _y *V _y =−I _t _y ;

According to the projection relationship, the relative motion speed of the virtual reality device and the obstacle can be obtained;

among them,

They are the partial derivatives of the gray value pair x, y, t, respectively.

According to the assumption that the obstacle has a small variation distance in the first image and the third image, and the brightness between the first image and the third image is constant, and the brightness between the second image and the fourth image is constant, that is, In the case of 2D+t dimension (spatial two-dimensional plus time dimension), it is assumed that the brightness of a certain unit of the obstacle located at (x, y, t) is I(x, y, t). The volume unit moves dx, dy, dt between two image frames. So according to the same assumption of brightness:

I(x, y, t)=I(x+dx, y+dy, t+dt);

The above formula is derived from the Taylor series:

which is:

or

make

Available: I _x *V _x +I _y *V _y =-I _t

thereby,

The partial derivatives of the gray value pairs x, y, t, respectively, can be estimated from the image, which has two unknowns, V _x and V _y .

According to the assumption that the obstacle has a small variation distance in the first image and the third image, the V _x and V _y of all the primitives or pixels in the tiny distance are the same, two unknowns, multiple equations, and the least squares method is easy. Find the values of V _x and V _y . Then, according to the projection relationship, the relative motion speed of the virtual reality device and the obstacle can be obtained.

Similarly, at the second time point, the distance between the virtual reality device and the obstacle can be derived according to the above formula d=bf/(L1-L2).

In the above calculations, the imaging of the same volume unit of the obstacle in different images is directly given. However, how to determine that one primitive or pixel in one image and another primitive or pixel in another image represent the same unit of the obstacle is a complicated problem.

In still another embodiment provided by the present application, the computing module of the virtual reality device derives the virtual reality according to the first image, the second image, the third image, and the fourth image formed by the obstacle in the binocular camera module. The relative movement speed of the device and the obstacle, and the distance between the virtual reality device and the obstacle at the second time point, specifically include:

Determining, from the first image, a first primitive, the first primitive being an image of a body unit of the obstacle in the first image;

Determining, from the second image, the third image, and the fourth image, the second primitive, the third primitive, and the fourth primitive corresponding to the first primitive;

According to the correspondence between the first picture element, the second picture element, the third picture element and the fourth picture element, the relative motion speed of the virtual reality device and the obstacle is obtained, and the distance between the virtual reality device and the obstacle at the second time point is derived. .

In an implementation manner provided by the present application, a method of detecting whether a primitive or a pixel in a different image corresponds is given.

In still another embodiment provided by the present application, the first primitive is determined from the first image, and the second primitive and the third primitive corresponding to the first primitive are respectively determined from the second image, the third image, and the fourth image. The picture element and the fourth picture element specifically include:

Finding primitives that characterize physical features of the obstacle from the first image;

Searching for the second primitive, the third primitive, and the fourth primitive of the primitive representing the same physical feature of the obstacle from the second image, the third image, and the fourth image, respectively.

Specifically, big data technology can be used to establish a physical feature database of obstacles. For example, it can be built The shape parameters of the contours of the head, shoulders, and feet of the human body. These contours and environmental backgrounds tend to have higher contrast in each image. It is assumed that according to the shape, contrast and other parameters of the contour, a certain pixel or pixel in the first image is found to be the head feature of the human body. In addition, according to the shape, the contrast and other parameters of the contour, a certain pixel or pixel in the second image is found as the head feature of the human body. Then, it can be considered that the primitive representing the human head feature in the first image is identical to the body unit of the human body represented by the primitive representing the human head feature in the second image. Similarly, the third picture element and the fourth picture element that represent the same physical feature of the obstacle may be searched from the third image and the fourth image. Therefore, on the basis of this, the relative motion speed of the virtual reality device and the obstacle and the distance between the virtual reality device and the obstacle can be calculated.

Searching, from the first image, the second image, the third image, and the fourth image, the first primitive, the second primitive, and the third primitive that represent the same sift (Scale-invariant feature transform) feature The fourth picture element.

Specifically, the sift feature in each image can be found, and then the relative motion speed of the virtual reality device and the obstacle and the distance between the virtual reality device and the obstacle are calculated according to the same sift feature in different images.

The first primitive, the second primitive, the third primitive, and the fourth primitive that can match each other are searched from the first image, the second image, the third image, and the fourth image by using an image convolution method.

Specifically, the corresponding area in different images may be determined by using image convolution, so that the relative motion speed of the virtual reality device and the obstacle and the distance between the virtual reality device and the obstacle may be calculated.

Further, in a further embodiment provided by the application, the method further includes:

Determining, by the plurality of times, the first primitive, the second primitive, the third primitive, and the fourth primitive, wherein the first primitive, the second primitive, the third primitive, and the fourth primitive are different each time. ;

Deriving the relative motion speed and virtual reality of multiple independent virtual reality devices and obstacles The distance of the device from the obstacle at the second time;

The relative motion speed of the virtual reality device and the obstacle repeatedly acquired repeatedly and the distance between the virtual reality device and the obstacle at the second time point are optimized by the least squares method to obtain the relative motion speed of the optimized virtual reality device and the obstacle. And the distance of the optimized virtual reality device from the obstacle at the second point in time.

Specifically, the least moving method can be used to optimize the relative motion speed of the virtual reality device and the obstacle calculated according to different target points and the distance between the virtual reality device and the obstacle.

S04: Acquire the acceleration of the virtual reality device at the second time point.

The acceleration module can be an electronic device capable of measuring acceleration forces. In an achievable embodiment provided by the present application, the piezoelectric effect of the piezoelectric ceramic or the quartz crystal can be utilized, and when the acceleration module is vibrated, the force applied to the piezoelectric element by the mass is also changed. When the measured vibration frequency is much lower than the natural frequency of the accelerometer, the change in force is proportional to the measured acceleration.

The acceleration module of the virtual reality device can acquire the acceleration of the virtual reality device at the second time point.

S05: Perform an obstacle avoidance instruction according to the relative motion speed of the virtual reality device and the obstacle, the distance of the virtual reality device from the obstacle at the second time point, and the acceleration of the virtual reality device at the second time point.

The computing module of the virtual reality device can calculate the relative motion speed of the virtual reality device and the obstacle, the distance of the virtual reality device from the obstacle at the second time point, and the acceleration of the virtual reality device at the second time point. Obtain the expectation that the virtual reality device will collide with the obstacle, so that the obstacle avoidance instruction fed back to the user can be made in time.

In the embodiment provided by the present application, the computing module of the virtual reality device is based on the relative motion speed of the virtual reality device and the obstacle, the distance of the virtual reality device from the obstacle at the second time, and the virtual reality device at the second time. The acceleration makes an obstacle avoidance instruction, so that the user of the virtual reality device can be guided to effectively avoid obstacles, and the experience satisfaction is good.

Finally, it should be understood that those skilled in the art can understand that all or part of the process of implementing the above embodiments can be completed by a computer program to instruct related hardware, and the program can be stored in a non-transitory computer. In a readable storage medium, the program, when executed, may include the flow of an embodiment of the methods as described above. The non-transitory computer readable storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

The above is the obstacle avoidance method in the embodiment of the present application. Based on the same idea, please refer to FIG. 3, and the present application also Providing a virtual reality device 1 comprising:

Binocular camera module 11 for:

The calculation module 12 is configured to:

According to the first image, the second image, the third image, and the fourth image formed by the obstacle in the binocular camera module 11, the relative motion speed of the virtual reality device 1 and the obstacle is obtained, and the virtual reality device 1 is in the second time. The distance between the point and the obstacle;

Acceleration module 13 for:

Obtaining an acceleration of the virtual reality device 1 at a second time point;

The calculation module 12 is further configured to:

The obstacle avoidance instruction is made according to the relative movement speed of the virtual reality device 1 and the obstacle, the distance of the virtual reality device 1 from the obstacle at the second time point, and the acceleration of the virtual reality device 1 at the second time point.

Further, in a further embodiment provided by the application, the calculating module 12 is configured to:

Calculating the parallax (L1-L2) of the obstacle in the third image and the fourth image;

Obtaining the distance of the virtual reality device 1 from the obstacle at the second time point according to the formula d=bf/(L1-L2); wherein b is the distance between the optical center C1 of the first camera and the second camera optical center C2 ;

f is the focal length of the first camera and the second camera.

Further, in a further embodiment provided by the application, the calculating module 12 is further configured to:

Calculating a parallax of the obstacle in the first image and the second image;

Deriving the distance of the virtual reality device 1 from the obstacle at the first time according to the formula d=bf/(L1-L2);

Calculating the virtual reality device 1 and the obstacle according to the distance of the virtual reality device 1 from the obstacle at the first time point, the distance of the virtual reality device 1 from the obstacle at the second time point, and the time difference between the first time point and the second time point The relative speed of movement of objects.

Finding V _x and the first image, the second image, the third image, and the fourth image formed by the obstacle in the binocular camera module 11, and the equation I _x *V _x +I _y *V _y = -I _t V _y ;

According to the projection relationship, the relative motion speed of the virtual reality device 1 and the obstacle can be obtained;

among them,

They are the partial derivatives of the gray value pair x, y, t, respectively.

According to the correspondence between the first picture element, the second picture element, the third picture element and the fourth picture element, the relative motion speed of the virtual reality device 1 and the obstacle is obtained, and the virtual reality device 1 is at the second time point and the obstacle the distance.

Searching from the first image, the second image, the third image, and the fourth image, the first primitive, the second primitive, the third primitive, and the fourth primitive that represent the same sift feature.

Determining the first primitive, the second primitive, the third primitive, and the fourth primitive independently and repeatedly, each time The first picture element, the second picture element, the third picture element, and the fourth picture element are different;

Deriving the relative motion speed of the virtual reality device 1 and the obstacle obtained by multiple independent repetitions, and the distance of the virtual reality device 1 from the obstacle at the second time point;

The relative motion speed of the virtual reality device 1 and the obstacle that are independently acquired repeatedly and the distance between the virtual reality device 1 and the obstacle at the second time point are optimized by the least squares method to obtain the optimized virtual reality device 1 and the obstacle The relative motion speed and the distance of the optimized virtual reality device 1 from the obstacle at the second time point.

In another embodiment of the present application, an electronic device is provided, including the virtual reality device described in any of the foregoing embodiments.

In another embodiment of the present application, a non-transitory computer readable storage medium is also provided, the non-transitory computer readable storage medium storing computer executable instructions executable by any of the above methods The obstacle avoidance method in the example.

4 is a schematic diagram of a hardware structure of an electronic device for performing an obstacle avoidance method according to an embodiment of the present application. As shown in FIG. 4, the device includes:

One or more processors 410 and memory 420, one processor 410 is exemplified in FIG.

The apparatus for performing the obstacle avoidance method may further include: an input device 430 and an output device 440.

The processor 410, the memory 420, the input device 430, and the output device 440 may be connected by a bus or other means, as exemplified by a bus connection in FIG.

The memory 420 is used as a non-transitory computer readable storage medium, and can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions corresponding to the obstacle avoidance method in the embodiment of the present application. Modules (eg, binocular camera module 11, calculation module 12, and acceleration module 13 shown in FIG. 3). The processor 410 executes various functional applications and data processing of the electronic device by executing non-volatile software programs, instructions, and modules stored in the memory 420, that is, implementing the above-described method embodiment obstacle avoidance method.

The memory 420 may include a storage program area and an storage data area, wherein the storage program area may store an operating system, an application required for at least one function; the storage data area may store data created according to usage of the virtual reality device, and the like. Moreover, memory 420 can include high speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 420 can optionally include remotely with respect to processor 410 Set up memory that can be connected to a virtual reality device over a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

Input device 430 can receive input numeric or character information and generate key signal inputs related to user settings and function control of the virtual reality device. Output device 440 can include a display device such as a display screen.

The one or more modules are stored in the memory 420, and when executed by the one or more processors 410, perform the obstacle avoidance method in any of the above method embodiments.

The above products can perform the methods provided by the embodiments of the present application, and have the corresponding functional modules and beneficial effects of the execution method. For technical details that are not described in detail in this embodiment, reference may be made to the method provided by the embodiments of the present application.

The electronic device of the embodiment of the present application exists in various forms, including but not limited to:

(1) Mobile communication devices: These devices are characterized by mobile communication functions and are mainly aimed at providing voice and data communication. Such terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.

(2) Ultra-mobile personal computer equipment: This type of equipment belongs to the category of personal computers, has computing and processing functions, and generally has mobile Internet access. Such terminals include: PDAs, MIDs, and UMPC devices, such as the iPad.

(3) Portable entertainment devices: These devices can display and play multimedia content. Such devices include: audio, video players (such as iPod), handheld game consoles, e-books, and smart toys and portable car navigation devices.

(4) Server: A device that provides computing services. The server consists of a processor, a hard disk, a memory, a system bus, etc. The server is similar to a general-purpose computer architecture, but because of the need to provide highly reliable services, processing power and stability High reliability in terms of reliability, security, scalability, and manageability.

(5) Other electronic devices with data interaction functions.

It is also to be understood that the terms "comprises" or "comprising" or "comprising" or any other variations are intended to encompass a non-exclusive inclusion, such that a process, method, article, Other elements not explicitly listed, or elements that are inherent to such a process, method, commodity, or equipment. In the absence of more restrictions, elements defined by the phrase "including one..." are not excluded from the process, method, article, or device that includes the element. There are also other identical elements.

The above description is only an embodiment of the present application and is not intended to limit the application. Various changes and modifications can be made to the present application by those skilled in the art. Any modifications, equivalents, improvements, etc. made within the spirit and scope of the present application are intended to be included within the scope of the appended claims.

Claims

An obstacle avoidance method, which is characterized by being applied to a virtual reality device, comprising the following steps:

At a first time point, acquiring a first image of the obstacle photographed by the first camera at the first angle and a second image of the obstacle photographed by the second camera at the second angle;

At a second time point, acquiring a third image of the obstacle photographed by the first camera at the third angle and a fourth image of the obstacle photographed by the second camera at the fourth angle;

Deriving, according to the first image, the second image, the third image, and the fourth image, a relative motion speed of the virtual reality device and the obstacle, and a distance between the virtual reality device and the obstacle at the second time point;

Acquiring the acceleration of the virtual reality device at the second time point;

The obstacle avoidance instruction is made according to the relative motion speed of the virtual reality device and the obstacle, the distance of the virtual reality device from the obstacle at the second time point, and the acceleration of the virtual reality device at the second time point.
The obstacle avoidance method according to claim 1, wherein the obtaining the distance between the virtual reality device and the obstacle at the second time according to the third image and the fourth image comprises:

Calculating the parallax (L1-L2) of the obstacle in the third image and the fourth image;

Deriving the distance of the virtual reality device from the obstacle at the second time according to the formula d=bf/(L1-L2); wherein b is the distance between the optical center C1 of the first camera and the second camera optical center C2;

f is the focal length of the first camera and the second camera.
The obstacle avoidance method according to claim 2, wherein the relative motion speed of the virtual reality device and the obstacle is derived according to the first image, the second image, the third image, and the fourth image, and specifically includes:

Calculating a parallax of the obstacle in the first image and the second image;

Deriving the distance of the virtual reality device from the obstacle at the first time according to the formula d=bf/(L1-L2);

Calculating the relative relationship between the virtual reality device and the obstacle according to the distance of the virtual reality device from the obstacle at the first time point, the distance of the virtual reality device from the obstacle at the second time point, and the time difference between the first time point and the second time point Movement speed.
The obstacle avoidance method according to claim 1, wherein the first image and the second image are The image, the third image, and the fourth image are derived to obtain a relative motion speed of the virtual reality device and the obstacle, specifically including:

Calculating V x and V y according to the first image, the second image, the third image, and the fourth image, and the equation I x *V x +I y *V y =−I t ;

According to the projection relationship, the relative motion speed of the virtual reality device and the obstacle can be obtained;

among them,
They are the partial derivatives of the gray value pair x, y, t, respectively.
The obstacle avoidance method according to claim 1, wherein the relative motion speed of the virtual reality device and the obstacle is derived according to the first image, the second image, the third image, and the fourth image, and the virtual reality device is in the first The distance between the second point and the obstacle, including:

Determining, from the first image, a first primitive, the first primitive being an image of a body unit of the obstacle in the first image;

Determining, from the second image, the third image, and the fourth image, the second primitive, the third primitive, and the fourth primitive corresponding to the first primitive;

According to the correspondence between the first picture element, the second picture element, the third picture element and the fourth picture element, the relative motion speed of the virtual reality device and the obstacle is obtained, and the distance between the virtual reality device and the obstacle at the second time point is derived. .
The obstacle avoidance method according to claim 5, wherein the first primitive is determined from the first image, and the second image corresponding to the first primitive is determined from the second image, the third image, and the fourth image, respectively. The yuan, the third picture element, and the fourth picture element, specifically include:

Finding primitives that characterize physical features of the obstacle from the first image;

Searching for the second primitive, the third primitive, and the fourth primitive of the primitive representing the same physical feature of the obstacle from the second image, the third image, and the fourth image, respectively.
The obstacle avoidance method according to claim 5, wherein the first primitive is determined from the first image, and the second image corresponding to the first primitive is determined from the second image, the third image, and the fourth image, respectively. The yuan, the third picture element, and the fourth picture element, specifically include:

Searching from the first image, the second image, the third image, and the fourth image, the first primitive, the second primitive, the third primitive, and the fourth primitive that represent the same scale invariant feature transform sift feature.
The obstacle avoidance method according to claim 5, wherein the first image is determined from the first image The second element, the third picture element, and the fourth picture element corresponding to the first picture element are respectively determined from the second image, the third image, and the fourth image, and specifically include:

The first primitive, the second primitive, the third primitive, and the fourth primitive that can match each other are searched from the first image, the second image, the third image, and the fourth image by using an image convolution method.
The obstacle avoidance method according to claim 5, wherein the method further comprises:

Determining, by the plurality of times, the first primitive, the second primitive, the third primitive, and the fourth primitive, wherein the first primitive, the second primitive, the third primitive, and the fourth primitive are different each time. ;

Deriving the relative motion speed of the virtual reality device and the obstacle obtained by multiple independent repetitions, and the distance of the virtual reality device from the obstacle at the second time point;

The relative motion speed of the virtual reality device and the obstacle repeatedly acquired repeatedly and the distance between the virtual reality device and the obstacle at the second time point are optimized by the least squares method to obtain the relative motion speed of the optimized virtual reality device and the obstacle. And the distance of the optimized virtual reality device from the obstacle at the second point in time.
A virtual reality device, comprising:

Binocular camera module for:

At a first time point, acquiring a first image of the obstacle photographed by the first camera at the first angle and a second image of the obstacle photographed by the second camera at the second angle;

At a second time point, acquiring a third image of the obstacle photographed by the first camera at the third angle and a fourth image of the obstacle photographed by the second camera at the fourth angle;

Calculation module for:

According to the first image, the second image, the third image and the fourth image formed by the obstacle in the binocular camera module, the relative motion speed of the virtual reality device and the obstacle is obtained, and the virtual reality device is at the second time point and the obstacle Distance of matter

Acceleration module for:

Acquiring the acceleration of the virtual reality device at the second time point;

The computing module is further configured to:

The obstacle avoidance instruction is made according to the relative motion speed of the virtual reality device and the obstacle, the distance of the virtual reality device from the obstacle at the second time point, and the acceleration of the virtual reality device at the second time point.
The virtual reality device of claim 10, wherein the computing module is configured to:

Calculating the parallax (L1-L2) of the obstacle in the third image and the fourth image;

Deriving the distance of the virtual reality device from the obstacle at the second time according to the formula d=bf/(L1-L2); wherein b is the distance between the optical center C1 of the first camera and the second camera optical center C2;

f is the focal length of the first camera and the second camera.
The virtual reality device of claim 11, wherein the computing module is further configured to:

Calculating a parallax of the obstacle in the first image and the second image;

Deriving the distance of the virtual reality device from the obstacle at the first time according to the formula d=bf/(L1-L2);

Calculating the relative relationship between the virtual reality device and the obstacle according to the distance of the virtual reality device from the obstacle at the first time point, the distance of the virtual reality device from the obstacle at the second time point, and the time difference between the first time point and the second time point Movement speed.
The virtual reality device of claim 11, wherein the computing module is configured to:

Finding V x and V according to the first image, the second image, the third image, and the fourth image formed by the obstacle in the binocular camera module, and the equation I x *V x +I y *V y =−I t y;

According to the projection relationship, the relative motion speed of the virtual reality device and the obstacle can be obtained;

among them,
They are the partial derivatives of the gray value pair x, y, t, respectively.
The virtual reality device of claim 10, wherein the computing module is configured to:

Determining, from the first image, a first primitive, the first primitive being an image of a body unit of the obstacle in the first image;

Determining, from the second image, the third image, and the fourth image, the second primitive, the third primitive, and the fourth primitive corresponding to the first primitive;

According to the correspondence between the first picture element, the second picture element, the third picture element and the fourth picture element, the derivation is obtained The relative motion speed of the virtual reality device and the obstacle, and the distance of the virtual reality device from the obstacle at the second time.
The virtual reality device of claim 14, wherein the computing module is configured to:

Finding primitives that characterize physical features of the obstacle from the first image;

Searching for the second primitive, the third primitive, and the fourth primitive of the primitive representing the same physical feature of the obstacle from the second image, the third image, and the fourth image, respectively.
The virtual reality device of claim 14, wherein the computing module is configured to:

Searching from the first image, the second image, the third image, and the fourth image, the first primitive, the second primitive, the third primitive, and the fourth primitive that represent the same scale invariant feature transform sift feature.
The virtual reality device of claim 14, wherein the computing module is configured to:

The first primitive, the second primitive, the third primitive, and the fourth primitive that can match each other are searched from the first image, the second image, the third image, and the fourth image by using an image convolution method.
The virtual reality device according to claim 14, wherein the calculation module is further configured to:

Determining, by the plurality of times, the first primitive, the second primitive, the third primitive, and the fourth primitive, wherein the first primitive, the second primitive, the third primitive, and the fourth primitive are different each time. ;

Deriving the relative motion speed of the virtual reality device and the obstacle obtained by multiple independent repetitions, and the distance of the virtual reality device from the obstacle at the second time point;

The relative motion speed of the virtual reality device and the obstacle repeatedly acquired repeatedly and the distance between the virtual reality device and the obstacle at the second time point are optimized by the least squares method to obtain the relative motion speed of the optimized virtual reality device and the obstacle. And the distance of the optimized virtual reality device from the obstacle at the second point in time.
A non-transitory computer readable storage medium, wherein the non-transitory computer readable storage medium stores computer instructions for causing the computer to perform the method of any of claims 1-9 .
An electronic device, comprising:

One or more processors; and,

a memory communicatively coupled to the one or more processors; wherein

The memory stores instructions executable by the one or more processors, the instructions being executed by the one or more processors to enable the one or more processors to:

At a first time point, acquiring a first image of the obstacle photographed by the first camera at the first angle and a second image of the obstacle photographed by the second camera at the second angle;

At a second time point, acquiring a third image of the obstacle photographed by the first camera at the third angle and a fourth image of the obstacle photographed by the second camera at the fourth angle;

Deriving, according to the first image, the second image, the third image, and the fourth image, a relative motion speed of the virtual reality device and the obstacle, and a distance between the virtual reality device and the obstacle at the second time point;

Acquiring the acceleration of the virtual reality device at the second time point;

The obstacle avoidance instruction is made according to the relative motion speed of the virtual reality device and the obstacle, the distance of the virtual reality device from the obstacle at the second time point, and the acceleration of the virtual reality device at the second time point.
A computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to execute The method of claims 1-9.