KR20220080526A - Virtual reality drone simulator - Google Patents

Abstract

A method of operating a virtual reality drone is provided. A virtual reality drone operation method according to an embodiment of the present invention includes the steps of starting the drone upon receiving a startup input in virtual reality; counting a preset time; outputting a warning signal when a departure input is received once within the preset time; outputting a disqualification signal when the departure input is received two or more times within the preset time; and when a departure input is received after the preset time, moving the drone based on the departure input, wherein the departure input includes direction information.

Description

Virtual Reality Drone Simulator {VIRTUAL REALITY DRONE SIMULATOR}

This application relates to a virtual reality drone simulator, and is filed through the 2020 VR/AR content idea commercialization support project supported by the Information and Communication Industry Promotion Agency.

A drone refers to an unmanned aerial vehicle (or unmanned aerial vehicle) (Unmanned Aerial Vehicle, UAV) that can fly and control by radio wave guidance without a pilot on board.

Drones were developed for military use, but as their scope of application has been expanded to various fields, they are being applied to high-altitude video and photography, product delivery, meteorological information collection, and pesticide spraying. As the supply of drones is expanded from military to civilian use, the demand for drones is also increasing rapidly, and while prices are falling and miniaturization, commercial use is increasing as mobility is strengthened. .

Drones are changing many fields, and in the media field, it has become possible to shoot accident sites where safety is not certain, and in the logistics field, delivery using drones has improved the accuracy, efficiency, and convenience of returns. It is expected that consumption patterns will change from 'purchase' to 'lease'. In the information and communication field, it is expected that a global network and communication network to replace the existing network will be built, and in the agricultural field, drones are used to replace the labor shortage such as pesticide spraying, crop seeding, and housekeeping protection.

As such, although the frequency of use of drones is gradually increasing, methods for practicing drone operation are limited. Specifically, the method of practicing drone operation is mainly performed outdoors, but this method has a disadvantage in that there is a high risk of safety accidents and drone failure or loss due to the inexperience of the operator.

Currently, drone simulators are continuously being developed to solve these problems. However, the existing drone simulators provide only a single control mode, do not introduce drone racing regulations at all, and the lack of interest-inducing factors makes the piloting practice boring quickly, making it impossible to achieve the intended purpose of drone pilot training. there is

A method of operating a virtual reality drone, a computer-readable storage medium, and a computer program stored in the computer-readable storage medium are provided.

A virtual reality drone operation method according to an embodiment of the present invention includes the steps of starting the drone upon receiving a startup input in virtual reality; counting a preset time; outputting a warning signal when a departure input is received once within the preset time; outputting a disqualification signal when the departure input is received two or more times within the preset time; and when a departure input is received after the preset time, moving the drone based on the departure input, wherein the departure input includes direction information.

When the start input is received in the virtual reality, starting the drone may include rotating a propeller of the drone.

The method may further include stopping rotation of a propeller of the drone and free-falling the drone when the drone receives a start-off input while moving.

The virtual reality drone operation method may further include, after outputting the warning signal, counting the preset time from the beginning again.

The preset time may be 3 seconds.

The start input may be manipulated using two direction manipulation keys, and the start input may be manipulated using the two direction manipulation keys or one direction manipulation key.

The two direction manipulation keys include a first direction manipulation key and a second direction manipulation key, and the starting input is by pressing the lower right direction key of the first direction manipulation key and the lower left direction key of the second direction manipulation key. can be entered as

The two direction manipulation keys include a first direction manipulation key and a second direction manipulation key, and the start input is by pressing any direction key of the first direction manipulation key or any direction key of the second direction manipulation key. can be entered as

The two direction manipulation keys include a first direction manipulation key and a second direction manipulation key, and the first direction manipulation key includes up, down, left, right, four direction keys, and pressing the upper direction key of the first direction manipulation key. In response, the drone performs an upward operation, and in response to pressing the downward direction key of the first direction control key, the drone performs a downward operation, and in response to pressing the left direction key of the first direction operation key , The drone performs an in-place rotation operation in a counterclockwise direction, and in response to pressing the right direction key of the first direction control key, the drone performs a stationary rotation operation in a clockwise direction, and the second direction operation key is four direction keys, up, down, left and right, and in response to pressing the upper direction key of the second direction manipulation key, the drone performs a forward operation, and in response to pressing the lower direction key of the second direction manipulation key, the drone performs a backward operation, and in response to pressing the left direction key of the second direction operation key, the drone performs a left movement operation, and in response to pressing the right direction key of the second direction operation key, the drone A right movement operation may be performed.

The virtual reality drone operation method may include, when the counting of the preset time is completed, operating a timer for recording a play time; and stopping the timer to output the play time when the disqualification signal is output or the drone arrives at the destination.

The virtual reality drone operation method may include: determining whether the drone deviates from a designated route; and outputting a penalty signal when the drone deviates from the designated route.

When the departure input is received after the preset time, moving the drone based on the departure input may include: when the forward input is received, moving the drone forward while displaying the drone to be tilted forward by a first angle; when a reverse input is received, moving the drone backward while displaying to be tilted backward by a second angle; when a left movement input is received, moving the drone to the left while displaying the drone to be inclined to the left by a third angle; and when a right movement input is received, moving the drone to the right while displaying the drone to be tilted to the right by a fourth angle.

A computer-readable storage medium in which a program according to an embodiment of the present invention is recorded, when the program is executed on a computer, the program causes the computer to operate a virtual reality drone according to an embodiment of the present invention Let the steps be executed.

A computer program stored in a computer-readable storage medium according to an embodiment of the present invention enables the computer to perform the virtual reality drone operation method according to an embodiment of the present invention when the computer program is executed on the computer .

By using the virtual reality drone simulator provided in the present application, it is possible to learn how to operate a drone without using an actual drone.

The virtual reality drone simulator provided in the present application uses objects to display the drone flight path, so that the user can learn precise and sophisticated drone operation.

The virtual reality drone simulator provided in the present application provides screens of various modes of the user's first-person view, third-person view, drone FPV mode, and manipulator mode, so that the user can efficiently learn to operate the drone.

In the virtual reality drone simulator provided here, the same rules applied to real drone racing are applied, so the user can learn to operate the drone regardless of time and place.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings are incorporated in, and are made a part of, this specification, illustrate embodiments in accordance with the present disclosure, and together with the specification serve to explain the principles of the present disclosure.
1 shows a schematic block diagram of a virtual reality drone system according to an embodiment of the present application.
2 shows a schematic diagram of a controller according to an embodiment of the present application.
3 shows a schematic diagram of a controller according to another embodiment of the present application.
4 shows a schematic diagram of a controller according to another embodiment of the present application.
5 shows a schematic diagram of a display according to an embodiment of the present application.
6 shows a schematic diagram of a display according to another embodiment of the present application.
7 is a flowchart illustrating a method of operating a virtual reality drone according to an embodiment of the present application.
8 is a flowchart of a virtual reality drone training method according to an embodiment of the present application.
9 shows an example of a state in which an object is arranged as viewed from above according to an embodiment of the present application.
10 is a flowchart of a method for displaying a virtual reality drone according to an embodiment of the present application.
11 illustrates an example of a user first-person view screen according to an embodiment of the present application.
12 shows an example of a third-person view screen of a drone according to an embodiment of the present application.
13A is for explaining a configuration for switching to a drone FPV mode screen according to an embodiment of the present application.
13B shows an example of a drone FPV mode screen according to an embodiment of the present application.
14 illustrates an example of a manipulator mode screen according to an embodiment of the present application.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in this specification are only exemplified for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiment according to the concept of the present invention These may be embodied in various forms and are not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention may have various changes and may have various forms, the embodiments will be illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes changes, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from other components, for example, without departing from the scope of rights according to the concept of the present invention, a first component may be named a second component, Similarly, the second component may also be referred to as the first component.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it is understood that other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle. Expressions describing the relationship between elements, for example, “between” and “between” or “directly adjacent to”, etc., should be interpreted similarly.

At least one term is defined as a term including a singular and a plural, and even if the at least one term does not exist, it is obvious that each component may exist in the singular or plural, and may mean the singular or plural. will be. In addition, that each component is provided in singular or plural may be changed according to embodiments.

The terminology used herein is used only to describe specific embodiments, and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these embodiments. Like reference numerals in each figure indicate like elements.

1 shows a schematic block diagram of a virtual reality drone system according to an embodiment of the present application.

Referring to FIG. 1 , a virtual reality drone system 10 includes a controller 100 , a processor 200 , and a display 300 , and may operate as a simulator.

A user may control a virtual drone in virtual reality (VR) by using the virtual reality drone system 10 . For example, a user may use the controller 100 to control a drone in virtual reality. Virtual reality may be implemented using the Unreal engine. Alternatively, the user may use the virtual reality drone system 10 to operate a virtual drone in augmented reality (AR).

The controller 100 may include two direction manipulation keys, a first direction manipulation key and a second direction manipulation key. In this case, the direction manipulation key may be implemented as a touch type, a button type, a wheel type, or a joystick type. The direction manipulation key may include up/down/left/right direction keys. The direction manipulation key may further include a diagonal direction key. Alternatively, when the direction manipulation key is a touch type, a wheel type, or a joystick type, the user may control the direction in all directions 360 degrees.

The first direction manipulation key and the second direction manipulation key may be implemented identically or may be implemented differently. For example, the first direction manipulation key may be a button type, and the second direction manipulation key may be implemented as a joystick type.

The controller 100 may be implemented as one device or two devices. For example, when the controller 100 is implemented as one device, the user may hold the controller 100 with both hands and operate two direction manipulation keys using each thumb. When the controller 100 is implemented as two devices, the user may hold the controller 100 in both hands and operate two direction manipulation keys using each thumb.

The structure of the controller 100 will be described later with reference to FIGS. 2 to 4 .

In response to the user pressing the upper direction key of the first direction control key, the drone performs an upward operation, and in response to the user pressing the downward direction key of the first direction operation key, the drone performs a descending operation, and the user In response to pressing the left direction key of the first direction control key, the drone performs a counterclockwise rotation in place, and in response to the user pressing the right direction key of the first direction control key, the drone rotates in place in a clockwise direction action can be performed.

In response to the user pressing the upper direction key of the second direction control key, the drone performs a forward operation, and in response to the user pressing the lower direction key of the second direction control key, the drone performs a backward operation, and the user In response to pressing the left direction key of the second direction control key, the drone may perform a left movement operation, and in response to the user pressing the right direction key of the second direction operation key, the drone may perform a right movement operation.

The user may operate the first direction manipulation key with the left hand and the second direction manipulation key with the right hand, but vice versa.

The user can start the drone by using the controller 100 , start it and move it.

The user may start the drone by using the first direction control key and the second direction control key. For example, the user may start the engine by pressing the lower right direction key (↘) of the first direction manipulation key and the lower left direction key (↙) of the second direction manipulation key. If there is no diagonal direction key in the controller 100, press the right key (→) and the lower key (↓) of the first direction control key, and the left key (←) and the lower key (↓) of the second direction control key You can turn on (start the engine) by pressing .

When the direction manipulation key of the controller 100 is a wheel type, the drone can be started by rolling the first direction manipulation key in the lower right direction and the second direction manipulation key in the lower left direction.

When the direction manipulation key of the controller 100 is a touch type, the drone can be started by touching the lower right side of the first direction manipulation key and touching the lower left side of the second direction manipulation key.

When the direction manipulation key of the controller 100 is a joystick type, the drone can be started by pushing the first direction manipulation key in the lower right direction and pushing the second direction manipulation key in the lower left direction.

The processor 200 may rotate the propeller of the drone in response to the received start input. A plurality of propellers may be implemented.

A method of turning off the ignition may be the same as a method of turning on the ignition. When a start-off input is received in a state in which the drone is started, the processor 200 may stop the rotating propeller of the drone. In this case, when a start-off input is received while the drone is flying, the processor 200 may free-fall the drone. It is also possible to start the drone in free fall and perform the operation again.

In the above description, for convenience of explanation, the engine is started by pressing the lower right direction key (↘) of the first direction operation key and the lower left direction key (↙) of the second direction operation key. However, it is not necessarily limited thereto, It will be said that it is possible to start the engine by using various other keys, such as pressing the lower direction key (↓) of the first direction operation key and the lower direction key (↓) of the second direction operation key.

The user may start the drone using the first direction control key and/or the second direction control key. That is, the user may start the drone by raising the drone or moving it forward, backward, left, or right. As described above, for the movement motion after the drone departs, upward, downward, counterclockwise rotation, clockwise rotation, forward, backward, leftward movement, and rightward movement may be applied.

The controller 100 may include a mode input key.

The mode input key may be implemented as one or a plurality of mode input keys.

For example, when one mode input key is implemented, the user may change the viewpoint by pressing the mode input key once. For example, whenever the user presses the mode input key, the user's first-person view screen, the drone third-person view screen, the drone FPV (Fist Person View) mode screen, and the remote controller mode screen may be changed in the order of the screen. It is obvious that the order may be changed.

When a plurality of mode input keys are implemented, for example, four, a viewpoint may be preset for each button. The first mode input key corresponds to the user's first-person view screen, the second mode input key corresponds to the drone third-person view screen, and the third mode input key corresponds to the drone FPV mode screen, and the fourth mode In the case of the input key, the remote controller mode screen may correspond.

The controller 100 may transmit an input received from the user to the processor 200 .

The processor 200 may be configured to provide screens of various viewpoints based on a user's input, or to start and start the drone based on the user's input and move it.

Alternatively, the processor 200 may provide a viewpoint option to the user via the display 300 . In response to a user selecting a viewpoint using a direction manipulation key of the controller 100 , the processor 200 may provide a corresponding viewpoint screen through the display 300 .

Alternatively, the display 300 may be implemented as a monitor and a head mounted display (HMD) device. The processor 200 tracks a user's motion from a sensor of the HMD device, and a viewpoint based on the tracked user's motion. A screen may be provided on the display 300 . The HMD device can wrap VR glasses, VR helmets, and the like. For example, the processor 200 may output an HMD wearing instruction to the monitor on a user first-person view screen, a drone third-person view screen, or a manipulator mode screen. The HMD wearing instruction may be in the form of an HMD icon. When the HMD device notifies the processor 200 that the user wears the HMD device, the processor 200 outputs a user first-person view screen, a drone third-person view screen, or a remote controller mode screen, etc. The screen can be output to the HMD device. That is, the user may watch the drone FPV mode screen after wearing the HMD device while watching the user first-person view screen, the drone third-person view screen, or the remote controller mode screen on the monitor. Accordingly, the user can play the drone from a dynamic and vivid first-person view of the drone.

The processor 200 may communicate with a server or memory to output a screen on the display 300 . For example, the HMD device is configured to receive a VR video stream transmitted from a server (or memory) by the processor 200 , and play an image based on the VR video stream, to provide the user with the ability to watch the VR video. do. The VR device may alternatively be any other device that can be used to play VR video.

The server is configured to store the VR video resource and provide a VR video stream to the HMD device. The server may be a server, a server cluster including a plurality of servers, or a cloud computing service center. Optionally, the server may be a background server configured to provide a network VR service, such as a background server configured to provide a network VR service website or application.

A communication connection may exist between the HMD device and the server. The communication connection may be a wireless network connection or may be a wired network connection.

In this case, in the case of the user's first-person view screen, the processor 200 may output a screen having a different resolution according to the user's field of view (FOV) obtained from the HMD device. For example, the processor 200 may divide a 360-degree viewing angle into a predetermined number of viewing angles, and output a screen corresponding to the obtained user's viewing angle in high resolution and a screen other than the user's viewing angle in low resolution. Accordingly, bandwidth occupancy may be reduced, and overhead may be reduced. In addition, the screen can be output at the same resolution on the drone third-person view screen, the drone FPV mode screen, and the remote controller mode screen.

The processor 200 may arrange a plurality of objects to train the user to operate the drone. The object may include an opening through which the drone may pass. For example, the object may be implemented in a diamond shape, a square shape, a ring shape, or the like. Objects may not overlap with drones. Accordingly, when the drone collides with an object, the object is floating and the direction and speed of the drone may change. For example, if the drone collides head-on with an object while moving forward, the drone may be bounced off in the opposite direction while decreasing its speed. If a part of the drone collides with an object while moving forward, it may bounce in the opposite direction to the colliding side and move forward.

Alternatively, the object may be implemented as a sphere without openings. In this case, when the drone touches the object, it may disappear without affecting the movement path of the drone. That is, the object may be overlapped with the drone.

The plurality of objects may include a first object, a second object, and a third object. A second object may be disposed after the first object, and a third object may be disposed after the second object.

The first object and the second object may have different rotation angles and/or heights. For example, the first object may be rotated 10m above the ground by 45 degrees to the left, and the second object may be rotated 10m above the ground by 45 degrees to the right. As another example, the first object may be rotated 10 m above the ground by 0 degrees, and the second object may be rotated 50 m above the ground by 90 degrees. In the above description, the first object and the second object have been described as a reference for convenience of description, but the present invention is not limited thereto, and all of the plurality of objects may have different rotation angles and/or heights. Of course, all of the plurality of objects may have the same rotation angle and/or height.

The processor 200 may display an indicator at the center of an object in an order through which the drone will pass among the plurality of objects. The indicator may have the same shape as the object, and may overlap with the drone. That is, the indicator may not collide with the drone.

The processor 200 may display the indicator in the center of the object in the next order when the drone comes into contact with the indicator while passing the object in the order through which the drone will pass.

The processor 200 may display a guide path passing through the center of each of the plurality of objects. The guide path may be overlapping with the drone.

For example, the guide path may be fixed at the center of the first object and at the center of the second object, and a region from the center of the first object to the center of the second object may vibrate dynamically.

As another example, the guide path may be fixed without moving while connecting the center of the first object, the center of the second object, and the center of the third object as a curve.

The processor 200 may determine whether the drone deviates from a designated path.

For example, when the drone passes without passing through the object, the processor 200 may determine that the drone deviates from a designated path. For example, when the drone passes the first object without passing through the second object, the processor 200 may determine that the drone deviates from a designated path. For example, the processor 200 may determine that the drone has deviated from a designated path when the drone does not pass through an object equal to or greater than a threshold and passes the next object.

As another example, the processor 200 may calculate a vertical distance from the drone to the guide path, and if the calculated vertical distance is equal to or greater than a threshold, it may be determined that the drone deviated from the designated path.

The processor 200 may be configured to output a penalty signal when determining that the drone deviated from a designated route. For example, the penalty signal may be a signal to return, a signal to notify that an object has been skipped, a signal that play time is added, and the like.

When the processor 200 determines that the drone deviated from the designated route, the processor 200 may relocate (or respawn) the drone to a location prior to departure. Alternatively, when the processor 200 determines that the drone deviated from the designated route, the processor 200 may relocate to the object that the drone last passed through.

When the drone leaves the object once or twice, the processor 200 may add a penalty time to the play time. For example, the penalty time may be 5 seconds. The processor 200 may output a disqualification signal when the drone deviates from the object three or more times.

When the angle formed by the first object, the second object, and the third object is less than a threshold, the processor 200 may display a direction change instruction around the second object. For example, when the third object is located to the right of the second object, the processor 200 may display a direction change instruction instructing to rotate to the right (yaw, yaw) around the second object. . When the third object is located to the left of the second object, the processor 200 may display a direction change instruction instructing to rotate to the left around the second object. The direction change instruction may include an image of the drone, an action to be performed (eg, ascend, descend, turn left, turn right, etc.), an arrow, and the like.

When the drone passes the first object and the third object is not in the field of view, the processor 200 may display a direction change instruction around the second object. For example, the processor 200 may display a direction change instruction based on the position of the third object.

When the first object is disposed at a corner of a building, the processor 200 may display a direction change instruction around the first object. The processor 200 may display a direction change instruction when the moving path is not straight while the first object is disposed at the corner of the building. That is, the processor 200 may not display the direction change instruction when the first object is disposed at the corner of the building but the progress path is straight.

The processor 200 may display a descending instruction around the last object among the plurality of objects. In this case, the endpoint may be vertically below the last object.

When the angle between the first object, the second object, and the end point is less than a threshold value, the processor 200 may display a descending instruction around the second object. For example, the threshold may be 180 degrees. The drone may perform descent based on the descent instruction and turn off the engine after landing at the destination.

When the drone moves in a direction opposite to the displayed direction change instruction, the processor 200 may output the direction change instruction until the drone moves in the correct direction.

The display 300 may output a drone control screen to the user in response to a command of the processor 200 . The display 300 may be implemented as a television, monitor, or HMD device. That is, when the display 300 provides a general image to the user, the display 300 is a television or monitor, and when the display 300 provides virtual reality or augmented reality to the user, the display 300 is an HMD device. can be

When the display 300 is implemented as an HMD device, the processor 200 may track the movement of the user's head (3 degrees of freedom, 3DoF) based on a sensor in the HMD device (head tracking, head tracking). In addition, the processor 200 may track the user's three-dimensional positional movement (6 degrees of freedom, 6DoF) based on the sensor.

The processor 200 may output a screen based on the movement of the user's head obtained from the sensor, the movement of the user's 3D position, and the mode input obtained from the controller 100 .

The structure of the display 300 will be described later with reference to FIGS. 5 and 6 .

2 shows a schematic diagram of a controller according to an embodiment of the present application.

Referring to FIG. 2 , the controller 101 may include a first direction manipulation key 110 , a second direction manipulation key 120 , and a mode input key 130 .

As the first direction manipulation key 110 , the second direction manipulation key 120 , and the mode input key 130 are the same as those described with reference to FIG. 1 , the detailed description will not be repeated again.

The first direction manipulation key 110 and the second direction manipulation key 120 may be button type. In this case, the terminals may be present in four upper, lower, left, and right portions of the first direction operation key 110 and the second direction operation key 120 , or the terminals may be present in a ring shape to enable input in all directions by 360 degrees.

The first direction manipulation key 110 and the second direction manipulation key 120 may be implemented as a touch type, a wheel type, or a joystick type.

The mode input key 130 may be implemented as four buttons. Alternatively, it may be implemented as one button or two buttons, or implemented in a slide manner.

3 shows a schematic diagram of a controller according to another embodiment of the present application.

Referring to FIG. 3 , the controller may be implemented as a first controller 103 and a second controller 105 .

The first controller 103 includes a first direction operation key 140 and a mode input key 160 , and the second controller 105 includes a first direction operation key 150 and a mode input key 160 . can do.

As the first direction manipulation key 140 , the second direction manipulation key 150 , and the mode input key 160 are the same as those described with reference to FIGS. 1 and 2 , the detailed description will not be repeated again.

Alternatively, the first controller 103 and the second controller 105 may include a gyro-sensor. That is, the processor of the virtual reality drone system may control the drone based on the inclination of the first controller 103 and the second controller 105 .

4 shows a schematic diagram of a controller according to another embodiment of the present application.

Referring to FIG. 4 , the controller 107 may include a first direction manipulation key 170 , a second direction manipulation key 180 , and a mode input key 190 .

As the first direction manipulation key 170 , the second direction manipulation key 180 , and the mode input key 190 may be applied in the same manner as described with reference to FIGS. 1 and 2 , the detailed description will not be repeated again.

5 shows a schematic diagram of a display according to an embodiment of the present application.

Referring to FIG. 5 , the display 301 may be implemented as an HMD device to provide virtual reality or augmented reality to a user.

The display 301 may include a sensor 310 . The sensor 310 may track the movement of the user's head and/or the movement of the 3D position.

The processor of the virtual reality drone system displays the user's field of vision in virtual reality or an image obtained from the drone's camera on the basis of the movement of the user's head and/or the three-dimensional position movement obtained from the sensor 310 . It can be output through

Accordingly, the virtual reality drone system displays a user first-person view screen, a drone third-person view screen, a drone FPV mode screen, or a manipulator mode screen based on the mode input from the user, the movement of the user's head, and the 3D position movement. It may be output through the display 301 .

The user first-person view screen may be a view that the user sees while moving in virtual reality or augmented reality. In the user first-person view screen, the user can move in virtual reality or augmented reality to discover the drone, and use the controller to control the drone. When the drone moves out of view, the view of the drone may be checked by checking the monitor of the electronic device in the virtual reality. Also, when the drone is in flight, the user can control the flight of the drone by changing to a third-person view, drone FPV mode, or remote controller mode.

The third-person view screen may be a mode for observing the drone's flight from behind the drone. When the user moves the head up, down, left, and right on the third-person view screen, the processor may instruct the display 310 to output the up, down, left, and right of the drone.

The drone FPV mode may be a mode in which an image captured by a camera of the drone is output. In the drone FPV mode, even if the user moves the head up, down, left and right, the processor may instruct the display 310 to output only the image captured by the drone's camera. That is, in the drone FPV mode, the sensor of the display 310 may not operate.

In the remote controller mode, an image captured by the drone's camera can be checked through the display of the electronic device in virtual reality. In the manipulator mode, even if the user moves the head up, down, left and right, the processor may instruct the display 310 to output only the display of the virtual reality. That is, the sensor of the display 310 may not operate in the manipulator mode.

6 shows a schematic diagram of a display according to another embodiment of the present application.

Referring to FIG. 6 , the display 303 may be implemented as a television or a monitor to provide a general image to a user.

The virtual reality drone system may output a user first-person view screen, a drone third-person view screen, a drone FPV mode screen, or a manipulator mode screen through the display 303 based on a mode input from the user.

The configuration described with reference to FIG. 5 may be identically applied to the user first-person view screen, the drone third-person view screen, the drone FPV mode screen, and the remote controller mode screen.

7 is a flowchart illustrating a method of operating a virtual reality drone according to an embodiment of the present application.

Referring to FIG. 7 , when a start input is received in virtual reality, the drone may be started ( 710 ). For example, you can start the drone using the two directional keys. When the drone is started, the propeller of the drone rotates, and a flight sound may be output. Turning off the drone can also be performed using the same direction control key. If the drone engine is turned off, the propeller stops and the flight sound may not be output. If the engine is turned off during flight, the drone can freely fall.

A preset time may be counted ( 720 ). The preset time may be 3 seconds.

It may be determined whether a departure input has been received within a preset time ( 730 ). In drone racing, by providing a preset time, it is possible for a user to have preparation time and to enable fair and accurate time recording.

It is irrelevant whether or not the start input is received within the preset time. The start input can be entered at any time.

When the departure input is received within a preset time, it may be determined whether the number of times received is one ( 740 ).

When the departure input is received once within a preset time, a warning signal may be output ( 750 ). After the step of outputting the warning signal, the preset time may be counted again from the beginning.

When the departure input is received two or more times within a preset time, a disqualification signal may be output ( 760 ). If the disqualification signal is output, the game may end.

If the departure input is not received within the preset time (ie, if the departure input is received after the preset time), the drone may be moved based on the departure input ( 770 ). In this case, when the counting of the preset time is completed, a timer for recording the play time may be operated. When a disqualification signal is output or the drone arrives at the destination, the timer can be stopped to output the play time.

The departure input may include direction information to move the drone. The start input may be operated using two direction control keys or one direction control key.

For example, the two direction manipulation keys include a first direction manipulation key and a second direction manipulation key, and the first direction manipulation key includes four up, down, left and right direction keys, and pressing the upper direction key of the first direction manipulation key. In response, the drone performs an ascending operation, in response to pressing the down direction key of the first direction control key, the drone performs a descending operation, and in response to pressing the left direction key of the first direction operation key, the drone The drone performs an in-place rotation operation in a counterclockwise direction, and in response to pressing the right direction key of the first direction control key, the drone performs a stationary rotation operation in a clockwise direction, and the second direction operation key includes four up, down, left, and right direction keys. and in response to pressing the upper direction key of the second direction control key, the drone performs a forward operation, and in response to pressing the lower direction key of the second direction operation key, the drone performs a backward operation, and a second direction operation In response to pressing the left direction key of the key, the drone may perform a left movement operation, and in response to pressing the right direction key of the second direction manipulation key, the drone may perform a right movement operation.

While the drone is moving (flying), it can be determined whether the drone deviates from a designated route. When the drone deviates from the designated route, a penalty signal may be output.

In addition, when a forward input is received, the drone may be moved forward while displaying to be inclined forward by the first angle. When a reverse input is received, the drone may be moved backward while displaying to be tilted backward by the second angle. When a left movement input is received, the drone may be moved to the left while displaying to be inclined to the left by a third angle. When a right movement input is received, the drone may be moved to the right while displaying to be inclined to the right by a fourth angle. For example, the first angle, the second angle, the third angle, and the fourth angle may all be the same. As another example, the first angle and the second angle may be the same, the third angle and the fourth angle may be the same, and the first angle and the third angle may be different.

8 shows a flowchart of a virtual reality drone training method according to an embodiment of the present application, and FIG. 9 shows an example of a state in which an object is disposed according to an embodiment of the present application as viewed from above.

Referring to FIG. 8 , a plurality of objects through which a drone passes may be arranged in virtual reality ( 810 ). The object may be implemented in a diamond shape, a rectangle, a ring shape, etc. including an opening through which the drone can pass, or may be implemented as a sphere without an opening. An example of a state in which the diamond-shaped object is arranged as viewed from the top may be as shown in FIG. 9 .

A plurality of objects may be displayed together with a guide path passing through the center of each object, the guide path may overlap the drone, and the plurality of objects may not overlap the drone. That is, the drone may collide with an object, and in the event of a collision, the object may be immobile, and the direction and speed of the drone may change.

The plurality of objects may include a first object, a second object, and a third object. A second object may be disposed after the first object, and a third object may be disposed after the second object. The third object may be the last object. After the third object, there may be an end point.

When the angle formed by the first object, the second object, and the third object is less than the threshold, a direction change instruction may be displayed around the second object. The direction change instruction may include an image of the drone, an action to be performed (eg, ascend, descend, turn left, turn right, etc.), an arrow, and the like.

When the drone passes the first object and the third object is not in view, a direction change instruction may be displayed around the second object.

When the first object is disposed at a corner of a building, a direction change instruction may be displayed around the first object. When the number of plays exceeds a certain number of times, it is determined that the user is familiar with the route, and the direction change instruction may not be displayed. For example, the predetermined number of times may be 10 times.

A descending instruction may be displayed around a third object that is the last object among the plurality of objects.

When the angle between the second object, the third object, and the end point is less than the threshold, a descending instruction may be displayed around the third object.

The guide path is fixed at the center of the first object and the center of the second object, and a region from the center of the first object to the center of the second object may vibrate dynamically.

The first object and the second object may have different rotation angles and heights.

An indicator may be displayed at the center of an object in the order in which the drone will pass among a plurality of objects, and when the drone comes into contact with the indicator while passing through an object in the order to pass, the indicator may be displayed in the center of the object in the next order.

In the above description, expressions such as first, second, and third are used for arbitrary objects for convenience of description, and this is not intended to limit the number of objects.

A direction input for operating the drone may be received ( 820 ). The direction input may be input through two direction manipulation keys.

In response to the direction input, the drone may pass through the plurality of objects ( 830 ).

In this case, it may be determined whether the drone deviates from the designated route.

As an example, when the drone passes without passing through the object, it may be determined that the drone deviated from a designated path. For example, when the drone does not pass through the first object and passes through the second object, it may be determined that the drone deviated from a designated path.

As another example, when the vertical distance from the drone to the guide path is equal to or greater than a threshold, it may be determined that the drone deviated from the designated path.

When the drone deviates from the designated route, a penalty signal may be output. For example, if the drone deviates from the object once or twice, 5 seconds may be added to the play time, and if the drone deviates from the object 3 or more times, a disqualification signal may be output.

When the drone passes the last object among the plurality of objects, a training score may be calculated based on the flight time of the drone, the number of objects the drone has passed, and the number of times the drone has collided ( 840 ). For example, subtract the play time (seconds) from the base score, add the number of times the object passed multiplied by the first weight, subtract the number of times that the object did not pass by the second weight, and A score may be calculated by subtracting a value obtained by multiplying the number of collisions by the third weight. For example, the basic score may be 1000 points, the first weight may be smaller than the second weight, and the second weight may be smaller than the third weight. In addition, when calculating the training score, whether or not the start-up of the drone (ON/OFF) may be additionally taken into consideration. For example, by counting the number of times the drone's engine is turned off during operation of the drone, it may be deducted from the score.

A training score may be output ( 850 ). Optionally, at least one of the flight time of the drone, the number of objects the drone has passed, the number of objects the drone has not passed, and the number of times the drone has collided may be output together.

A determination may be made 860 whether to proceed to the next stage based on the training score. For example, if the training score is higher than the passing reference score, it may proceed to the next stage. If the training score is lower than the passing criteria, you cannot proceed to the next stage, and you can replay the current stage.

10 shows a flowchart of a method for displaying a virtual reality drone according to an embodiment of the present application, FIG. 11 shows an example of a user first-person view screen according to an embodiment of the present application, and FIG. 12 is an example of the present application An example of a drone third-person view screen according to an embodiment is shown, FIG. 13A is for explaining a configuration for switching to a drone FPV mode screen according to an embodiment of the present application, and FIG. 13B is an embodiment of the present application An example of a drone FPV mode screen is shown according to FIG. 14 , and FIG. 14 shows an example of a manipulator mode screen according to an embodiment of the present application.

Referring to FIG. 10 , head tracking may be performed on a user who uses virtual reality ( 1010 ).

A mode input may be received from the user (1020). The user may input the mode by using the mode input key of the controller. The mode input is one of a first input, a second input, a third input, and a fourth input. When the first input is received, the user first-person view screen is provided, and when the second input is received, the drone third-person view screen is displayed. provided, when the third input is received, the drone FPV mode screen may be provided, and when the fourth input is received, the manipulator mode screen may be provided.

A screen may be provided according to the result of the head tracking and the mode input ( 1030 ).

On the provided screen, the user can control the position of the drone by using the direction control key. When the user manipulates the direction manipulation key, a corresponding fifth input may be received. That is, the position of the drone may be controlled in response to the fifth input. The fifth input includes at least one of a first upper direction key, a first lower direction key, a first left direction key, a first right direction key, a second upper direction key, a second lower direction key, a second left direction key, and a second right direction key, and , in response to pressing the first upper direction key, the drone performs an upward operation, in response to pressing the first downward direction key, the drone performs a descending operation, and in response to pressing the first left direction key, the drone moves In the opposite direction, the drone performs a standing rotation operation in the opposite direction, and in response to pressing the first right direction key, the drone performs a standing rotation operation in a clockwise direction, and in response to pressing the second up direction key, the drone performs a forward operation, In response to pressing the second down direction key, the drone performs a backward operation, in response to pressing the second left direction key, the drone performs a left movement operation, and in response to pressing the second right direction key, the drone moves to the right A movement operation can be performed.

Hereinafter, screens output in each view mode will be described with reference to FIGS. 11 to 14 .

Referring to FIG. 11 , a user first-person view screen may be provided in response to a first input from the user. In this case, information on the user's head tracking (head movement) may be received, and a screen may be output accordingly. For example, when the user bows his head, a screen illuminating the lower portion may be output, and when the user looks to the left, a screen illuminating the left may be output.

In addition, it is also possible to track the movement of the user's three-dimensional position. Accordingly, it is possible to provide a screen according to the tracked position movement, the result of the head tracking, and the mode input from the user.

When the display of the electronic device is outputted in virtual reality according to the user's positional movement and head direction, the display may output an image obtained from the drone's camera. For example, a user may move and move their head to discover an electronic device in virtual reality. The user may observe the movement of the drone through the display of the electronic device in virtual reality.

For example, an output method through the display of the electronic device in virtual reality may be as follows:

When the drone is on the ground before takeoff or hovering, it outputs a screen in the first angular direction with respect to the horizontal line, when a forward input is received, a screen lower than the first angular direction is output, and when a backward input is received, the first A screen higher than the angular direction may be output, and when a left movement input is received, the screen may be output to be inclined to the left by a second angle, and when a right movement input is received, the screen may be output to be inclined to the right by a third angle. The second angle and the third angle may be the same.

A scene output through the display of the electronic device in virtual reality may be the same as a scene output through the drone FPV mode screen.

In virtual reality, the user may move along with the flying drone through the user's first-person view screen, or may remain in an immobile posture while controlling the flying drone through the monitor of the electronic device.

Referring to FIG. 12 , a third-person view screen of the drone may be provided in response to a second input from the user.

The drone can be placed in a fixed position on the screen, and the screen can be output according to the movement of the drone. That is, the drone third-person view screen may be a mode for observing the drone's flight from behind the drone.

At this time, when the user rotates his head to the left, the left side of the drone is displayed, when the user rotates his head to the right, the right side of the drone is displayed, and when the user tilts his head to the left, tilt the screen to the left When the user tilts their head to the right, the screen is tilted to the right, when the user raises their head up, the upper side of the drone is displayed, and when the user lowers their head, the lower side of the drone is displayed. can

In addition, when a forward input is received, the drone is moved forward while displaying to be tilted forward by a first angle, and when a reverse input is received, the drone is displayed to tilt backward by a second angle and backward while displaying, and when a left movement input is received, the drone is moved to the third The drone may be moved to the left while displaying to be tilted to the left by an angle, and when a right movement input is received, the drone may be moved to the right while displaying to be tilted to the right by a fourth angle.

13A and 13B , an instruction to wear the HMD device may be output to the user.

If there is no HMD device, if the third input (eg, “A” key in FIG. 13A ) is pressed, the HMD device is recognized as being worn, and the drone FPV mode screen may be provided. In this case, the display is a monitor, and the user can operate the virtual drone through the monitor. The presence or absence of the HMD device may be determined based on whether the HMD device and the processor are paired.

If the HMD device is present, when the user wears the HMD device, the HMD device may notify the processor that the user is wearing the HMD device. The processor may provide the drone FPV mode screen to the HMD device in response to a signal indicating that the user is wearing the HMD device. That is, the user may watch the drone FPV mode screen after wearing the HMD device while watching the user first-person view screen, the drone third-person view screen, or the remote controller mode screen on the monitor. Accordingly, the user can play the drone from a dynamic and vivid first-person view of the drone. In this case, the display includes the monitor and the HMD device, and the user may operate the virtual drone through the monitor and/or the HMD device.

For example, from a user's first-person view or a drone's third-person view, it is possible to look at a drone that is stationary on the floor. Here, when a third input (eg, “A” key in FIG. 13A ) is received, a goggles icon may be output and the goggles icon may be enlarged. In this case, it may be detected whether a user other than the virtual world (in the real world) wears the HMD device. If the user is not wearing the HMD device, the goggles icon will blink, if the user is wearing the HMD device, the goggles icon will be magnified (HMD device wearing visual effect), and the image obtained from the drone’s camera can be output to the HMD device. have.

When the drone is on the ground before takeoff or hovering, it outputs a screen in the first angular direction with respect to the horizontal line, when a forward input is received, a screen lower than the first angular direction is output, and when a backward input is received, the first A screen higher than the angular direction may be output, and when a left movement input is received, the screen may be output to be inclined to the left by a second angle, and when a right movement input is received, the screen may be output to be inclined to the right by a third angle. The second angle and the third angle may be the same.

The drone FPV mode screen may be the same as a scene output through the display of the electronic device in virtual reality from the user's first-person view.

Even while the user is operating the drone in drone FPV mode through the HMD device, the processor can provide a screen to the monitor. In this case, the screen output to the monitor may be one of a user first-person view screen, a drone third-person view screen, a remote controller mode screen, or a drone FPV mode screen.

Referring to FIG. 14 , a manipulator mode screen may be provided in response to a fourth input from the user.

The electronic device may be displayed in virtual reality.

An image obtained from the drone's camera may be displayed on the display of the electronic device. At this time, when the user rotates his head to the left, the left side of the display is displayed, when the user rotates his head to the right, the right side of the display is displayed, and when the user tilts his head to the left, tilt the screen to the left When the user tilts their head to the right, the screen is tilted to the right, when the user lifts their head up, the upper side of the display is displayed, and when the user lowers their head, the lower side of the display is displayed. can

The display can further show the drone's flight time, battery status, altitude, distance traveled, and speed.

A scene output through the display of the electronic device in virtual reality may be the same as a scene output through the drone FPV mode screen.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (14)

starting the drone upon receiving a start input in virtual reality;
counting a preset time;
outputting a warning signal when a departure input is received once within the preset time;
outputting a disqualification signal when the departure input is received two or more times within the preset time; and
If a departure input is received after the preset time, moving the drone based on the departure input
including,
The departure input includes direction information,
How to operate a virtual reality drone.
According to claim 1,
Starting the drone upon receiving the start input in the virtual reality comprises:
rotating the propeller of the drone
A virtual reality drone operation method comprising a.
According to claim 1,
When the drone receives a start-off (OFF) input while moving, stopping the rotation of the propeller of the drone and free-falling the drone
A virtual reality drone operation method further comprising a.
According to claim 1,
After outputting the warning signal, counting the preset time again from the beginning
A virtual reality drone operation method further comprising a.
According to claim 1,
The preset time is 3 seconds,
How to operate a virtual reality drone.
According to claim 1,
The start input is operated using two directional control keys,
The start input is operated using the two direction control keys or one direction control key,
How to operate a virtual reality drone.
7. The method of claim 6,
The two direction manipulation keys include a first direction manipulation key and a second direction manipulation key,
The starting input is
Input by pressing the lower right direction key of the first direction operation key and the lower left direction key of the second direction operation key,
How to operate a virtual reality drone.
7. The method of claim 6,
The two direction manipulation keys include a first direction manipulation key and a second direction manipulation key,
The starting input is
input by pressing an arbitrary direction key of the first direction operation key or an arbitrary direction key of the second direction operation key,
How to operate a virtual reality drone.
7. The method of claim 6,
The two direction manipulation keys include a first direction manipulation key and a second direction manipulation key,
The first direction operation key includes four up, down, left and right direction keys,
In response to pressing the upper direction key of the first direction control key, the drone performs an ascending operation,
In response to pressing the down direction key of the first direction manipulation key, the drone performs a lowering operation,
In response to pressing the left direction key of the first direction operation key, the drone performs a counterclockwise rotation in place,
In response to pressing the right direction key of the first direction operation key, the drone performs a clockwise rotation in place,
The second direction manipulation key includes four up, down, left and right direction keys,
In response to pressing the upper direction key of the second direction manipulation key, the drone performs a forward operation,
In response to pressing the lower direction key of the second direction manipulation key, the drone performs a reverse operation,
In response to pressing the left direction key of the second direction manipulation key, the drone performs a left movement operation,
In response to pressing the right direction key of the second direction operation key, the drone performs a right movement operation,
How to operate a virtual reality drone.
According to claim 1,
When the counting of the preset time is completed, operating a timer for recording the play time; and
When the disqualification signal is output or the drone arrives at the destination, stopping the timer to output the play time
A virtual reality drone operation method further comprising a.
11. The method of claim 10,
determining whether the drone deviates from a designated route; and
When the drone deviates from the designated route, outputting a penalty signal
A virtual reality drone operation method further comprising a.
According to claim 1,
If a departure input is received after the preset time, moving the drone based on the departure input includes:
when a forward input is received, moving the drone forward while displaying it to be tilted forward by a first angle;
when a reverse input is received, moving the drone backward while displaying to be tilted backward by a second angle;
when a left movement input is received, moving the drone to the left while displaying to be inclined to the left by a third angle; and
When a right movement input is received, moving the drone to the right while displaying the drone to be tilted to the right by a fourth angle
A virtual reality drone operation method comprising a.
A computer-readable storage medium having a program recorded therein, comprising:
13. A computer-readable storage medium, when the program is executed on a computer, the program causes the computer to execute the steps of the method of any one of claims 1 to 12.
A computer program stored in a computer-readable storage medium, comprising:
A computer program, which, when the computer program is executed on a computer, makes the computer capable of performing the method according to any one of claims 1 to 12.

Images