CN113534835B - A kind of tourism virtual remote experience system and method - Google Patents

Abstract

The present invention provides a system and method for virtual remote experience of tourism, including: an unmanned aerial vehicle, wherein the unmanned aerial vehicle is equipped with a binocular camera, which is used to collect panoramic video frames in real time; a sight tracking module, which uses a trained deep neural network model Obtain the line of sight direction of the user; a flight control module, the flight control module uses the tracked line of sight direction to control the flight direction of the unmanned aerial vehicle to realize remote flight control of the unmanned aerial vehicle; the server processing module is used to convert the binocular The panoramic video frames collected by the camera are processed and sent to the naked-eye 3D display screen; the naked-eye 3D display screen is used to display information and provide users with an immersive flight experience. The invention can realize the control of the drone by using the sight line, and cooperate with the 5G high-speed network and the naked-eye 3D display screen to obtain an excellent immersive experience.

Description

A kind of tourism virtual remote experience system and method

technical field

The invention relates to tourism experience technology, in particular to a virtual remote experience system and method for tourism, and the technology involved includes the technical fields of drone control, sight tracking, deep learning and the like.

Background technique

With the improvement of people's living standards, people's demand for tourism is growing rapidly. However, with the impact of the global epidemic, it has brought great harm to the tourism industry. To avoid aggregation, many amusement scenes are closed. For example: Guangzhou Tower, a famous scenic spot, was temporarily closed due to the impact of the local epidemic in May-June 2021 and will not accept tourists.

Therefore, virtual tourism has become a real demand of the people, and compared with traditional tourism, virtual tourism has many advantages, such as no need to travel, no need to crowd, low carbon and environmental protection, low time cost, fast and efficient, etc. There are also some methods related to virtual tours in the existing technology, but the control experience of virtual tours is not good, and manual control mode is mostly used for interaction, which lacks immersive experience, and the visited attractions are limited. Modeling, you can only participate in the tourism of this fixed tourist attraction, the modeling scene is not a real scene, and it lacks an immersive experience.

The system and method for virtual remote experience of tourism proposed by the present invention realize information collection of tourism scenes by means of unmanned aerial vehicles, track the user's sight line, control the flying direction and speed of unmanned aerial vehicles, and achieve the state of man-machine integration. , the user feels that he is an aircraft, not just controlling the aircraft, and the existing technology only passively receives the drone information, and cannot achieve the state of man-machine integration through the sight tracking technology. Combined with the naked-eye 3D display screen, the real Immersive experience effect.

The innovation of the present invention is mainly reflected in:

1) A travel virtual remote experience system and method proposed by the present invention is suitable for long-distance virtual travel, introducing drones to participate in virtual travel, avoiding the single problem of traditional virtual travel tourism scenes, and because the drones fly flexibly, A close-up and/or distant observation can be made at any time according to the user's viewpoint.

2) The present invention adopts the method of deep learning to realize sight tracking, so as to control the flight state of the UAV, and perform speed control and direction control. The proposed deep learning method can solve the problem of tracking the user, and then convert it into a flight control command. , so that participants can get an immersive virtual tourism experience that integrates human and machine.

SUMMARY OF THE INVENTION

The present invention provides a travel virtual remote experience system and method, the system includes:

Unmanned aerial vehicle, the unmanned aerial vehicle is equipped with a binocular camera for real-time acquisition of panoramic video frames;

The gaze tracking module uses the trained first deep neural network model to obtain the user's gaze direction;

A flight control module, the flight control module utilizes the tracked direction of sight to control the flight direction of the unmanned aerial vehicle, so as to realize remote flight control of the unmanned aerial vehicle;

The server processing module is used to process the panoramic video frames collected by the binocular camera and send them to the naked-eye 3D display screen;

The naked-eye 3D display screen is used to display information and provide users with an immersive flight experience.

Optionally, the UAV and the server processing module perform high-speed communication through a 5G communication module, and send the panoramic video frame data obtained by the binocular camera to the server processing module.

Optionally, the experience system further includes: a 3D speaker module for playing audio information captured during the flight of the drone.

Optionally, the control module controls the flying direction and speed of the UAV through the line of sight direction and the line of sight concentration; the higher the line of sight concentration, the faster the forward direction; the flight direction is adjusted following the line of sight direction.

Optionally, the system also has a console module, which realizes remote control of the UAV through each operation button; and/or a voice control module, which realizes the remote control of the UAV through voice commands; and/or a gesture control module , to realize remote control of the drone through the recognized gestures.

Correspondingly, the present invention also proposes a virtual travel remote experience method, which is characterized in that:

UAV is used to collect panoramic video frames in real time, and the UAV is equipped with a binocular camera;

Utilize the gaze tracking module to obtain the user's gaze direction, and the gaze direction is obtained by using the trained first deep neural network model;

Utilize the flight control module to realize the remote flight control to the unmanned aerial vehicle, and the described flight control module utilizes the described line-of-sight direction to control the flight direction of the unmanned aerial vehicle;

Use the server processing module to process the panoramic video frames collected by the binocular camera and send them to the naked-eye 3D display screen;

The naked eye 3D display screen is used for information display, providing users with an immersive flight experience.

Optionally, the UAV and the server processing module perform high-speed communication through a 5G communication module, and send the panoramic video frame data obtained by the binocular camera to the server processing module.

Optionally, the method further includes: using a 3D speaker module to play audio information captured during the flight of the drone.

Optionally, the method further includes: the control module controls the flying direction and speed of the UAV by using the line of sight direction and the line of sight concentration; the higher the line of sight concentration, the faster the forward direction; the flight direction is adjusted following the line of sight direction.

Optionally, the method further includes: using a console module to realize remote control of the UAV, and realizing the remote control of the UAV through each operation button; and/or using a voice control module to realize the remote control of the UAV , realize the remote control of the UAV through voice commands; and/or realize the remote control of the UAV by using the gesture control module, and realize the remote control of the UAV through the recognized gestures.

Beneficial effects:

1) A kind of tourism virtual remote experience system and method proposed by the present invention is suitable for remote virtual tourism, introduces drones to participate in virtual tourism, and avoids the single problem of traditional virtual tourism tourism scenes, such as can fly Canton Tower, Baiyun Mountain , Pearl River Day/Night View, etc., and due to the flexible flight of the drone, close-range and/or long-distance observation can be carried out at any time according to the user's viewpoint.

2) The present invention adopts the method of deep learning to realize sight tracking, so as to control the flight state of the UAV, and perform speed control and direction control. The proposed deep learning method can solve the problem of tracking the user, and then convert it into a flight control command. , so that participants can get an immersive virtual tourism experience that integrates human and machine.

Description of drawings

Figure 1 is a functional schematic diagram of a travel virtual remote experience system.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer and clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

As shown in Figure 1, the present invention proposes a virtual travel remote experience system, the system includes:

Unmanned aerial vehicle, the unmanned aerial vehicle is equipped with a binocular camera for real-time acquisition of panoramic video frames;

The gaze tracking module uses the trained first deep neural network model to obtain the user's gaze direction;

A flight control module, the flight control module utilizes the tracked direction of sight to control the flight direction of the unmanned aerial vehicle, so as to realize remote flight control of the unmanned aerial vehicle;

The server processing module is used to process the panoramic video frames collected by the binocular camera and send them to the naked-eye 3D display screen;

The naked-eye 3D display screen is used to display information and provide users with an immersive flight experience.

Optionally, the UAV and the server processing module perform high-speed communication through a 5G communication module, and send the panoramic video frame data obtained by the binocular camera to the server processing module.

Optionally, the experience system further includes: a 3D speaker module for playing audio information captured during the flight of the drone.

Optionally, the control module controls the flying direction and speed of the UAV through the line of sight direction and the line of sight concentration; the higher the line of sight concentration, the faster the forward direction; the flight direction is adjusted following the line of sight direction.

Optionally, the system also has a console module, which realizes remote control of the UAV through each operation button; and/or a voice control module, which realizes the remote control of the UAV through voice commands; and/or a gesture control module , to realize remote control of the drone through the recognized gestures.

Optionally, if the control distance of the drone can be met, the user can configure the drone by himself; when the conditions do not allow, he can also rent the drone of the scenic spot, and the user can rent the drone after paying for it. The integrated control center of scenic spots is responsible for the management of drones. For example, if there are multiple drones in the air that may conflict with their flying positions, an alarm will be given in advance, and the required flight position and speed will be given. In emergency situations The lessor directly obtains the flight permission to avoid the collision of different drones. After the risk is removed, the authority will be returned to the user.

Optionally, the first deep neural network is used for tracking the sight line, judging its changing state, and then converting it into flight control instructions, such as: left, right, up, down, and so on.

Optionally, the first deep neural network is a DRCNN network, and the DRCNN includes: one or more convolution layers, one or more pooling layers, and a fully connected layer; the convolution kernel size used by the convolution layer is 3 *3; The excitation function used by the DRCNN is the sigmod function;

Optionally, described DRCNN utilizes sine exponential loss function (Sine-Index-Softmax) to improve the accuracy of line of sight tracking; Described sine exponential loss function is:

其中，θ_yi表示为样本i与其对应标签y_i的向量夹角，其中b_yi表示样本i在其标签y_i处的偏差，b_j表示输出节点j处的偏差；所述N表示训练样本个数；所述w_yi表 示样本i在其标签y_i处的权重。

Among them, θ _yi represents the vector angle between sample _i and its corresponding label yi, where b _yi represents the deviation of sample i at its label y _i , and b _j represents the deviation of output node j; the N represents the number of training samples number; the w _yi represents the weight of sample _i at its label yi.

Optionally, the pooling method of the pooling layer is as follows:

S=f(elogw+LOSS _SIS );

Among them, s represents the output of the current layer, f() represents the activation function, and w represents the weight of the current layer.

Optionally, the line of sight concentration is obtained through a second deep neural network, and the second deep neural network is realized by an attention mechanism, and the second deep neural network is specifically an attention neural network. Convolutional features of the first neural network; can also be trained independently to obtain convolutional features suitable for its model itself. The attentional neural network divides the user's attention into multiple speed levels. Optionally, the speed levels are 1, 2, 3, 4, 5, 6, 7...N. The serial number represents the speed level. The smaller the number is, the faster the flight speed is. On the contrary, the larger the number is, the slower the flight speed is.

The excitation function adopted by the attention neural network is the cosine exponential excitation function, denoted as g(), where

Among them, θ _yi represents the vector angle between the sample _i and its corresponding label yi; the N represents the number of training samples; the w _yi represents the weight of the sample _i at its label yi.

Correspondingly, the present invention also proposes a virtual travel remote experience method, which is characterized in that:

UAV is used to collect panoramic video frames in real time, and the UAV is equipped with a binocular camera;

Utilize the gaze tracking module to obtain the user's gaze direction, and the gaze direction is obtained by using the trained first deep neural network model;

Utilize the flight control module to realize the remote flight control to the unmanned aerial vehicle, and the described flight control module utilizes the described line-of-sight direction to control the flight direction of the unmanned aerial vehicle;

Use the server processing module to process the panoramic video frames collected by the binocular camera and send them to the naked-eye 3D display screen;

The naked eye 3D display screen is used for information display, providing users with an immersive flight experience.

Optionally, the UAV and the server processing module perform high-speed communication through a 5G communication module, and send the panoramic video frame data obtained by the binocular camera to the server processing module.

Optionally, the method further includes: using a 3D speaker module to play audio information captured during the flight of the drone.

Optionally, the method further includes: the control module controls the flying direction and speed of the UAV by using the line of sight direction and the line of sight concentration; the higher the line of sight concentration, the faster the forward direction; the flight direction is adjusted following the line of sight direction.

Optionally, the method further includes: using a console module to realize remote control of the UAV, and realizing the remote control of the UAV through each operation button; and/or using a voice control module to realize the remote control of the UAV , realize the remote control of the UAV through voice commands; and/or realize the remote control of the UAV by using the gesture control module, and realize the remote control of the UAV through the recognized gestures.

Optionally, if the control distance of the drone can be met, the user can configure the drone by himself; when the conditions do not allow, he can also rent the drone of the scenic spot, and the user can rent the drone after paying for it. The integrated control center of scenic spots is responsible for the management of drones. For example, if there are multiple drones in the air that may conflict with their flying positions, an alarm will be given in advance, and the required flight position and speed will be given. In emergency situations The lessor directly obtains the flight permission to avoid the collision of different drones. After the risk is removed, the permissions are returned to the user.

Optionally, the first deep neural network is used for tracking the sight line, judging its changing state, and then converting it into flight control instructions, such as: left, right, up, down, and so on.

Optionally, the first deep neural network is a DRCNN network, and the DRCNN includes: one or more convolution layers, one or more pooling layers, and a fully connected layer; the convolution kernel size used by the convolution layer is 3 *3; The excitation function used by the DRCNN is the sigmod excitation function.

Optionally, described DRCNN utilizes sine exponential loss function (Sine-Index-Softmax) to improve the accuracy of line of sight tracking; Described sine exponential loss function is:

其中，θ_yi表示为样本i与其对应标签y_i的向量夹角，其中b_yi表示样本i在其标签y_i处的偏差，b_j表示输出节点j处的偏差；所述N表示训练样本个数；所述w_yi表 示样本i在其标签y_i处的权重。

Among them, θ _yi represents the vector angle between sample _i and its corresponding label yi, where b _yi represents the deviation of sample i at its label y _i , and b _j represents the deviation of output node j; the N represents the number of training samples number; the w _yi represents the weight of sample _i at its label yi.

Optionally, the pooling method of the pooling layer is as follows:

S=f(elogw+LOSS _SIS );

Among them, s represents the output of the current layer, f() represents the activation function, and w represents the weight of the current layer.

Optionally, the line of sight concentration is obtained through a second deep neural network, and the second deep neural network is realized by an attention mechanism, and the second deep neural network is specifically an attention neural network. Convolutional features of the first neural network; can also be trained independently to obtain convolutional features suitable for its model itself. The attentional neural network divides the user's attention into multiple speed levels. Optionally, the speed levels are 1, 2, 3, 4, 5, 6, 7...N. The serial number represents the speed level. The smaller the number is, the faster the flight speed is. On the contrary, the larger the number is, the slower the flight speed is.

The excitation function adopted by the attention neural network is the cosine exponential excitation function, denoted as g(), where

Among them, θ _yi represents the vector angle between the sample _i and its corresponding label yi; the N represents the number of training samples; the w _yi represents the weight of the sample _i at its label yi.

The present application also proposes a computer-readable medium storing computer program instructions, wherein the program instructions can execute any of the above methods proposed by the present invention.

In the description of this specification, description with reference to the terms "one embodiment," "example," "specific example," etc. means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one aspect of the present invention. in one embodiment or example. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example.

Program code embodied on a computer readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples (a non-exhaustive list) of computer readable storage media include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), Erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Computer program code for carrying out operations of the present invention may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, but also conventional Procedural programming language - such as the "C" language or similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through the Internet connect). The above-mentioned integrated units implemented in the form of software functional units can be stored in a computer-readable storage medium. The above-mentioned software function unit is stored in a storage medium, and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute the methods described in the various embodiments of the present invention. some steps. The aforementioned storage medium includes: U disk, removable hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes .

The above-mentioned integrated units implemented in the form of software functional units can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute the methods described in the various embodiments of the present invention. some steps. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes .

The above are only the preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied to other related All technical fields are similarly included in the scope of patent protection of the present invention. The above-disclosed preferred embodiments of the present invention are provided only to help illustrate the present invention. The preferred embodiments are not exhaustive of all details, nor do they limit the invention to only the described embodiments. Obviously, many modifications and variations are possible in light of the contents of this specification. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can well understand and utilize the present invention. The present invention is to be limited only by the claims and their full scope and equivalents.

Claims (10)

1. A travel virtual remote experience system, the system comprising: Unmanned aerial vehicle, the unmanned aerial vehicle is equipped with a binocular camera for real-time acquisition of panoramic video frames; The gaze tracking module uses the trained first deep neural network model to obtain the user's gaze direction; A flight control module, the flight control module uses the tracked line of sight direction to control the flight direction of the UAV, so as to realize the remote flight control of the UAV; The server processing module is used to process the panoramic video frames collected by the binocular camera and send them to the naked-eye 3D display screen; The naked-eye 3D display screen is used to display information and provide users with an immersive flight experience; The first deep neural network is used for tracking the sight line, judging its changing state, and then converting it into flight control instructions, including but not limited to: left, right, up, and down; The first deep neural network is a DRCNN network, and the DRCNN includes: one or more convolution layers, one or more pooling layers, and a fully connected layer; the convolution kernel size used by the convolution layer is 3*3 ; The excitation function adopted by the DRCNN is the sigmod function; The DRCNN utilizes a sine exponential loss function to improve the accuracy of line-of-sight tracking; the sine exponential loss function is:

Among them, θ _yi represents the vector angle between sample _i and its corresponding label yi, where b _yi represents the deviation of sample i at its label y _i , and b _j represents the deviation of output node j; the N represents the number of training samples number; the w _yi represents the weight of the sample _i at its label yi; The pooling method of the pooling layer is as follows: S=f(elogw+LOSS _SIS ); Among them, s represents the output of the current layer, f() represents the activation function, and w represents the weight of the current layer. 2 . The system according to claim 1 , wherein the UAV and the server processing module perform high-speed communication through a 5G communication module, and send the panoramic video frame data obtained by the binocular camera to the server processing module. 3 . 3. The system according to claim 1, further comprising: a 3D speaker module for playing audio information captured during the flight of the drone. 4. The system according to claim 1, wherein the control module controls the direction and speed of UAV flying by sight direction and sight concentration; the higher the sight concentration, the faster the advancing direction; the flying direction is adjusted following the sight direction . 5. The system according to claim 1, further comprising a console module, which realizes remote control of the drone through each operation button; and/or a voice control module, which realizes the remote control of the drone by voice commands ; and/or a gesture control module to realize remote control of the drone through the recognized gesture. 6. A travel virtual remote experience method, characterized in that: UAV is used to collect panoramic video frames in real time, and the UAV is equipped with a binocular camera; Utilize the gaze tracking module to obtain the user's gaze direction, and the gaze direction is obtained by using the trained first deep neural network model; Utilize the flight control module to realize the remote flight control of the UAV, and the flight control module uses the tracked line of sight direction to control the flight direction of the UAV; Use the server processing module to process the panoramic video frames collected by the binocular camera and send them to the naked-eye 3D display screen; The naked eye 3D display screen is used for information display, providing users with an immersive flight experience; The first deep neural network is used for tracking the sight line, judging its changing state, and then converting it into flight control instructions, including but not limited to: left, right, up, and down; The first deep neural network is a DRCNN network, and the DRCNN includes: one or more convolution layers, one or more pooling layers, and a fully connected layer; the convolution kernel size used by the convolution layer is 3*3 ; The excitation function adopted by the DRCNN is the sigmod function; The DRCNN utilizes a sine exponential loss function to improve the accuracy of line-of-sight tracking; the sine exponential loss function is:

Among them, θ _yi represents the vector angle between sample _i and its corresponding label yi, where b _yi represents the deviation of sample i at its label y _i , and b _j represents the deviation of output node j; the N represents the number of training samples number; the w _yi represents the weight of the sample _i at its label yi; The pooling method of the pooling layer is as follows: S=f(elogw+LOSS _SIS ); Among them, s represents the output of the current layer, f() represents the activation function, and w represents the weight of the current layer. 7 . The method according to claim 6 , wherein the UAV and the server processing module perform high-speed communication through a 5G communication module, and send the panoramic video frame data obtained by the binocular camera to the server processing module. 8 . 8. The method according to claim 6, further comprising: using a 3D speaker module to play audio information captured during the flight of the drone. 9. The method according to claim 6, the method further comprises: the control module controls the direction and speed of the UAV flying through the line of sight direction and the line of sight concentration; the higher the line of sight concentration, the faster the forward direction; the flight The direction is adjusted to follow the line of sight. 10. The method according to claim 6, further comprising: using a console module to realize remote control of the drone, and realizing remote control of the drone through each operation button; and/or using a voice control module to achieve The remote control of the UAV is realized by voice commands; and/or the remote control of the UAV is realized by using the gesture control module, and the remote control of the UAV is realized by the recognized gesture.

Images