WO2023056803A1

WO2023056803A1 - Holographic presentation method and apparatus

Info

Publication number: WO2023056803A1
Application number: PCT/CN2022/116191
Authority: WO
Inventors: 谢国勇; 刘嘉兴; 许秋子
Original assignee: 深圳市瑞立视多媒体科技有限公司
Priority date: 2021-10-09
Filing date: 2022-08-31
Publication date: 2023-04-13
Also published as: CN113891063A; CN113891063B

Abstract

The present application relates to a holographic presentation method and apparatus. The method comprises: acquiring viewpoint information of a first user (S200), wherein the viewpoint information shows the binocular positions of the first user and binocular view angles of the first user, and the first user is a user holding an interaction control pen; acquiring pose information of the interaction control pen (S202); determining a first included angle between the view angle and a pointing direction (S204); and if the first included angle is greater than a first angle threshold value, adjusting an output frequency of a head-mounted display apparatus to a first frequency, and if the first included angle is not greater than the first angle threshold value, adjusting the output frequency of the head-mounted display apparatus to a second frequency, wherein the first frequency is greater than the second frequency (S206). In the present description, an output frequency of a head-mounted display apparatus is adjusted according to a first included angle between a view angle of a first user and a pointing direction of an interaction control pen, thereby facilitating the first user in identifying, from an holographic image, a portion of content that the first user expects to interact with. The method in the present application is suitable for a multi-person teaching scenario.

Description

A holographic display method and device

technical field

The present application relates to the technical field of data processing, and in particular to a holographic display method and device.

Background technique

Holographic display technology (Front-Projected Holographic Display), also known as virtual imaging technology, is a technology that uses the principles of interference and diffraction to record and reproduce real three-dimensional images of objects. , glasses and other auxiliary devices to watch the advantages of three-dimensional images. With the continuous development of display technology, holographic display technology has gained more and more attention.

In theaters, science and technology museums, museums, and even virtual reality (VR) helmets and other application scenarios, curved or ring-shaped holographic images can enable viewers to obtain a surround-view effect, thereby greatly improving the viewing experience of viewers and allowing viewers There is an immersive feeling. However, in the prior art, there are existing technical solutions that can realize the interaction between the user and the holographic image.

Contents of the invention

The present application provides a holographic display method and device to solve the problem in the prior art that it is difficult to realize the interaction between a user and a holographic image.

In a first aspect, the present application provides a holographic display method, the method is based on a holographic display system, and the holographic display system includes: a display processing device, a sand table display device, a head-mounted display device and an interactive control pen; The display processing device is electrically connected to the sand table display device, the head-mounted display device, and the interactive control pen respectively; the holographic display method is executed by the display processing device, and the method includes:

Acquiring viewpoint information of the first user, wherein the viewpoint information shows the binocular position of the first user and the binocular angle of view of the first user, and the first user holds the interactive control pen user;

Obtaining the pose information of the interactive control pen, wherein the pose information shows the direction of the interactive control pen;

determining a first angle between the viewing angle and the pointing;

If the first angle is greater than the first angle threshold, adjust the output frequency of the head-mounted display device to the first frequency; if the first angle is not greater than the first angle threshold, the head-mounted display The output frequency of the display device is adjusted to a second frequency, wherein the first frequency is greater than the second frequency.

In an optional embodiment of this specification, if the first included angle is greater than the first angle threshold, the output frequency of the head-mounted display device is adjusted to the first frequency; if the first included angle is smaller than the The first angle threshold, then adjusting the output frequency of the head-mounted display device to a second frequency, before the first frequency is greater than the second frequency, the method further includes:

acquiring the number of users, wherein the users include the first user and a second user, and the second user is a user who wears the head-mounted display device and does not hold the interactive control pen;

If the number of users is greater than the number threshold, adjust the value of the first frequency to a first value, and if the number of users is not greater than the number threshold, then adjust the value of the first frequency to a second value , wherein the first value is smaller than the second value.

Obtain viewpoint information of each second user, wherein the viewpoint information shows the binocular position of the second user and the binocular angle of view of the second user, and the second user wears the headset A user of a display device who does not hold the interactive control pen;

determining the density center of the distribution of the binocular positions of each second user in a specified environment, wherein the specified environment is the environment where the sand table display device is located;

Determining a connecting line between the density center and a designated point as a designated line, wherein the designated point is the intersection of the pointing point of the interactive control pen and the holographic image;

If the second included angle between the pointing and the specified line is greater than the second angle threshold, then adjust the output frequency of the sand table display device to a third frequency, if the second included angle is not greater than the The second angle threshold is to adjust the output frequency of the sand table display device to a fourth frequency, wherein the third frequency is greater than the fourth frequency.

In an optional embodiment of the present specification, the viewpoint information includes six degrees of freedom information of a head-mounted display device worn by the first user, wherein the viewpoint information is obtained according to a viewpoint information tracking algorithm.

In an optional embodiment of the present specification, the pose information of the interactive control pen includes six degrees of freedom information of the interactive control pen, wherein the position three degrees of freedom in the six degrees of freedom information of the interactive control pen The information is obtained according to the image information collected by the dynamic auxiliary camera, and the rotation 3-degree-of-freedom information in the six-degree-of-freedom information of the interactive control pen is obtained according to the data collected by the IMU module in the interactive control pen. The auxiliary camera is set on the sand table display device.

In an optional embodiment of this specification, the head-mounted display device includes shutter-type active 3D glasses and at least three reflective marking points embedded in the casing of the shutter-type active 3D glasses. Structurally, the shutter type active 3D glasses are used to receive the holographic image transmitted by the sand table type display device, so as to obtain the correct left and right eye images.

In an optional embodiment of this specification, the head-mounted display device is provided with a first pressure sensor, and when the head-mounted display device is worn on the user's head, the first pressure sensor generates the first A signal is sent to the display processing device; the interactive control pen is provided with an IMU module, and when the posture of the interactive control pen changes, a second signal is generated and sent to the display processing device; wherein, obtaining Viewpoint information of the first user, including:

If the first signal is detected at a specified moment and the second signal is detected within a specified time range from the specified moment, the viewpoint information of the first user is acquired.

In the second aspect, the present application provides a holographic display device, including:

The first acquisition module is configured to: acquire the viewpoint information of the first user, wherein the viewpoint information shows the binocular position of the first user and the binocular angle of view of the first user, and the first user is a user holding the interactive control pen;

[Incorporation by reference (Rule 20.6) 21.09.2022]
The second acquisition module is configured to: acquire the pose information of the interactive control pen, wherein the pose information shows the direction of the interactive control pen;

[Incorporation by reference (Rule 20.6) 21.09.2022]
A first included angle determination module, configured to: determine a first included angle between the viewing angle and the orientation;

[Incorporation by reference (Rule 20.6) 21.09.2022]
The first adjustment module is configured to: adjust the output frequency of the head-mounted display device to the first frequency if the first included angle is greater than the first angle threshold; if the first included angle is not greater than the first angle threshold, adjust the output frequency of the head-mounted display device to a second frequency, wherein the first frequency is greater than the second frequency.

[Incorporation by reference (Rule 20.6) 21.09.2022]
In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete mutual communication through the communication bus;

[Incorporation by reference (Rule 20.6) 21.09.2022]
memory for storing computer programs;

[Incorporation by reference (Rule 20.6) 21.09.2022]
The processor is configured to implement the steps of any holographic display method in the first aspect when executing the program stored in the memory.

[Incorporation by reference (Rule 20.6) 21.09.2022]
In a fourth aspect, the present application provides a computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any holographic presentation method in the first aspect are implemented.

[Incorporation by reference (Rule 20.6) 21.09.2022]
Compared with the prior art, the above-mentioned technical solutions provided by the embodiments of the present application have the following advantages:

[Incorporation by reference (Rule 20.6) 21.09.2022]
In the holographic display method and device in this specification, the interaction between the first user and the holographic image is mainly realized through the interactive control pen. An included angle to adjust the output frequency of the head-mounted display device. If the first included angle is greater than the first angle threshold, it indicates that there is a large position difference between the parts of the content that the first user wishes to interact with in the holographic image. and/or angle differences may cause the first user to be unable to accurately select the part of the holographic image that he wishes to interact with. The part of the image that you want to interact with. In the case that the first included angle is not greater than the first angle threshold, it indicates that the first user has good interaction conditions with the part of the content that he wants to interact with. At this time, reducing the output frequency of the head-mounted display device can make more Users participate in the viewing of holographic images, which is conducive to the reasonable allocation of holographic image display resources. The method in this application is applicable to teaching scenarios for multiple people.

[Incorporation by reference (Rule 20.6) 21.09.2022] Description of drawings

[Incorporation by reference (Rule 20.6) 21.09.2022]
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description serve to explain the principles of the invention.

[Incorporation by reference (Rule 20.6) 21.09.2022]
In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, In other words, other drawings can also be obtained from these drawings without paying creative labor.

[Incorporation by reference (Rule 20.6) 21.09.2022]
FIG. 1 is a schematic diagram of a scene involved in a holographic presentation process provided by an embodiment of the present application;

[Incorporation by reference (Rule 20.6) 21.09.2022]
FIG. 2 is a schematic flowchart of a holographic display process provided by the embodiment of the present application;

[Incorporation by reference (Rule 20.6) 21.09.2022]
Fig. 3 is a schematic diagram of a holographic display device corresponding to some steps in the process of Fig. 2;

[Incorporation by reference (Rule 20.6) 21.09.2022]
FIG. 4 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

[Incorporation by reference (rule 20.6) 21.09.2022] Specific implementation methods

[Incorporation by reference (Rule 20.6) 21.09.2022]
In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of this application, but not all of them. Based on the embodiments in the present application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present application.

[Incorporation by reference (Rule 20.6) 21.09.2022]
The process in this specification is based on a holographic display system, and the holographic display system includes: a display processing device, a sand table display device, a head-mounted display device, and an interactive control pen; the display processing device is connected with the sand table display device, the The head-mounted display device and the interactive control pen are electrically connected respectively, and an exemplary scene is shown in FIG. 1 . The head-mounted display device can be in one-to-one correspondence with the user. Optionally, different head-mounted display devices output pictures to the user at the same frequency. In the case that multiple users (not two or more users) watch the holographic images displayed on the sand table display device at the same time, the display processing device outputs display signals to each head-mounted display device respectively.

[Incorporation by reference (Rule 20.6) 21.09.2022]
The method in this manual is applicable to the teaching scene. When there are multiple users, one of the multiple users is a teacher, and the other users are students.

[Incorporation by reference (Rule 20.6) 21.09.2022]
In the scene where multiple users watch the holographic images displayed on the sand table display device at the same time, the user who wears the head-mounted display device and holds the interactive control pen is the first user, and the user who wears the head-mounted display device and does not hold the interactive control pen The user is the second user.

[Incorporation by reference (Rule 20.6) 21.09.2022]
As shown in Figure 2, the holographic display method and device in this specification include the following steps:

[Incorporation by reference (Rule 20.6) 21.09.2022]
S200: Obtain viewpoint information of the first user.

[Incorporation by reference (Rule 20.6) 21.09.2022]
In this specification, the viewpoint information shows the binocular position of the first user and the binocular angle of view of the first user, and the first user is a user holding the interactive control pen.

[Incorporation by reference (Rule 20.6) 21.09.2022]
In an optional embodiment of the present specification, the viewpoint information includes six degrees of freedom information of a head-mounted display device worn by the first user, wherein the viewpoint information is obtained according to a viewpoint information tracking algorithm. Optionally, the head-mounted display device includes shutter-type active 3D glasses and at least three reflective marking points, the at least three reflective marking points are embedded on the shell structure of the shutter-type active 3D glasses, and the shutter-type active The 3D glasses are used to receive the holographic images transmitted by the sand table display device, so as to obtain the correct images for the left and right eyes.

[Incorporation by reference (Rule 20.6) 21.09.2022]
In the scenario where multiple users watch the holographic image displayed on the sand table display device at the same time, the process of determining the first user among the users may be as follows. For each head-mounted display device, determine the first position of the head-mounted display device in the specified environment (ie, the binocular position of the user wearing the head-mounted display device). And determine the second position of the interactive control pen in the specified environment. Among the head-mounted display devices, the head-mounted display devices whose first position is within a specified distance range (preset range, which can be 0.5 meters to 1.2 meters) from the second position are determined as the first pending devices. For each first pending device, determine the first included angle corresponding to the first pending device. Determine the first undetermined device corresponding to the first included angle not greater than the third angle threshold (the third angle threshold is a preset value, greater than the first angle threshold), as the second undetermined device. Among the second undetermined devices, the second undetermined device with the closest distance to the designated point is determined as the head-mounted display device worn by the first user.

[Incorporation by reference (Rule 20.6) 21.09.2022]
S202: Obtain pose information of the interactive control pen.

[Incorporation by reference (Rule 20.6) 21.09.2022]
The pose information of the interactive control pen shows the direction of the interactive control pen. In an optional embodiment of this specification, the pose information of the interactive control pen includes six degrees of freedom information of the interactive control pen, wherein the position three-degree-of-freedom information in the six-degree-of-freedom information of the interactive control pen is Obtained according to the image information collected by the dynamic-assisted camera, the rotation 3-degree-of-freedom information in the six-degree-of-freedom information of the interactive control pen is obtained according to the data collected by the IMU module in the interactive control pen, the dynamic-assisted camera It is arranged on the sand table type display device.

[Incorporation by reference (Rule 20.6) 21.09.2022]
S204: Determine a first included angle between the viewing angle and the orientation.

[Incorporation by reference (Rule 20.6) 21.09.2022]
In order to determine the positions of each head-mounted display device and the interactive control pen, angles such as the first included angle are determined. In an optional embodiment of this specification, a three-dimensional coordinate system may be established in advance in a designated environment, and the coordinate origin of the three-dimensional coordinate system is the center of the display screen of the sand table display device.

[Incorporation by reference (Rule 20.6) 21.09.2022]
S206: If the first included angle is greater than the first angle threshold, adjust the output frequency of the head-mounted display device to the first frequency; if the first included angle is not greater than the first angle threshold, turn the head The output frequency of the wearable display device is adjusted to a second frequency, wherein the first frequency is greater than the second frequency.

[Incorporation by reference (Rule 20.6) 21.09.2022]
In this specification, the output frequency of the head-mounted display device is adjusted according to the first included angle between the viewing angle of the first user and the direction of the interactive control pen. When the first included angle is greater than the first angle threshold, it means There is a large position difference and/or angle difference between the parts of the content that the first user wishes to interact with in the holographic image, which may cause the first user to be unable to accurately select the part of the content that the first user wishes to interact with in the holographic image. , increasing the output frequency of the head-mounted display device is beneficial for the first user to recognize part of the content in the holographic image that he wishes to interact with. In the case that the first included angle is not greater than the first angle threshold, it indicates that the first user has good interaction conditions with the part of the content that he wants to interact with. At this time, reducing the output frequency of the head-mounted display device can make more Users participate in the viewing of holographic images, which is conducive to the reasonable allocation of holographic image display resources.

[Incorporation by reference (Rule 20.6) 21.09.2022]
How to determine the first frequency will now be described. In an optional embodiment of the present specification, the number of users is acquired (the number of users may be acquired in real time, so as to adjust the first frequency in real time). Wherein, the user includes the first user and a second user, and the second user is a user who wears the head-mounted display device and does not hold the interactive control pen. If the number of users is greater than the number threshold, adjust the value of the first frequency to a first value, and if the number of users is not greater than the number threshold, then adjust the value of the first frequency to a second value , wherein the first value is smaller than the second value. Both the first value and the second value can be preset values, which can be set by the administrator of the holographic display system.

Furthermore, this specification not only adjusts the output frequency of the head-mounted display device to rationally allocate display resources, but also can reduce resource consumption while satisfying viewing needs of users. In an optional real-time in this specification, the viewpoint information of each second user is obtained (it may be obtained in real time, so as to adjust the output frequency of the sand table display device in real time). Wherein, the viewpoint information shows the binocular position of the second user and the binocular angle of view of the second user, and the second user wears the head-mounted display device and does not hold the The user who controls the pen interactively. Determining the density center of the distribution of the binocular positions of each second user in a specified environment, wherein the specified environment is the environment where the sand table display device is located. A line connecting the density center and a designated point is determined as a designated line, wherein the designated point is an intersection of the pointing point of the interactive control pen and the holographic image. If the second included angle between the pointing and the specified line is greater than the second angle threshold, then adjust the output frequency of the sand table display device to a third frequency, if the second included angle is not greater than the The second angle threshold is to adjust the output frequency of the sand table display device to a fourth frequency, wherein the third frequency is greater than the fourth frequency.

In actual application scenarios, there may be a holographic display system that only needs to provide a display to the user without interacting with the user. Then, in an optional embodiment of this specification, in order to determine whether to interact with the user, the head-mounted display device is provided with a first pressure sensor, and the head-mounted display device is worn on the head of the user. When it is on, the first pressure sensor generates a first signal and sends it to the display processing device; the interactive control pen is provided with an IMU module, and when the posture of the interactive control pen changes, a second signal is generated, sent to the display processing device; wherein, obtaining the viewpoint information of the first user includes: if the first signal is detected at a specified moment, and the second signal is detected within a specified time range from the specified moment signal, the viewpoint information of the first user is acquired.

In related technologies, in order to realize LED multi-viewpoint 3D display, shutter-type active 3D glasses with a refresh rate of at least 240 Hz are required and support multi-viewpoint display functions. The refresh rate of existing shutter-type active 3D glasses on the market is only 120Hz, and does not support multi-viewpoint display function.

Exemplarily, the receiving frequency of the interface chip at the input end of the sending device cannot reach the minimum standard of 240 Hz. The sending device communicates with the 3D display screen through a network cable, and the transmission frequency can be much higher than 240Hz. Exemplarily, this manual adopts an interface chip with a 4K bandwidth and a refresh rate of 60Hz, and the amount of input data is equivalent to 1080P with a bandwidth of 240Hz. The sending device in this manual cuts the received 4K picture into 4 (corresponding to two shutter-type active 3D glasses) 1080P bandwidth pictures and transmits them to the LED screen receiving card for display at a refresh rate of 240Hz or higher. It is equivalent to realizing the LED display system capable of displaying 240Hz video stream.

In this manual, multiple shutter-type active 3D glasses receive images from the same 3D display screen. For example, in the order of left 1, left 2, right 1, and right 2, two matching images are presented at an interval of 4.166ms. A total of 4 pictures of the shutter-type active 3D glasses. The shutter-type active 3D glasses itself supports a display refresh rate of 120Hz, but the synchronization signal is a square wave signal, a 60Hz square wave signal, a single cycle already includes a peak half cycle and a valley half cycle, and there are 120 state switches per second , which can meet the 120Hz display refresh rate of ordinary shutter-type active 3D glasses.

And this synchronization method can effectively distinguish the left and right eyes, for example, the peak is defined as the left eye, and the trough is defined as the right eye. Then, for a 240Hz 3D image display system, the frequency of the synchronous signal is 60Hz. Under the 60Hz synchronous signal, the peak and trough can still be used to distinguish the left and right eyes.

In an optional embodiment of this specification, the shutter-type active 3D glasses are shutter-type active 3D glasses with adjustable signal delay. Every time a peak signal is received, the left eye becomes transparent for 4.166ms (the peak signal will last for 8.333ms, this The time-varying transparent state lasts only 4.166ms. When it becomes transparent, you can receive the picture on the 3D display screen. When it is not transparent, the display area can display black), and the right eye becomes transparent every time a trough is received 4.166 ms. Then adjust the signal delay of one of the glasses to 4.166ms to distinguish No. 1 and No. 2 glasses.

This specification does not specifically limit the way of interaction. For example, the way of interaction may be to enlarge, shrink, move, rotate, select, and replace the holographic image output by the sand table display device.

In an optional embodiment of this description, the sand table display device is an LED (Light Emitting Diode, light emitting diode) display device. The image specification corresponding to the first display signal is 1920 pixels*1080 pixels, and the output frequency of the first display signal is 120 Hz, so as to realize alternately output images to the binoculars of the user at a frequency of 60 Hz.

After receiving the first display signal, the sand table display device outputs a holographic image according to the first display signal for viewing by a user wearing the head-mounted display device.

In order to enable the sand table display device and the head-mounted display device to cooperate, the display processing device generates a synchronization signal according to the first display signal of the left eye and the first display signal of the right eye, so that the head-mounted display device collects The holographic images output by the sand table display device are used to alternately display the collected holographic images to the user wearing the head-mounted display device. It can be seen that the synchronous signal is used to realize alternate binocular output images for the user.

In an optional embodiment of this specification, the head-mounted display device is shutter-type 3D glasses.

In addition, when the holographic display system in this specification provides holographic image display for multiple users at the same time, since the viewpoint of each user (that is, the viewpoint of the head-mounted display device worn by the user) is different to varying degrees, The display processing device in this specification determines its viewpoint for each head-mounted display device. Since the number of head-mounted display devices is not unique, and the first display signal received by the sand table display device is not unique, the sand table display device needs to send information to different head-mounted display devices according to different first display signals. Holographic display.

Furthermore, in an optional embodiment of the present specification, the holographic image displayed by the sand table display device will also cooperate with an interactive handle to provide users with a more interactive experience. Specifically, the display processing device judges whether the contour of the holographic image displayed by the sand table display device intersects with the indicating end of the interactive handle. If not, it means that the pointing end of the interactive handle is not pointing at the holographic image, causing the position indicated by the interactive handle on the holographic image to be unclear, then the holographic image displayed by the sand table display device can be enlarged until the outline of the holographic image is consistent with the holographic image. Intersect the indicating end of the interaction handle described above.

It can be known from the foregoing that the holographic display in this description requires the cooperation of a sand table display device and a head-mounted display device. This coordination is achieved to a certain extent by means of synchronization signals. In an optional embodiment of this specification, a synchronous signal can be used to control all head-mounted display devices; in another optional embodiment of this specification, each head-mounted display device can be controlled separately The devices generate synchronous signals separately to control different head-mounted display devices separately.

Specifically, for a head-mounted display device, the synchronization signal controls the head-mounted display device to collect the content displayed by the sand table display device when the sand table display device displays the holographic image under the head-mounted display device's viewpoint. , and display it to the user according to the collected results.

In an optional embodiment of this specification, the head-mounted display device collects the holographic image output by the sand table display device according to the synchronization signal, and alternately displays the holographic image to the user wearing the head-mounted display device. Collected holographic images.

The holographic display method in this manual adopts the method of cooperating with the sand table display device and the head-mounted display device to display holographic images to the user. The sand table display device is highly controllable, and the brightness of its display can be adjusted according to the The brightness of the scene is adjusted, and even in a dark scene, a better holographic image display effect can be obtained. Since the holographic images in this description are assisted by a sand table display device, compared with the existing VR glasses, the head-mounted display device in this manual does not need to perform more complicated data processing, which is beneficial to reduce the size of the head-mounted display. The volume and weight of the device make the user feel more comfortable.

In addition, existing 3D display products usually only have a very narrow observation point of view or can only see the ideal effect at a fixed position, that is, the existing 3D display products only support a single point of view, but the 3D stereoscopic interactive display of single point of view display The system is increasingly unable to meet the needs of customers. However, the holographic display method in this specification adopts the combination of sand table display device and head-mounted display device, even if the user's point of view changes (which may be caused by the user's movement), it can also show the user a three-dimensional image. The holographic image is conducive to improving the user experience.

In addition, the display processing device in the holographic display system controls the display of the sand table display device according to the viewpoint of the head mounted display device. content, and collect holographic images suitable for users to watch. Furthermore, the holographic display system includes multiple head-mounted display devices. When different head-mounted display devices are worn by different users, the viewpoints of different head-mounted display devices will be different. Through the method in this specification, it is possible to realize Through the same sand table display device, multiple head-mounted display devices with different viewpoints can simultaneously display holographic images.

In order to realize the control of the sand table display device flexibly and efficiently, in an optional embodiment of this specification, after the display processing device generates the first display signals corresponding to each viewpoint, it performs Combining processing: outputting the composite signal obtained after the combining processing to the sand table display device.

In an optional embodiment of this specification, the image specification corresponding to the composite signal is 1920 pixels*1080 pixels. The output frequency of the composite signal is 120*k Hz. Among them, k is the number of head-mounted display devices. When k is equal to 2, the output frequency of the composite signal is 240 Hz.

In order to realize the data transmission between the display processing device and the sand table display device, and effectively drive the sand table display device in conjunction with the multi-viewpoint output scene, in an optional embodiment of this specification, the holographic display system may also include receive card. After the display processing device generates the composite signal, it outputs the composite signal to the receiving card. The receiving card restores the composite signal to each first display signal, and drives the sand table type display device to display the holographic image according to each first display signal.

Specifically, the receiving card outputs each first display signal to the sand table display device in time division, so that the sand table display device performs time division display of the holographic image according to each first display signal. When the sand table display device performs holographic image display according to the first display signal corresponding to the i-th head-mounted display device (i can be any integer, indicating any one), the i-th head-mounted display device displays the sand table according to the synchronization signal. If the holographic image displayed by the head-mounted display device is collected, the collected result matches the viewpoint of the i-th head-mounted display device.

It can be seen that the receiving card in this specification has a certain interface function to cooperate with multiple sand table display devices to display holographic images simultaneously, and/or cooperate with different types of sand table display devices to display holographic images.

Through the holographic display method in this specification, it is possible to display holographic images with strong stereoscopic effect and high image quality for users in various scenes with different brightness. However, in the actual scene, the user may move in the site where the sand table display device is installed according to his own needs, and there may be a mismatch between the holographic image displayed by the sand table display device and the user's actual viewpoint due to the user's movement. In addition, when there are multiple users, the viewpoints of the multiple users are also different to some extent.

In order to be able to provide holographic images with excellent display effects while the user is moving, and to display holographic images from different viewpoints at different times, in an optional embodiment of this specification, the holographic display system also includes: a motion capture camera . The motion capture camera is arranged on the sand table display device. The number of motion capture cameras in this manual, the installation positions and shooting angles on the sand table display device are determined according to the setting, shape, size, etc. of the sand table display device. Specifically, the number of motion capture cameras is four, which are respectively arranged at the corners of the sand table display device. The motion capture camera is used for image acquisition (shooting) of the environment around the sand table display device, so as to capture the position of the user in the scene, and then obtain the viewpoint of the head-mounted display device.

In other implementations of this specification, the motion capture camera may be arranged above the sand table display device to collect images of the environment from a bird's-eye view.

The motion capture camera may perform image acquisition under the control of the display processing device.

In an optional embodiment of this specification, the motion capture camera first acquires the 3D coordinate information of the head-mounted display device in space, and then combines the obtained human eye viewpoint of the user wearing the head-mounted display device in the motion capture space Coordinate information, so as to complete the tracking and positioning of the viewing angle of the user, and then transmit the positioning information to the display processing device. The position signal of the display processing device in the motion capture space is synchronized to the holographic image, and the three-dimensional position of the head-mounted display device in the motion capture space is used as the position of the virtual camera in the holographic image (that is, the position of the head-mounted display device position) to calculate the viewpoint picture of the virtual camera.

Specifically, the display processing device may include: a display server, a display processor, and a synchronization transmitter. The first display signal is generated by the display server; the composite signal and/or synchronization signal is generated by the display processor; the synchronization signal is sent to the head-mounted display device by the synchronization transmitter. Optionally, the aforementioned left-eye display signal and right-eye display signal are generated by a display server, and the display server sends the left-eye display signal and right-eye display signal to the display processor after generating the left-eye display signal and right-eye display signal. The display processor generates a first display signal according to the left-eye display signal and the right-eye display signal, and sends it to the sand table display device. The synchronization signal is sent to the HMD by the synchronization transmitter.

Optionally, the display server is electrically connected to the display processor through a DP or HDMI cable; the display processor is electrically connected to the receiving card through a network port; and the display processor is electrically connected to the synchronization transmitter through a cable. The receiving card is electrically connected with the sand table display device through a cable.

In an optional embodiment of this specification, the sand table display device may be composed of small-pitch LED display screens, and the sand table display device includes an LED display body, a 3D video fuser, a 3D signal transmitter, and the like.

Wherein, the three-dimensional video fusion device is connected to the sending card of the LED display body. The three-dimensional signal transmitter is connected to the three-dimensional video fuser, and the three-dimensional video fuser receives the viewpoint picture information of the virtual camera (that is, the head-mounted display device) processed by the display processor in the display processing device, adjusts the display frequency and The picture is displayed on the LED display body through the sending card to display the left and right eye pictures, and the display processor is equipped with a holographic three-dimensional display algorithm. The three-dimensional signal transmitter synchronously sends the display frequency to the holographic 3D glasses, so as to realize real-time display of images adjusted according to different viewpoint positions of users wearing the holographic 3D glasses.

In this description, in the virtual three-dimensional scene, the three-dimensional position of the head-mounted display device in the motion capture space is used as the position of the virtual camera in the virtual space, so that the position of a virtual camera can be simulated to capture the location of the head-mounted display device. position, the virtual three-dimensional scene is set according to the position of the virtual camera in the virtual space.

Then the picture taken by the virtual camera (that is, the picture under the viewpoint of the head-mounted display device) is the three-dimensional virtual scene picture (that is, a holographic image) seen by the user wearing the head-mounted display device. The virtual three-dimensional scene captured by the camera is the 3D perspective picture viewed by the user. The display processing device can process the picture to be displayed in real time according to the viewpoint position information of the head-mounted display device in the motion capture space, and at the same time use the sand table display device to display the viewpoint picture, which can achieve excellent holographic three-dimensional display effect and obtain excellent Excellent out-of-screen sense and holographic visual sense.

Compared with other products, the holographic display system provided by this application has greatly reduced cost, more delicate and richer picture display, and richer colors. The display effect is stable, the user is more convenient to use, and the operation control is simpler. Among them, the sand table display device has delicate display effect, high brightness, and low cost. It adopts LED active 3D stereoscopic display technology combined with the spatial position information of the observation point, so as to realize the 3D stereoscopic effect of virtual objects suspended on the LED display screen, which can improve the perception of users. , to improve the effect of the holographic display.

In addition, in an optional embodiment of this specification, the process in this specification can not only be used to display holographic images to users, but also realize interaction with users. Specifically, when the display processing device receives the interaction instruction, it generates a second display signal and outputs it to the sand table display device, so that the sand table display device outputs a holographic image for displaying the interactive handle according to the second display signal, so that the user can Operate against the interactive handle to achieve interaction.

Optionally, the interaction instruction is triggered by a specified action performed by the user through the interactive control pen, and the specified action may be collected by a motion capture camera. The holographic image of the interactive handle can be a rod-shaped holographic image, and one end of the interactive handle cooperates with the user's hand movement, and moves in the holographic image generated by the sand table display device following the user's hand movement.

Based on the same idea, this specification further provides a holographic display device, as shown in Figure 3, the holographic display device includes one or more of the following modules:

The first acquisition module 300 is configured to: acquire the viewpoint information of the first user, wherein the viewpoint information shows the binocular position of the first user and the binocular angle of view of the first user, and the first user is the user holding the interactive control pen;

The second obtaining module 302 is configured to: obtain pose information of the interactive control pen, wherein the pose information shows the direction of the interactive control pen;

The first included angle determination module 304 is configured to: determine a first included angle between the viewing angle and the orientation;

The first adjustment module 306 is configured to: adjust the output frequency of the head-mounted display device to the first frequency if the first included angle is greater than the first angle threshold; if the first included angle is not greater than the first angle threshold When an angle threshold is reached, the output frequency of the head-mounted display device is adjusted to a second frequency, wherein the first frequency is greater than the second frequency.

In an optional embodiment of the present specification, the holographic display device further includes a second adjustment module. The second adjustment module is configured to: obtain the number of users, wherein the users include the first user and the second user, and the second user wears the head-mounted display device and does not hold the Users who interact with the control pen; if the number of users is greater than the number threshold, adjust the value of the first frequency to the first value; if the number of users is not greater than the number threshold, then adjust the value of the first frequency The value is adjusted to a second value, wherein the first value is less than the second value.

In an optional embodiment of the present specification, the holographic display device further includes a third adjustment module, and the third adjustment module is configured to: acquire viewpoint information of each second user, wherein the viewpoint information shows that the first The binocular position of the second user and the binocular angle of view of the second user, the second user is a user who wears the head-mounted display device and does not hold the interactive control pen; determine each second user The density center of the distribution of the binocular position in the specified environment, wherein the specified environment is the environment where the sand table display device is located; determine the connection line between the density center and the specified point as the specified line, Wherein, the designated point is the intersection of the pointing point of the interactive control pen and the holographic image; if the second angle between the pointing point and the designated line is greater than a second angle threshold, the sand table display device The output frequency of the sand table display device is adjusted to the third frequency, and if the second included angle is not greater than the second angle threshold, the output frequency of the sand table display device is adjusted to the fourth frequency, wherein the third frequency is greater than the fourth frequency.

In an optional embodiment of this specification, the head-mounted display device is provided with a first pressure sensor, and when the head-mounted display device is worn on the user's head, the first pressure sensor generates the first A signal is sent to the display processing device; the interactive control pen is provided with an IMU module, and when the posture of the interactive control pen changes, a second signal is generated and sent to the display processing device; wherein, the first An acquisition module 300, specifically configured to acquire the viewpoint information of the first user if the first signal is detected at a specified time and the second signal is detected within a specified time range from the specified time.

Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application. Please refer to FIG. 4 , at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Wherein, the memory may include a memory, such as a high-speed random-access memory (Random-Access Memory, RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. Of course, the electronic device may also include hardware required by other services.

The processor, the network interface and the memory can be connected to each other through an internal bus, which can be an ISA (Industry Standard Architecture, industry standard architecture) bus, a PCI (Peripheral Component Interconnect, peripheral component interconnection standard) bus or an EISA (Extended Industry Standard Architecture, extended industry standard architecture) bus, etc. The bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one double-headed arrow is used in FIG. 4 , but it does not mean that there is only one bus or one type of bus.

Memory for storing programs. Specifically, the program may include program code, and the program code includes computer operation instructions. Storage, which can include internal memory and nonvolatile storage, provides instructions and data to the processor.

The processor reads the corresponding computer program from the non-volatile memory into the memory and runs it, forming a holographic display device and/or a second holographic display device on a logical level. The processor executes the program stored in the memory, and is specifically used to execute any of the aforementioned holographic display processes.

It should be noted that in this article, relative terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these No such actual relationship or order exists between entities or operations. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

The above are only specific embodiments of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Accordingly, the present invention will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

A holographic display method, the method is based on a holographic display system, and the holographic display system includes: a display processing device, a sand table display device, a head-mounted display device, and an interactive control pen; the display processing device and the sand table display device The display device, the head-mounted display device, and the interactive control pen are electrically connected respectively; the holographic display method is executed by the display processing device, and the method includes:

Acquiring viewpoint information of the first user, wherein the viewpoint information shows the binocular position of the first user and the binocular angle of view of the first user, and the first user holds the interactive control pen user;

acquiring pose information of the interactive control pen, wherein the pose information shows the direction of the interactive control pen;

determining a first angle between the viewing angle and the pointing;

If the first angle is greater than the first angle threshold, adjust the output frequency of the head-mounted display device to the first frequency; if the first angle is not greater than the first angle threshold, the head-mounted display The output frequency of the display device is adjusted to a second frequency, wherein the first frequency is greater than the second frequency.
The method according to claim 1, wherein if the first included angle is greater than a first angle threshold, the output frequency of the head-mounted display device is adjusted to the first frequency, and if the first included angle is smaller than The first angle threshold, then adjusting the output frequency of the head-mounted display device to a second frequency, before the first frequency is greater than the second frequency, the method further includes:

acquiring the number of users, wherein the users include the first user and a second user, and the second user is a user who wears the head-mounted display device and does not hold the interactive control pen;

If the number of users is greater than the number threshold, adjust the value of the first frequency to a first value, and if the number of users is not greater than the number threshold, then adjust the value of the first frequency to a second value , wherein the first value is smaller than the second value.
The method according to claim 1, wherein if the first included angle is greater than a first angle threshold, the output frequency of the head-mounted display device is adjusted to the first frequency, and if the first included angle is smaller than The first angle threshold, then adjusting the output frequency of the head-mounted display device to a second frequency, before the first frequency is greater than the second frequency, the method further includes:

Obtain viewpoint information of each second user, wherein the viewpoint information shows the binocular position of the second user and the binocular angle of view of the second user, and the second user wears the headset A user of a display device who does not hold the interactive control pen;

determining the density center of the distribution of the binocular positions of each second user in a specified environment, wherein the specified environment is the environment where the sand table display device is located;

Determining a connecting line between the density center and a designated point as a designated line, wherein the designated point is the intersection of the pointing point of the interactive control pen and the holographic image;

If the second included angle between the pointing and the specified line is greater than the second angle threshold, then adjust the output frequency of the sand table display device to a third frequency, if the second included angle is not greater than the The second angle threshold is to adjust the output frequency of the sand table display device to a fourth frequency, wherein the third frequency is greater than the fourth frequency.
The method according to claim 1, wherein the viewpoint information includes six degrees of freedom information of a head-mounted display device worn by the first user, wherein the viewpoint information is obtained according to a viewpoint information tracking algorithm .
The method according to claim 1, wherein the pose information of the interactive control pen includes six degrees of freedom information of the interactive control pen, wherein position 3 in the six degrees of freedom information of the interactive control pen The degree of freedom information is obtained according to the image information collected by the motion-assisted camera, and the rotation 3-degree-of-freedom information in the six-degree-of-freedom information of the interactive control pen is obtained according to the data collected by the IMU module in the interactive control pen, so The dynamic auxiliary camera is arranged on the sand table display device.
The method according to claim 1, wherein the head-mounted display device includes shutter-type active 3D glasses and at least three reflective marking points embedded in the shutter-type active 3D glasses. In terms of the shell structure, the shutter type active 3D glasses are used to receive the holographic image transmitted by the sand table type display device, so as to obtain the correct left and right eye images.
The method according to claim 1, wherein a first pressure sensor is provided on the head-mounted display device, and when the head-mounted display device is worn on the user's head, the first pressure sensor Generate a first signal and send it to the display processing device; the interactive control pen is provided with an IMU module, and when the posture of the interactive control pen changes, generate a second signal and send it to the display processing device; wherein , to obtain the viewpoint information of the first user, including:

If the first signal is detected at a specified moment and the second signal is detected within a specified time range from the specified moment, the viewpoint information of the first user is acquired.
A holographic display device, characterized in that the device comprises:

The first acquisition module is configured to: acquire the viewpoint information of the first user, wherein the viewpoint information shows the binocular position of the first user and the binocular angle of view of the first user, and the first user is a user holding the interactive control pen;

The second acquisition module is configured to: acquire the pose information of the interactive control pen, wherein the pose information shows the direction of the interactive control pen;

A first included angle determination module, configured to: determine a first included angle between the viewing angle and the orientation;

The first adjustment module is configured to: adjust the output frequency of the head-mounted display device to the first frequency if the first included angle is greater than the first angle threshold; if the first included angle is not greater than the first angle threshold, adjust the output frequency of the head-mounted display device to a second frequency, wherein the first frequency is greater than the second frequency.
An electronic device, characterized in that it includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete mutual communication through the communication bus;

memory for storing computer programs;

The processor is configured to implement the steps of the holographic display method according to any one of claims 1-7 when executing the program stored in the memory.
A computer-readable storage medium, on which a computer program is stored, wherein, when the computer program is executed by a processor, the steps of the holographic presentation method according to any one of claims 1-7 are realized.