WO2024198148A9 - Extended reality display system - Google Patents

Abstract

An extended reality display system (1), comprising: a first presentation device (11), which comprises a first presentation apparatus (111) and a first acoustic apparatus (112), the first presentation apparatus (111) being configured to display to a user a first visual effect, and the first acoustic apparatus (112) being configured to playback to the user an audio stream associated with a timeline of the first visual effect; a second presentation device (12), which comprises a second presentation apparatus (121), a head support member (122), a sensor assembly (123) and a signal transceiver (124), the head support (122) engaging the user's head, the second presentation apparatus (121) being coupled to the head support member (122) to be supported thereon and being configured to display to the user a second visual effect related to the timeline of the first visual effect, the sensor assembly (123) being arranged on the second presentation apparatus (121) and configured to measure the orientation of the second presentation device (12), and the signal transceiver (124) being connected to the sensor assembly (123); a central module (13), which is connected to the first presentation device (11); and one or more transmission apparatuses (14), which are connected to the central module (13) and the second presentation device (12).

Description

Extended Reality Display System Technical Field

The present application relates to an extended reality display system.

Background Art

Extended reality (XR) is a technology that integrates virtual content with the real environment, including virtual reality (VR), augmented reality (AR) and mixed reality (MR). In recent years, with the development of 5G networks, image processing, tracking technology, etc., XR technology has been widely used in live performances, bringing immersive, interactive and innovative audio-visual experience to the audience.

There are two main forms of XR technology in live performances: one is to build a fully virtual environment, with actors performing against a green screen background, and the image uses technical means to replace the green screen background with a designed virtual scene; the other is to synthesize the produced images with the real images on the real stage to superimpose visual effects. Both forms require the use of movie screens/LED screens, cameras, sensors and other equipment to achieve.

The advantages of applying XR technology to live performances are: it can break the limitations of space and time, allowing the audience to feel the atmosphere and scenes of different places, cultures, and historical periods; it can enhance the audience's sense of participation and immersion, allowing the audience to interact and communicate with actors and virtual content; it can enhance the creativity and artistry of the performance, allowing actors and directors to have more freedom and possibilities to express their ideas and emotions; it can reduce the cost and risk of performances, and reduce dependence on physical venues, props, lighting and other resources.

However, XR technology also has some problems in live performances: it requires high equipment investment and maintenance costs, as well as a professional technical team to support it; it requires high-speed and stable network connection and data transmission capabilities, as well as efficient and accurate image rendering capabilities; it needs to balance the coordination and realism between virtual content and real content to avoid causing audiovisual disharmony or discomfort; and it is necessary to consider the differences in the audience's acceptance of XR technology and usage habits, and provide corresponding guidance or education.

In order to overcome the above problems, it is expected to develop a system that replaces the real stage with a movie screen/LED screen and combines XR technology in it. At present, the effect of simulating the real stage with a movie screen is poor, because the movie screen can only provide a two-dimensional picture. Even if the user uses 3D glasses to obtain a simulated 3D visual effect, it is limited to the range of the movie screen. Filling the part of the user's field of view that is outside the range of the movie screen will greatly enhance the sense of presence of simulating a real stage.

Summary of the invention

In order to solve the problems in the prior art and enhance the sense of presence of simulating a real stage, the present application proposes an extended reality display system, comprising: a first presentation device, which includes a first presentation device and a first audio device, wherein the first presentation device is configured to display a first visual effect to a user, and the first audio device is configured to play an audio stream related to the timeline of the first visual effect to the user; a second presentation device, which includes a second presentation device, a head support, a sensor assembly and a signal transceiver, wherein the head support engages the user's head, the second presentation device is coupled to the head support to be supported thereon and is configured to display a second visual effect related to the timeline of the first visual effect to the user, the sensor assembly is arranged on the second presentation device and is configured to measure the orientation of the second presentation device, and the signal transceiver is connected to the sensor assembly; a hub module, which is connected to the first presentation device; and one or more transmission devices, which are connected to the hub module and the second presentation device.

According to an optional embodiment, the first presentation device includes one or more of the following items: a movie screen, an LCD screen and a stage; the first audio device includes a speaker; the first visual effect includes a video stream and a live performance; and the second visual effect includes one or more of the following items: a video stream, a preset three-dimensional image and a preset two-dimensional image.

According to an optional embodiment, the second presentation device is an integrated head-mounted display and includes an energy storage device and a processor; the signal transceiver is arranged on the second presentation device; the energy storage device is arranged on the head support and supplies power to the second presentation device, the sensor assembly and the signal transceiver with the aid of a power connector; and the processor is arranged on the head support and connected to the second presentation device, the sensor assembly and the signal transceiver with the aid of a data connector to adjust the second visual effect in the second presentation device according to the measurement results of the sensor assembly and the data received by the signal transceiver.

According to an optional embodiment, the second presentation device is a split head mounted display and includes an energy storage device and a processor; the signal transceiver is separately arranged from the second presentation device and the head support; the energy storage device is separately arranged from the head support and supplies power to the second presentation device, the sensor assembly and the signal transceiver by means of a power connector; and The processor is disposed separately from the head support and is connected to the second presentation device, the sensor assembly and the signal transceiver by means of a data connection to adjust the second visual effect according to the measurements of the sensor assembly and the data received by the signal transceiver.

According to an optional embodiment, the second presentation device includes an outer lens and a self-luminous display component superimposed on the outer lens.

According to an optional embodiment, the self-luminous display component is configured to provide different images to the left eye and the right eye of the user respectively.

According to an optional embodiment, the self-luminous display component is configured to provide the same picture to the left eye and the right eye of the user respectively.

According to an optional embodiment, the sensor assembly includes one or more of the following items: a compass, a video camera, a proximity sensor, a light sensor, a position sensor, a gyroscope and a GPS module.

According to an optional embodiment, the one or more transmission devices are integrated in the hub module.

According to an optional embodiment, the hub module is connected to the first presentation device and the transmission device via a data line; and the one or more transmission devices are connected to the second presentation device via a wireless network.

With the help of the extended reality display system of the present application, the first visual effect in the user's field of view can be superimposed with the second visual effect in the second presentation device (such as MR glasses or AR glasses). In this way, when the user watches the first visual effect in the first presentation device, he can obtain a sense of presence similar to watching a real stage, and can also interact with other users wearing the second presentation device with the help of the hub module. In addition, the extended reality display system of the present application has low requirements for the venue, and can use existing cinemas to simulate the real stage, greatly reducing the implementation cost.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects of the present application will be more fully understood from the detailed description below in conjunction with the following drawings. It should be noted that the scales of the drawings may be different for the purpose of clear description, but this will not affect the understanding of the present application.

FIG. 1 is a schematic diagram of an extended reality display system according to the present application.

FIG. 2 is a schematic diagram of a first presentation device of the extended reality display system according to the present application.

FIG. 3 a is a schematic diagram of a first implementation of a second presentation device of an extended reality display system according to the present application.

FIG. 3 b is a schematic diagram of a second embodiment of a second presentation device of the extended reality display system according to the present application.

DETAILED DESCRIPTION

The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described here. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art. It should be noted that when an element is referred to as "fixed to" or "set to" another element, it can be directly on another element or indirectly on the other element. When an element is referred to as "connected to" another element, it can be directly connected to another element or indirectly connected to the other element. It should be understood that the orientation or positional relationship indicated by the terms "length", "width", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. is based on the orientation or positional relationship shown in the accompanying drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation to the present application. In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "multiple" and "several" mean two or more, unless otherwise clearly and specifically defined.

FIG1 is a schematic diagram of an extended reality display system according to the present application. The extended reality display system 1 includes a first presentation device 11, a second presentation device 12, a hub module 13, and one or more transmission devices 14 (only one is shown in the figure). The first presentation device 11 is used to output a first visual effect and an audio stream to the user. The second presentation device 12 is used to output a second visual effect to the user. The detailed structure of the first presentation device 11 and the second presentation device 12 will be described in detail below. The hub module 13 includes a memory 131 and is connected to the first presentation device 11. One or more transmission devices 14 are connected to the hub module 13 and the second presentation device 12. Since the first presentation device 11 is usually stationary, and the second presentation device 12 is usually worn by the user and moves with the user, the hub module 13 The first presentation device 11 and the one or more transmission devices 14 may be connected via a data line (ie, a wired connection), and the one or more transmission devices 14 may be connected to the second presentation device 12 via a wireless network (ie, a wireless connection).

The extended reality display system may include multiple second presentation devices 12. The distance between the second presentation device 12 and the first presentation device 11 does not exceed the first distance. The distance between adjacent second presentation devices 12 does not exceed the second distance. The purpose of such an arrangement is to provide the best viewing experience to the user and make the viewing experience of each user substantially the same.

The hub module 13 includes a non-volatile memory, a data processing unit and a read/write memory. The non-volatile memory has a memory element in which a computer program is stored. When the computer program is executed, the data processing unit outputs a first visual effect and an audio stream to the first presentation device 11, and outputs a second visual effect to the second presentation device 12. The hub module 13 further includes a bus controller, a serial communication port, an I/O component, an A/D converter, a time and date input and transmission unit, an event counter, an interrupt controller, etc. (not shown). In the case where the data processing unit is described as performing a specific function, this means that the data processing unit affects a specific part of a program stored in the non-volatile memory or a specific part of a program stored in the read/write memory.

The one or more transmission devices 14 may be commercially available routers or access points (APs) to transmit data and instructions from the hub module 13 to one or more second presentation devices 12 within a certain range. The one or more transmission devices 14 may be integrated in the hub module 13.

FIG2 is a schematic diagram of a first presentation device of an extended reality display system according to the present application. The first presentation device 11 includes a first presentation device 111 and a first audio device 112. The first presentation device 111 is used to show a first visual effect to a user, which may be a display device such as a commercially available movie screen or LCD screen, or a performance venue such as a stage or a theater. In other words, the first visual effect includes a video stream and a live performance. The size of the first presentation device 111 is large enough so that a user at a distance from the first presentation device 111 can also clearly see the first visual effect displayed on the first presentation device 111. The first audio device 112 may be a speaker, which is used to play an audio stream to the user. The output power of the first audio device 112 is large enough so that a user at a distance from the first audio device 112 can also clearly hear the audio stream played in the first audio device 112. The time axis of the first visual effect, the audio stream, and the second visual effect are related. The first visual effect, the audio stream, and the second visual effect may be pre-recorded or generated, pre-rehearsed, or generated in real time.

FIG. 3a is a first embodiment of a second presentation device of an extended reality display system according to the present application. Schematic diagram of the type. The second presentation device 12 includes a second presentation device 121, a head support 122, a sensor assembly 123, and a signal transceiver 124. The head support 122 (e.g., temples) engages the user's head, and a physiological sensor for sensing the physiological condition of the user's face may be embedded therein. The physiological sensor is used to notify the hub module 13 when an abnormality in the user's physiological condition is detected. The hub module 13 may suspend or terminate the operation of the first presentation device 11 and/or the second presentation device 12 according to the user's physiological condition. The second presentation device 121 is coupled to the head support 122 to be supported thereon, and is used to display the second visual effect to the user. The sensor assembly 123 is disposed on the second presentation device 121 and is used to measure the orientation of the second presentation device 12. The orientation of the second presentation device 12 is related to the display mode of the second visual effect. Specifically, when the user turns his head, the orientation of the second presentation device 12 changes. At this time, the second presentation device 121 causes the display position of the second visual effect therein to change accordingly to ensure that the second visual effect does not block the main part of the first visual effect in the user's field of view. The signal transceiver 124 is connected to the sensor assembly 123 to send data from the sensor assembly 123 to the hub module 13 and receive control instructions from the hub module 13. The sensor assembly 123 includes one or more of the following devices: a compass, a video camera, a proximity sensor, a light sensor, a position sensor, a gyroscope, and a GPS module.

The second presentation device 12 further includes a mechanical connector, a power connector, a data connector, an energy storage device 125, and a processor 126. The mechanical connector generally includes a buckle, a hinge, or an adhesive hook, etc., which is disposed between the second presentation device 121 and the head support 122 to assist the head support 122 to be removably coupled to the second presentation device 121. The power connector generally includes a wire, etc., which is connected between the second presentation device 121 and the head support 122 to transfer power to the physiological sensor and the sensor assembly 123. The data connector generally includes a data line, etc., which is connected between the second presentation device 121 and the head support 122 to transmit the signal from the physiological sensor and the sensor assembly 123 to the signal transceiver 124 disposed on the second presentation device 121, and then to the hub module 13.

In the first embodiment, the second presentation device 12 is an integrated head mounted display. The signal transceiver 124 is arranged on the second presentation device 121. The energy storage device 125 (e.g., a battery) is arranged on the head support 122 and supplies power to the second presentation device 121, the physiological sensor, the sensor assembly 123, and the signal transceiver 124 by means of a power connection. The processor 126 is arranged on the head support 122 and is connected to the second presentation device 121, the sensor assembly 123, and the signal transceiver 124 by means of a data connection to adjust the second visual effect in the second presentation device 121 according to the measurement results of the sensor assembly 123 and the data received by the signal transceiver 124.

The second presentation device 121 includes an external lens and a self-luminous display component superimposed on the external lens. The external lens can be a polarized lens, a shading lens, and an ordinary lens. The advantages of setting a polarized lens in the second presentation device 121 include: being able to effectively filter out reflections and glare, improving visual clarity and comfort; being able to prevent ultraviolet rays and harmful light from damaging the eyes and protecting eye health; being able to make colors more vivid and enhance the beauty and three-dimensional sense of the picture. In the case where the external lens is a polarized lens, the polarization directions of the polarized lens in the left eye and the right eye are orthogonal to filter out a portion of the picture in the first visual effect displayed in the first presentation device 111, respectively, so that the pictures seen by the left eye and the right eye of the user are different. In conjunction with this, the self-luminous display component is configured to provide different pictures to the left eye and the right eye of the user, respectively, so that the user obtains the perception of a three-dimensional image. The self-luminous display component can also be configured to provide the same picture to the left eye and the right eye of the user, respectively, so that users who are not suitable for viewing three-dimensional images obtain the perception of a two-dimensional image.

FIG3 b is a schematic diagram of a second embodiment of the second presentation device of the extended reality display system according to the present application. Compared with the first embodiment, in the second embodiment, the second presentation device 12 is a split head-mounted display. The signal transceiver 124 is separately arranged from the second presentation device 121 and the head support 122. The energy storage device 125 is separately arranged from the head support 122 and supplies power to the second presentation device 121, the physiological sensor, the sensor assembly 123 and the signal transceiver 124 by means of a power connector. The processor 126 is separately arranged from the head support 122 and is connected to the second presentation device 121, the sensor assembly 123 and the signal transceiver 124 by means of a data connector to adjust the second visual effect in the second presentation device 121 according to the measurement results of the sensor assembly 123 and the data received by the signal transceiver 124.

It can be understood by those skilled in the art that, in addition to outputting the first visual effect and the audio stream to the first presentation device 11 and the second visual effect to the second presentation device 12 in a purely computer-readable program code, the operation can also be logically programmed to enable the hub module to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the hub module can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the devices for implementing various functions can be considered as structures within the hardware component accompanied by software modules.

The following describes the actual application scenario of the extended reality display system of the present application in an example form. First, the user sits in front of the first presentation device 11 (such as a movie screen) and wears the second presentation device 12 (such as MR glasses). Then, the first presentation device 11 plays a pre-recorded video. The second presentation device 12 plays the visual effects (such as simulated lights, fireworks, etc.) synchronized with the timeline of the concert images. The part of the user's field of view that is outside the display range of the first presentation device 111 of the first presentation device 11 (i.e., beyond the boundary of the first presentation device 111) can be filled with the three-dimensional image in the second presentation device 12. In this way, when the user watches the three-dimensional picture in the first presentation device 11, he can obtain a sense of presence similar to watching a real stage, and can also interact with other users wearing the first presentation device 11 with the help of the hub module 13.

The foregoing description of the embodiments has been provided for illustrative and descriptive purposes. It is not intended to be exhaustive or to limit the embodiments to the described variants. Many modifications and variations will be apparent to those skilled in the art. The embodiments are selected and described in order to best illustrate the principles and practical applications, so that those skilled in the art can understand the embodiments in their various embodiments and various modifications suitable for their intended use. Within the framework of the embodiments, the above-mentioned components and features can be combined between different embodiments.

Claims (10)

An extended reality display system, comprising: A first presentation device (11), comprising a first presentation means (111) and a first audio device (112), wherein the first presentation means (111) is configured to display a first visual effect to a user, and the first audio device (112) is configured to play an audio stream associated with a timeline of the first visual effect to the user; A second presentation device (12), comprising a second presentation apparatus (121), a head support (122), a sensor assembly (123) and a signal transceiver (124), wherein the head support (122) engages with the user's head, the second presentation apparatus (121) is coupled to the head support (122) to be supported thereon and is configured to display a second visual effect related to a timeline of the first visual effect to the user, the sensor assembly (123) is disposed on the second presentation apparatus (121) and is configured to measure an orientation of the second presentation apparatus (12), and the signal transceiver (124) is connected to the sensor assembly (123); A hub module (13) connected to the first presentation device (11); and One or more transmission devices (14) connected to the hub module (13) and the second presentation device (12). The extended reality display system according to claim 1, wherein: The first presentation device (111) includes one or more of the following items: a movie screen, a liquid crystal screen and a stage; The first audio device (112) includes a speaker; The first visual effect includes video streaming and live performance; and The second visual effect includes one or more of the following items: a video stream, a preset three-dimensional image, and a preset two-dimensional image. The extended reality display system according to claim 1, wherein: The second presentation device (12) is an integrated head mounted display and comprises an energy storage device (125) and a processor (126); The signal transceiver (124) is arranged on the second presentation device (121); The energy storage device (125) is arranged on the head support (122) and supplies power to the second presentation device (121), the sensor assembly (123) and the signal transceiver (124) by means of a power connection; and The processor (126) is arranged on the head support (122) and is connected to the second presentation device (121), the sensor assembly (123) and the signal transceiver (124) by means of a data connection to adjust the second visual effect in the second presentation device (121) according to the measurement results of the sensor assembly (123) and the data received by the signal transceiver (124). The extended reality display system according to claim 1, wherein: The second presentation device (12) is a split head mounted display and comprises an energy storage device (125) and a processor (126); The signal transceiver (124) is arranged separately from the second presentation device (121) and the head support (122); The energy storage device (125) is separately arranged from the head support (122) and supplies power to the second presentation device (121), the sensor assembly (123) and the signal transceiver (124) by means of a power connection; and The processor (126) is arranged separately from the head support (122) and is connected to the second presentation device (121), the sensor assembly (123) and the signal transceiver (124) by means of a data connection to adjust the second visual effect according to the measurement results of the sensor assembly (123) and the data received by the signal transceiver (124). The extended reality display system according to claim 1, wherein: The second presentation device (121) comprises an outer lens and a self-luminous display component superimposed on the outer lens. The extended reality display system according to claim 5, wherein: The self-luminous display member is configured to provide different images to the left eye and the right eye of the user, respectively. The extended reality display system according to claim 5, wherein: The self-luminous display member is configured to provide the same picture to the left eye and the right eye of the user, respectively. The extended reality display system according to claim 1, wherein: The sensor assembly (124) includes one or more of the following: a compass, a video camera, a proximity sensor, a light sensor, a position sensor, a gyroscope, and a GPS module. The extended reality display system according to claim 1, wherein: The one or more transmission devices (14) are integrated in the hub module (13). The extended reality display system according to claim 1, wherein: The hub module (13) is connected to the first presentation device (11) and the transmission device (14) via a data line; and The one or more transmission devices (14) are connected to the second presentation device (12) via a wireless network.