TW201722520A - System and method for delivering media over network - Google Patents

Abstract

A method for delivering media from a server to a client device over a network is disclosed. A Virtual-Reality (VR) scene application running on the server generates a virtual VR 3D environment containing 3D models. The server checks the status of each 3D model in a predetermined order, and then, only those 3D models which are not pre-stored in the client device will all be rendered by the server into a left eye frame and a right eye frame of 2D video stream. The server then sends to the client device the frames and meta data of the 3D models which are pre-stored in the client device via the network. The client device uses a combined VR frame of these frames as a background for rendering the 3D models which are pre-stored in the client device so as to generate a mixed VR frame of video stream for output.

Description

System and method for transmitting media over a network

The present invention relates to a system and method for transmitting media such as images and sounds through a network, and more particularly to a method for rendering a 3D object of a virtual reality (VR) image on a user device. The 2D video stream of the VR scene provided by the server is combined by rendering the 3D object at the user device.

In the past few years, online games have become the world trend. With the development of cloud computing related systems and technologies, a technology that uses server to stream game content and provide services has also been introduced.

A traditional method of providing a cloud game service relies on a server for almost all operations, that is, when a cloud game service is to be provided, the server needs to generate a 3D object including a plurality of objects that can be moved or controlled by the participant. Virtual 3D environment. In the known art, these 3D objects may contain sound effects, which are then combined with the control of the participant (player), which combines the virtual 3D environment with the 3D object and renders it onto a stereoscopic 2D game screen on the gaming machine. Then, the server immediately transmits the image and stereo, and transmits the 2D video stream containing the sound to the player through the network, and the player device only needs to decode and display the 2D video stream after receiving. No additional 3D rendering calculations are required. However, the aforementioned conventional techniques of performing rendering calculations for a large number of players on the same server will result in an excessive load on the server performing 3D rendering calculations; in addition, the screens seen by the player are transmitted in a destructively compressed 2D video stream. Therefore, both the quality of the image and the sound are inferior to the quality of the original displayed 3D object, and the large amount of network communication bandwidth between the server and the player device becomes a major problem.

Virtual-Reality (VR) technology has become popular in the near future. In order to provide a human VR visual experience, the virtual VR scene must contain a dedicated human The image viewed by the left eye and another image for the right eye of the human. The present invention provides a system and method for transmitting media such as images and sounds over a network by combining 3D objects in a user device to combine 2D video streams of a VR scene provided by a server.

Accordingly, the main object of the present invention is to provide a system and method for transmitting media such as video and audio through a network, which can reduce the load of the server, improve the quality of images and sounds displayed on the user device, and save the server. Communication bandwidth with the user device; the method of the present invention is characterized in that the user device renders a 3D object (also referred to as a 3D model) to combine the 2D video stream of the VR scene provided by the server to reach the user device. The result of rendering a 3D object of a virtual reality (Virtual-Reality; VR) image.

To achieve the above object, the present invention provides a system and method for transmitting media over a network, the media including a plurality of images. The system includes a server and a user device, and the method includes the following steps: Step (A): executing a virtual reality (VR) application on a server to generate a virtual reality comprising a plurality of 3D models a VR 3D environment, each of the 3D models being associated with a state indicating whether the 3D model is pre-existing in a user device; and step (B): the server checks the states of the 3D models to determine which one The 3D model is to be encoded as a left eye frame and a right eye frame included in a 2D video stream, and is encoded by encoding the 3D models that are not pre-existing in the user device to the left eye frame and the In the right eye frame; step (C): the server transmits at least the left eye frame and the right eye frame of the 2D video stream to the user device through the network; wherein the server also not The 3D models in the user device are transmitted to the user device in a predetermined order; and when the user device receives the 3D models transmitted by the server, the user device stores the 3D models And send a message to the wait a server to change the states of the 3D models and indicating that the 3D models are now pre-existing in the user device; and step (D): the user device receives the left eye shadow received from the server And the right eye frame decoding, and using the left eye frame and the right eye frame as rendering the pre-existing users A background image of the 3D model in the device, but not included in the left eye frame and the right eye frame, thereby generating a hybrid VR frame as an output video stream containing one of the VR scenes.

In an embodiment, in the step (D), after the user device decodes the left eye frame and the right eye frame received by the server, the user device further displays the left eye frame and the right eye shadow. The cells are merged into a combined VR frame, and then the combined VR frame is used as the background image to render the 3D models of the pre-existing user devices but not included in the left eye frame and the right eye frame. Thereby generating the hybrid VR frame as the output video stream containing the VR scene.

In an embodiment, the server further includes: a VR scene transmitter, which is a library compiled in the VR application or dynamically linked to the VR application during execution time; wherein the VR scene The transmitter maintains a list of all of the states including all of the 3D model and each of the 3D models, the state is used to indicate that the state of the 3D model is "Not Ready", "Loading" And one of "Ready for Client"; and a VR scene server for executing a server program on the server by the VR application; wherein the VR scene server system As a relay station of the VR scene transmitter and the message transmission between the user devices, the VR scene server also downloads the server program as one of the 3D models necessary for the user device to download from the server.

In an embodiment, the user device further includes: a VR scene client, which is a program running on the user device, configured to generate the output video stream and communicate with the server through the network; A frame combiner is configured to merge the left eye frame with the right eye frame into the merged VR frame; and a VR scene cache for storing at least one 3D model previously downloaded from the server.

1‧‧‧Server

3‧‧‧Network (base station)

4‧‧‧Network

21‧‧‧User device (smart phone)

22‧‧‧User device (notebook)

23‧‧‧User device (desktop)

51‧‧‧User eyes

52‧‧‧Projection surface

70‧‧‧Server

71, 71a‧‧ people

72, 72a‧‧‧ houses

73, 73a‧‧ frames

74‧‧‧User device

75‧‧‧Video Streaming Frame

81, 81a‧‧‧ voice

82, 82a‧‧‧ voice

83, 83a, 1711, 1712, 1713‧‧ ‧ frames

85‧‧‧Video Streaming Frame

100, 1100‧‧‧ application

110, 1110‧‧‧ Scene Transmitter (Library)

120, 1120‧‧‧ Scene Server

121‧‧‧data packer

170, 1170‧‧‧ Scene Client (Program)

1111, 1171‧‧ ‧ frame combiner

190, 1190‧‧‧ Scene cache

60~67, 661, 60a~67a, 661a‧‧ steps

101~114, 122, 124, 172, 174, 176, 192, 194, 1101, 1112~1115, 1122, 1124, 1172, 1174, 1176, 1192, 1194‧‧

1 is a schematic diagram of a standard embodiment of a system for transmitting media over a network according to the present invention. Figure.

2 is a schematic diagram of an embodiment of a system architecture of the present invention.

3A is a flow chart of an embodiment of a method for transmitting media over a network according to the present invention.

FIG. 3B is a flow chart of another embodiment of a method for transmitting media through a network according to the present invention.

4A, 4B and 4C are diagrams showing an embodiment of how the method of the present invention transmits a video stream and a 3D model.

Figures 5A, 5B and 5C show a schematic diagram of how the method of the present invention determines which 3D model has to be encoded into a frame.

6A, 6B, and 6C are diagrams showing an embodiment of how the method of the present invention transmits a video stream with sound and 3D sound.

Figures 7A, 7B and 7C show a schematic diagram of how the method of the present invention determines which 3D sound is to be encoded into a video stream with sound.

Figure 8 is a schematic diagram of an embodiment of a system architecture of a virtual reality (VR) scenario system of the present invention.

Figure 9 is a diagram showing an embodiment of the function of the frame combiner of the virtual reality (VR) scene system of the present invention.

FIG. 10 is a schematic diagram of a second embodiment of a system architecture of a VR scene system of the present invention.

11 is a schematic diagram of a third embodiment of a system architecture of a VR scenario system of the present invention.

In order to more clearly describe the system and method for transmitting media through the network as proposed by the present invention, the following detailed description will be made in conjunction with the drawings.

One of the applications of the present invention is an online game in which a player uses a user device to play a game on a server through a network, and the server acts according to the instructions of the player and generates video on the user device; for example, when When a player takes action on the user device, this action is It will be transmitted to the server device, and an image will be calculated thereon, and then the image will be transmitted back to the user device. In many online games, the 2D image produced by the server includes 3D rendering of other objects in the line of sight.

The invention provides a 3D model and 3D sound required by the user device by the server, and performs 3D rendering analysis of the object in the line of sight range between the server and the user device. For example, the server provides some or all 3D models. And 3D sound to the user device, and interpret each 3D model or 3D sound related interpretation materials, such as position, orientation and status data.

For example, at the beginning of the game, all game-related images (including related 3D rendering) on the user device are generated by the network routing server to become a stereo 2D video stream. The system of the present invention transmits the 3D model and the 3D sound and other media and the rendering information to the user device through the network in the line of sight, and the object closer to the eye (the eye) is preferentially pushed. The system of the present invention performs 3D model and 3D sound rendering as much as possible on the user device, and secondly performs rendering such as 3D model or 3D sound on the server.

The 3D model or a 3D sound is stored on the user device, and the server only needs to provide the interpretation data of the object (3D model or 3D sound) to the user device, and the user device can render the objects according to the present and present the result to the user device. Any stereo 2D video provided by the server; the server will not render this 3D model and 3D sound unless requested by the user device. The arrangement of the method of the present invention saves GPU computing on the server, and the server can maintain a dynamic database, including 3D models and 3D sounds, to improve the performance of communication with the user.

In the present invention, the user device displays the following combinations: (a) a 3D scene rendered on the server, and the result is a stereo 2D video stream, transmitted to the user and played by the user device. And (b) downloading from the server and storing on the user device, and the user device renders the 3D model and the result of the 3D sound by itself, the stereo 2D video stream and the 3D model and 3D sound rendered on the user device. The mix will create a colorful 3D scene and an exciting surround sound effect while reducing bandwidth usage.

In one embodiment, the stereo 2D video stream transmitted to the user device carries the 3D model and the 3D sound interpretation data, and the user device detects whether the 3D model and the 3D sound are present, if not, the user The device will download the required 3D model and 3D sound from the server. After downloading, the user device will store it and create a list of materials for later reconstruction. In this way, the delay of the video stream and the need for a large amount of bandwidth will be improved. And the user device will render it by itself, and the better quality image will be obtained (because it has not been compressed by video).

The aforementioned interpretation data will allow the user device to correctly mix the results rendered by the user device with the 3D model and the 3D sound without missing or repeating any 3D model or 3D sound, and the stereo provided by the server. 2D video streaming; as described above, when the user device stores all the required 3D models and 3D sounds, the user device can reconstruct the complete 3D scene and sound. At this time, the server no longer needs to perform any rendering, Until a new one is added but a new 3D model or 3D sound is not stored on the user device side, and when a new 3D model is encountered, the server will use this new 3D model and all subsequent objects. Render until the user device can render this new 3D model on its own. In addition, if a new 3D sound is encountered, the server will render the 3D sound until it is available for the user device.

The user device will store (cache) the downloaded 3D model and 3D sound on its own storage device as much as possible, so as to avoid repeated downloading in the future execution, and the bandwidth cost of the network will further Reduced, and if it can't be saved, the download and rendering will be done at execution time.

FIG. 1 is a schematic diagram of a standard embodiment of a system for transmitting media over a network according to the present invention. The server 1 is used to execute a service providing application, which may be, but is not limited to, a cloud online game service; a plurality of user devices 21, 22, 23 may be linked (logged in) through a network 4 to The server 1 uses the services provided by the application running on the server 1. In this embodiment, the network 4 is an internet network, and the user devices 21, 22, and 23 are any electronic devices connectable to the Internet, such as, but not limited to, a smart phone 21. A tablet, a notebook computer 22, a desktop computer 23, a video game console, or a smart TV, some of the user devices 21, 22 are wirelessly connected to the network 4 through a mobile base station, while other user devices are It can be connected to the network 4 through a router and wired; the application running on the server 1 can generate a virtual 3D environment containing a plurality of 3D models and 3D sounds, each 3D model or 3D sound. The collocation state indicates whether the 3D model or the 3D sound is pre-stored in a user device 21, 22, 23. In a preferred embodiment of the present invention, each user device has a corresponding independent application, that is, an application only provides a service to a user device, but multiple applications can be simultaneously Executed on one server to provide services to a large number of user devices. As shown, the user devices 21, 22, 23 are connected to the server 1 via the network 4 to obtain media generated by the application and containing at least some 3D models and 3D sounds. The system architecture and its features are detailed in Figure 2 and related descriptions.

FIG. 2 is a schematic diagram of an embodiment of a system architecture of the present invention.

In the present invention, the application 100 is run on a server 1 for generating a 3D image 3D sound rendering result, typically a 3D game. The 3D scene transmitter 110 is a library that is statically linked to the application 100 at compile time or dynamically linked during execution of the application 100; the 3D scene client (program) 170 is one Programs executed on the user devices 21, 22, 23 for generating and outputting 3D images and 3D sound rendering results generated by the application 100. In this embodiment, each user device 21, 22, 23 corresponds to a separate application 100 and a scene transmitter 110.

In the present invention, the 3D scene client 170 and the 3D scene cache 190 form a program and execution method of the user terminal, so as to exert the computing power of the user device itself to render the 3D model and the 3D sound.

The 3D scene server 120 is a server program executed on the server 1 together with the application 100, and serves as a 3D scene transmitter 110 of the server 1, and a 3D scene client of the user devices 21, 22, and 23. 170, relay stations for message transmission. At the same time, it is also used as a file download server for the user terminals 21 of the user devices 21, 22, 23 to download the necessary 3D model and 3D sound from the server 1. The 3D scene transmitter 110 maintains a list of all 3D models and 3D sounds, as well as the state of the model or sound, which is used to indicate the status of each 3D model or 3D sound as (1) "Not Ready (not One of "" ready", (2) "Loading", and (3) "Ready for Client".

The main program of the application 100 transmits the 3D scene information to the 3D scene transmitter 110 by calling the library through the API (path 101 of FIG. 2), and the 3D scene information includes the name, location, speed, and attribute. , seating and all other 3D models and 3D sound rendering required. After the 3D scene transmitter 110 receives such material, the following program can be executed.

Step (a): For the 3D model, all the 3D models to be rendered are sorted, and the ordering manner may be, relative to a virtual position (such as a 3D projection surface or a user's eyes), sorted from near to far.

For 3D sound, all 3D sounds to be rendered are sorted in such a way as to be relative to a virtual position (such as a 3D projection surface or the user's eyes), sorted from near to far.

In some cases, one of the 3D models A in the 3D scene may be wrapped or superimposed on another 3D model B. For example, model A may be a house and model B may be a table in a house. In this case, which model is closer to the simulated position is an ambiguous problem. At this time, model A and model B will be regarded as the same 3D model, which can be called 3D model (A+B).

Some of the known information about the scene can be used to assist in sorting. For example, the ground in the game can be considered as a large and flat 3D model under other 3D objects. Usually, the user's eyes will be higher than the ground, therefore, The 3D model of the ground needs special handling in sorting to avoid it appearing in front of other 3D models.

Step (b): For the 3D model, look for the first 3D model "M" that does not have a "Ready for Client" state from the closest point (closest to the eye), in other words, the state of the first 3D model "M" It is in the "Not Ready" state. (After that, the "Not Ready" state is referred to as the NR state); of course, there may be no such 3D models (for example, all 3D models to be displayed are marked as " Ready for Client status.

For the 3D voice, look for the first 3D sound "S" that does not have the "Ready for Client" status from the closest point (closest to the eyes), in other words, the state of the first 3D sound "S" is "Not Ready". "Status, (after this, the "Not Ready" status is simply referred to as the NR status); of course, there may be no such 3D sounds present (eg all 3D sounds to be displayed are marked as "Ready for Client" status) ).

Step (c): For the 3D model, the server renders the 3D model M and all subsequent 3D models, that is, all 3D models that are farther away from the eye than M. (If there is no 3D model M, it is displayed on a black screen), the result after encoding and rendering is a 2D video stream frame.

For 3D voices, all 3D sounds that do not have a "ready for client" state are rendered (played) on server 1 (if there is no such 3D sound, a muting is generated), and then the result of the encoding is rendered as a step (C) Stereo of 2D video streaming video frames. Note: The 3D sound after the 3D model S is only rendered when its state is not "Ready for Client", which is different from the 3D model in step (C).

Step (d): The following six pieces of information are transmitted to the 3D scene server 120 (path 112): [Info 112-A], [Info 112-B], [Info 112-C], [Info 112-D], [ Info 112-E] and [Info 112-F]), and the 3D scene server 120 transmits the above information to the 3D scene client 170 (path 122).

[Info 112-A] is the status information (or interpretation data) of all 3D models before the 3D model M. Note that there may be no such models exist. This type of model has "Ready for Client" The state means that these models have been preloaded on the client device, and the 3D scene client (program) 170 on the client devices 21, 22, 23 can already render the models themselves. In order to reduce the data transmission bandwidth, the 3D scene transmitter 110 does not have to transmit all the state information, as long as the difference between the current rendering and the last rendering is transmitted in the state information.

[Info 112-B] If the server finds the 3D model M and the status of its user device is "Not Ready", the server will change its user status to "Loading" and send a download instruction of a 3D model M. The client is required to download the 3D model M; if the user status is already "Loading", do not send any indication, as the download instruction has been sent.

[Info 112-C] is the encoded video stream frame in step (C).

[Info 112-D] refers to the status information (or interpret data) of all 3D sounds with the status "ready for client" (and may not have such 3D sounds). These sound types all have "Ready for Client". The state means that the sounds have been preloaded on the client device, and the 3D scene client (program) 170 on the client devices 21, 22, 23 can already render (play) the sounds themselves. In order to reduce the data transmission bandwidth, the 3D scene transmitter 110 does not have to transmit all the state information, as long as the difference between the current rendering and the last rendering is transmitted in the state information.

[Info 112-E] If the server finds the 3D sound S and its user status is "Not Ready", change its user status to "Loading" and send a 3D sound S download instruction, requesting the client to download this 3D sound S; if the user status is already "Loading", do not send any instructions, as the download instructions have been sent.

[Info 112-F] is the encoded stereo in step (C).

Each time the main program of the application 100 updates the new 3D scene data to the 3D scene transmitter 110, steps (a) to (d) are repeated. Usually, the main program of the application 100 is updated in each rendering. Such information.

Once the 3D scene client 170 receives the aforementioned data, the rendering procedure described later is performed.

Step (i): Decode the video frame of [Info 112-C] and use this frame as the background for subsequent 3D model rendering; in addition, decode the stereo with [Info 112-F] video and use it as the subsequent 3D sound. The background sound rendered.

Step (ii): rendering all 3D models in [Info 112-A] on the video frame encoded in step (i), in order to reduce the network bandwidth occupation, the 3D scene client 170 will store this one [Info] 112-A] information into the memory, so the next 3D scene transmitter 110 can only transmit the difference between the state of the next rendering and the current rendering [Info 112-A], and does not need to transmit all state information; similarly, When all 3D sounds belonging to [Info 112-D] are rendered, they are mixed with the stereo decoded in step (i), and in order to reduce the network bandwidth occupation, the 3D scene client 170 will store this one [ Info 112-D] information is stored in the memory, so the next 3D scene transmitter 110 can only transmit the difference between the next rendering and the current rendering [Info 112-D], and does not need to transmit all state information.

Step (iii): In step (ii), a stereo video frame transmitted from the server is mixed, and the 3D model and the 3D sound rendered by the 3D scene client 170 are self-rendered, and the mixed result is outputted into a sound. The output video stream (path 176).

If the status of [Info 112-B] is provided, the 3D scene client 170 will process the 3D model M according to the following procedure.

Step (I): Search 3D scene cache 190 (path 174), which includes a 3D model database previously downloaded and stored in user devices 21, 22, 23.

Step (II): If there is already a 3D model M in the 3D scene cache 190, step (V) is performed.

Step (III): If there is no 3D model M in the 3D scene cache 190, the 3D scene client 170 will send a download request to the 3D scene server 120 (path 172), and the 3D scene server 120 will return The data of the 3D model M is given to the 3D scene client 170 (path 124).

Step (IV): After the 3D model is completely downloaded, the 3D scene client 170 is about to be stored in the 3D scene cache 190 (path 194), whereby when there is a similar demand next time, there is no need to download again.

Step (V): The 3D scene client 170 will extract the 3D model M (path 192) from the 3D scene cache 190.

Step (VI): Once the download is completed (or has been downloaded earlier), the 3D scene client 170 can extract the 3D model M; the 3D scene client 170 will send a "3D model is ready on client". The message on the device is sent to the 3D scene server 120 (path 113), and the 3D scene server 120 will forward the message to the 3D scene transmitter 110 (path 114).

Step (VII): After receiving the message, the 3D scene transmitter 110 changes the state of the 3D model M from "Loading" to "Ready for Client".

Step (VIII): In the next rendering, the 3D scene transmitter 110 will know The 3D model M is now preloaded in the user device, so the 3D scene client 170 will be rendered by itself, so the server 1 will no longer need to render the 3D model M.

If the status of [Info 112-E] is provided, the 3D scene client 170 will prepare the 3D sound S according to the following procedure. (similar to the foregoing for those described in [Info 112-B])

Step (I): Search 3D scene cache 190 (path 174), which includes a 3D sound database previously downloaded and stored in user devices 21, 22, 23.

Step (II): If the 3D scene cache 190 is provided with a 3D sound, step (V) is performed.

Step (III): If the 3D scene cache 190 does not have a 3D sound, the 3D scene client 170 will send a download request to the 3D scene server 120 (path 172), and the 3D scene server 120 will return The 3D sound data is given to the 3D scene client 170 (path 124).

Step (IV): After the 3D sound is completely downloaded, the 3D scene client 170 is about to be stored in the 3D scene cache 190 (path 194), whereby when there is a similar demand next time, there is no need to download again.

Step (V): The 3D scene client 170 will extract the 3D sound S (path 192) from the 3D scene cache 190.

Step (VI): Once the download is completed (or has been downloaded earlier), the 3D scene client 170 can extract the 3D sound S; the 3D scene client 170 will send a "3D sound is ready on client". The message on the device is sent to the 3D scene server 120 (path 113), and the 3D scene server 120 will forward the message to the 3D scene transmitter 110 (path 114).

Step (VII): After receiving the message, the 3D scene transmitter 110 changes the state of the 3D sound S from "Loading" to "Ready for Client".

Step (VIII): In the next rendering, the 3D scene transmitter 110 will know that the 3D sound S is now preloaded in the user device, so the 3D scene client 170 will be requested to render (play) by itself, yes, servo Device 1 no longer needs to render this 3D sound S.

At the very beginning, there are no 3D models and 3D sounds in the user devices 21, 22, 23, so the 3D scene transmitter 110 will render all 3D models and 3D sounds and encode the result into stereo 2D video. Streaming, 3D scene transmitter 110 will download the 3D model download requirements [Info 112-B] and 3D sound download requirements [Info 112-E], from the nearest point to the 3D projection surface (or the user's eyes) The 3D scene client 170 will download each 3D model or 3D sound from the 3D scene server 120, or extract it from the 3D scene cache 190 one by one; and when more 3D models When the 3D sound is available to the 3D scene client 170, the 3D scene transmitter 110 will automatically notify the 3D scene client 170 to render the models and sounds themselves, and reduce the 3D model and 3D sound rendered by the 3D scene transmitter 110. The number, by which, the 3D model and the 3D sound in the encoded 2D video stream will be less and less until all the 3D models and 3D sounds are available on the last 3D scene client 170, and then, In this stage, only the black screen with no sound remains, in other words, the server 1 does not need to transmit the 2D video stream to the user devices 21, 22, 23, and the communication between the server 1 and the user devices 21, 22, and 23. The bandwidth occupancy can also be greatly reduced.

In the present invention, when a new 3D model N appears in the real scene, the 3D scene transmitter 110 will (1) notify the 3D scene client 170 to render all the 3D models located before the new 3D model N (relatively (2) notifying the 3D scene client 170 to download the new 3D model N, and the 3D scene transmitter 110 will render the new 3D model N and all subsequent models, The result is encoded into a 2D video stream of sound, and then the 2D video stream of the sound is transmitted to the 3D scene client 170, so that the 3D scene client 170 is a 3D model N in the user device. The 3D image and sound rendering result of the application 100 is continuously reproduced before being prepared.

When a new 3D sound T appears in the real scene, the 3D scene transmitter 110 will (1) notify the 3D scene client 170 to download the new 3D sound T, and (2) the 3D scene transmitter 110 will render this. The new 3D sound T, and the result is encoded into a stereo, and then the stereo and 2D video stream is transmitted to the 3D scene client 170, so that the 3D scene client 170 can be the 3D sound T on the user device. The 3D image and sound rendering results of the application 100 are continuously reproduced before being prepared. In this program, only the new 3D sound T is rendered, and the 3D scene transmitter 110 does not need to render all the 3D sounds behind the other 3D sounds T. This is because the essence of the sound is different from the image, and the image will be Block the display of the subsequent image, but the sound will not.

The background music can be regarded as a 3D sound with a predetermined 3D position. In order to download the background music as early as possible, the defined predetermined 3D position should be as close as possible to the user's eyes.

To reduce the load on the server or to avoid the noise generated by unstable network data transmission, the server can abandon the encoding of all 3D sounds in the video. In this case, the 3D sound is played only on the user device after it is downloaded and pre-stored in a user device.

For the 3D voice, the server 1 checks the state of the 3D sound to determine which 3D sound needs to be encoded as a stereo 2D video stream, which is encoded in a non-pre-existing manner. The 3D sound in the user device is encoded into the video frame; wherein when a 3D sound is encoded as stereo in the video frame, the volume of the left and right channels is determined by its position and the speed of the user's ear; Background music can be defined as a 3D sound effect at a predetermined location.

FIG. 3A is a flowchart of an embodiment of a method for transmitting media through a network according to the present invention; when starting to transmit an image through a network (step 60), executing an application on a server generates a plurality of In the virtual 3D environment of the 3D model (step 61), each 3D model is associated with a state indicating whether the 3D model is pre-existing in the user device.

The server then verifies the state of the 3D model (step 62) to determine which 3D model needs to be encoded into the 2D video stream frame, and the 3D model that is not pre-existing in the user device will be encoded into the frame; the server will A virtual position (usually a 3D projection surface or a user's eye) prevails, from near to far, to check the state of each 3D model one by one. In the test, when the first 3D model in the user device is not pre-existing At this time, the discovered 3D model is marked as an NR state, and then, regardless of whether the subsequent 3D model is pre-existing in the user device, the 3D model M and all 3D models behind it are encoded into the frame ( Step 63); and when the position of any 3D model is changed, or when the virtual position for the sorting reference is changed, the aforementioned test is re-executed, and whether the 3D model has to be encoded into the video frame is determined according to the latest test result. .

Step 64: After the 2D video stream frame encoding, the server will stream the 2D video stream frame and the 3D model that is not pre-existing in the user device (that is, the 3D model with the NR state and all the 3D models behind it), Transmitted to the user device in a predetermined order, the predetermined sequence is from the point closest to the point of the 3D projection surface (or the user's eye) to the point of the farthest point 3D projection surface; the user device receives the 2D video stream The frame (step 65), the user device decodes the frame from the server and uses the frame as a background for rendering the 3D model pre-existing in the user device but not included in the frame, thereby generating a sound output. The video stream is mixed (step 66); when the user device receives the 3D model transmitted by the server, the user device stores the 3D model and then transmits a message to the server, notifying the server to change the state of the 3D model. It is "pre-existing in the user device". After that, the user device mixes the video stream and the self-rendering result sent by the server into a new video.

In step 62, when a new 3D model appears in the 3D environment, whether the 3D model behind it is pre-existing in the user device, the new 3D model and all 3D models behind it are encoded into the frame.

In step 64, the server also transmits status information (or interpretation data) of the 3D model that is not encoded into the video stream frame to the user device; when receiving and verifying the status information, the user device performs the following manner. If any of the received 3D models in the status information is non-pre-existing in the user device, the user device sends a request to the server to download the 3D model (step 661), and the status information includes each The interpretation data encoded into the frame, each interpretation data includes a name of a 3D model, a position, a velocity, a direction, and an attribute and the state of each 3D model.

FIG. 3B is a flow chart showing another embodiment of a method for transmitting media through a network according to the present invention; when starting to transmit a voice over a network (step 60a), executing an application on a server generates a complex number A virtual 3D environment of 3D sounds (step 61a), each 3D sound is associated with a state indicating whether the 3D sound is pre-existing in the user device.

The server then checks the status of the 3D sound (step 62a) to determine which 3D sound needs to be encoded into the 2D video stream frame, and the 3D sound that is not pre-existing in the user device will be encoded into the frame; the server will A virtual position (usually a 3D projection surface or a user's eye) prevails, from near to far, one by one to check the state of each 3D sound, in the test, when the first 3D sound in the user device is not pre-existing At this time, the 3D sound of this discovery is marked as an NR state.

Step 64a: After encoding the video stream frame containing the sound, the server sends the 2D video stream frame of the sound and the 3D sound (that is, the 3D sound with the NR state) not pre-preserved in the user device, The predetermined sequence is transmitted to the user device in a sequence from the point closest to the 3D projection surface (or the user's eye) to the other point at the farthest point 3D projection surface; the user device receives the video string containing the sound After the stream frame (step 65a), the user device decodes the audio (ie, sound) included in the video stream and uses the audio as a 3D sound that is pre-existing in the user device but not included in the video stream frame. a background to thereby generate a mixed audio (step 66a); when the user device receives the 3D sound transmitted by the server, the user device stores the 3D sound and then transmits a message to the server, notifying the server to change the 3D The state of the sound is "now pre-existing in the user device", after which the user device mixes the audio in the video stream sent by the server with the result of self-rendering (playing) the 3D sound. , A new audio.

In step 62a, when a new 3D sound appears in the 3D environment, the new 3D sound is encoded into the 2D video stream frame with sound. However, the new 3D sound does not appear. Whether the other 3D sounds are rendered is different from the 3D model in the aforementioned step 62.

In step 64a, the server also transmits status information of the 3D sound that is not encoded into the frame to the user device; when receiving and verifying the status information, the user device performs the following manner: if the received status information is If any 3D sound is non-pre-existing in the user device, the user device sends a request to the server to download the 3D sound (step 661a), and the status information includes each of the interpreted data that is not encoded into the frame. Each interpretation data includes a name of a 3D sound, a position, a speed, a direction, and an attribute and the state of each 3D sound.

Referring to Figures 4A, 4B and 4C, there is shown a schematic diagram of an embodiment of the method of the present invention for transmitting video streams and 3D models.

As shown in FIG. 4A, when the initial user device 74 logs into the application 70 running on the server, no 3D model is pre-stored in the user device, so the server renders all 3D models (including one person 71 and its In the latter house 72), all 3D models should be displayed on the screen of the user device, and the server encodes the rendering result into a 2D video stream frame 73, and then transmits the frame 73 to the user device 74. In this stage, the frame 73 includes a person 71 and a house 72, and the user device 74 outputs only this frame 73 without rendering other objects.

Next, as shown in FIG. 4B, the server 70 begins to transmit the 3D model to the user device, starting from the 3D model closest to the 3D projection surface of the user device screen; in this embodiment, the person 71 is compared to the house 72. Closer to the 3D projection surface (or the user's eyes), therefore, the 3D model of the person 71 is first transmitted to the user device 74, and after the 3D model of the person 71 is transmitted and stored on the user device 74, the user device 74 transmits a The message to the server 70, the 3D model of the informant 71 is now pre-stored in the user device 74; after that, the server 70 renders the house 72, encodes the rendering result as a 2D video stream frame 73a, and transmits the frame 73a to the person. The interpreting data of 71a is sent to the user device 74, and the user device 74 automatically renders the person using the interpreting data, and then combines the rendering result of the person with the frame 73a (including the house) to obtain the same output result; the program (for example, the server The method of transmitting the 3D model to the user device 74 one at a time will be repeated again and again until all 3D models that the client needs to display have been transmitted and pre-stored in the user device 74.

As shown in FIG. 4C, a user device 74 has all 3D models (including 3D models of people and houses), the server does not need to perform rendering operations, and no need to transmit video stream frames (element 75). The server only needs to transmit the interpretation data of the 3D model (including the person 71a and the house 72a) to the user device 74. The user device will be able to render all 3D models on its own to achieve the same output.

Referring to Figures 6A, 6B and 6C, there is shown a schematic diagram of an embodiment of the method of the present invention for transmitting video streams and 3D sounds with sound.

As shown in FIG. 6A, when the initial user device 74 logs into the application 70 running on the server, no 3D sound is pre-stored in the user device, so the server renders all 3D sounds (including a sound 81 and Subsequent to one of the sounds 82), all 3D sounds should be presented on the speaker of the user device, and the server encodes the rendering result into a video stream frame 83 of sound, and then the video stream of the audio stream is streamed. 83 is transmitted to the user device 74. In this stage, the audio video stream frame 83 includes a sound 81 and a sound 82, and the user device 74 outputs only the video stream frame 83 of the sound without rendering (playing) other sounds.

Next, as shown in FIG. 6B, the server 70 begins to transmit the 3D sound to the user device, starting from the 3D sound closest to the 3D projection surface of the user device screen; in this embodiment, the sound 81 is compared with the sound 82. Closer to the 3D projection surface (or the user's eyes), therefore, the 3D sound of the sound 81 is first transmitted to the user device 74, and after the 3D sound of the sound 81 is transmitted and stored on the user device 74, the user device 74 transmits a The message is sent to the server 70 to inform that the sound 81 is now pre-existing in the user device 74; after that, the server 70 renders the sound 82, encodes the 2D video stream frame 83a whose rendering result is a sound, transmits the frame 83a, and the sound. The interpretation data of 81 is sent to the user device 74, and the user device 74 automatically renders (plays) the sound by using the interpreted data, and combines the rendering result of the sound with the frame 83a (including the sound) to obtain the same output result; for example, the program (for example The server transmits the 3D sound to the user device 74 one at a time.) It will be repeated again and again until all 3D sounds that need to be played on the speaker of the user device have been transmitted and pre-existed to the user. At the time of centering 74.

As shown in FIG. 6C, a user device 74 possesses all 3D sounds (including 3D sounds of sound 81 and sound 82), the server does not need to perform rendering operations, and thus the video stream frame (element 85) only The image is included, and the sound is not included; the server only needs to transmit the interpreted material of the 3D sound 81 (including the sound 81a and the sound 82a) to the user device 74. The user will then be able to render (play) all 3D sounds themselves to get the same output.

Referring to Figures 5A, 5B and 5C, there is shown a schematic diagram of how the method of the present invention determines which 3D model has to be encoded into a frame.

In the present invention, the server sorts all 3D models to be rendered in a predetermined order, which is a relative virtual position (such as the 3D projection surface 52 of the user device screen, or User's eyes 51), from near to far. As shown in FIG. 5A, the four objects A, B, C, and D need to be displayed on the screen of the user device, wherein the object A is closest to the projection surface 52, and then the object B, the object C, and the object D in turn, as the initial user device. When logging into the application running on the server, no 3D sound is pre-existing in the user device. Therefore, the server renders all objects A, B, C, and D, and encodes the rendering result into a video stream. The frame is then transmitted to the user device. At the same time, the server starts to transmit the 3D models of the object A, the object B, the object C and the object D in a predetermined order, that is, the 3D model of the object A is transmitted first, and then the object B, the object C and Object D until all 3D models displayed on the user device are transmitted.

As shown in FIG. 5B, after the 3D models of the objects A and B are pre-stored in the user device, the server will find the object when the server, according to the predetermined order from the near to the far, and the state of the 3D model is checked. C is the first object that is not pre-existing in the user device. Therefore, the server renders the object C and all other objects (such as object D) located after the object C, regardless of whether the 3D model of the object D is pre-existing in the user device. At this time, the server does not render the 3D models of the objects A and B, and the reason is that the objects A and B are both pre-existing in the user device and before the object C.

As shown in Figure 5C, when a new object E is displayed in the virtual 3D environment created by the application, the object E and all subsequent objects are rendered by the server, whether or not the object is pre-stored in the user device. For example, as shown in FIG. 5C, compared with the object B, the object C, and the object D, the new object E is relatively close to the 3D projection surface 52, although the 3D model of the object B is pre-existing in the user device, but due to the object B is located after the new object E, so the server will render all the objects E, C, B and D, even if the object B may only be partially covered by other objects in front of it.

Referring to Figures 7A, 7B and 7C, there is shown a schematic diagram of an embodiment of how the method of the present invention determines which 3D sound is to be encoded into a video stream with sound.

In the present invention, the server sorts all 3D sounds to be rendered in a predetermined order, which is: relative to a virtual position (such as the 3D projection surface 52 of the user device screen, or the user's eyes 51), from near To the far order. As shown in Fig. 7A, the four 3D sounds A, B, C, and D need to be played on the speaker of the user device, wherein the sound A is closest to the projection surface 52, and then the sound B, sound C, and sound D are in turn, when the initial user When the device logs into the application running on the server, no 3D sound is pre-existing in the user device. Therefore, the server renders all the sounds A, B, C and D, and encodes the rendering result into a sound. Video streaming video, then Transfer this frame to the user device. At the same time, the server starts to transmit the data of sound A, sound B, sound C and sound D in a predetermined order, that is, the 3D sound of sound A is transmitted first, followed by sound B, sound C and sound. D, until all 3D sounds are stored in the user device.

As shown in FIG. 7B, after the 3D sounds of the sounds A and B are pre-stored in the user device, the server will find the sound when the server, in the predetermined order from the near to the far, and the state of the 3D sound is checked. C is the first sound that is not pre-existing in the user device. Therefore, the server will render the sound C and all other sounds (such as sound D) located after the sound C, and the server will not 3D the sounds A and B. The sound is rendered, as in this phase, sounds A and B are pre-stored in the user device.

As shown in Figure 7C, when a new sound E is added to the virtual 3D environment created by the application, the sound E will be rendered by the server, but this rendering will not affect the rendering of other sounds. Different from the 3D model in FIG. 5C, as shown in FIG. 7C, compared with the sound B, the sound C, and the sound D, the new sound E is relatively close to the 3D projection surface 52, unlike the 3D in the 5C chart. Models, sounds pre-existing in the user device (such as sounds A and B) will still be rendered for the user device, but non-pre-existing user device sounds (such as sounds E, C, and D) are rendered by the server.

The foregoing technology of the present invention can also be applied to a Virtual Reality (VR) scenario system to transmit a 3D model and a VR video stream generated by a VR scenario application executed by a server to the network. User equipment, which will be described in detail below.

In order to provide a human VR visual experience, the virtual VR scene must contain an image for the left eye of the human eye and another image for the human right eye. Please refer to FIG. 8 , which is a schematic diagram of a first embodiment of a system architecture of a VR scenario system according to the present invention.

In the present invention, the scene server 1120 is a server computer software executed on a server 1 having a VR scene application 1100 (also referred to as a VR application or application) for generating a complex 3D model. A virtual VR 3D environment. The VR Scene Application 1100 also runs on Server 1, typically a VR game. The VR scenario server 1120 is a server program executed on the server 1 together with the application program 1100, and is used as the VR scene transmitter 1110 of the server 1 and the VR scene client of the user devices 21, 22, and 23. 1170, the relay station for message transmission. The VR scene server 1120 also serves as a file download server for the VR scene client 1170 of the user devices 21, 22, 23 to download the necessary 3D model from the server 1. The VR scene transmitter 1110 is a library, and the VR scene application 1100 is compiled at the same time. Static link, or dynamically linked to the VR scene application 1100 during execution. The VR scene client (program) 1170 is a program executed on the user devices 21, 22, 23 for generating and outputting a 3D image rendering result generated by the VR scene application 1100 in the user device. In this embodiment, each user device 21, 22, 23 corresponds to a separate VR scenario application 1100 and a VR scenario transmitter 1110. The VR Scene Transmitter 1110 maintains a list listing all 3D models and whether each 3D model has been stored in the state of the user device. This state is used to indicate that the status of each 3D model in the user device is (1) "Not One of Ready (not ready), (2) "Loading", and (3) "Ready for Client".

Server 1 will check the status of these 3D models to determine which 3D models need to be encoded in a left eye frame of a 2D video stream, and which 3D models need to be encoded in a right eye shadow of the 2D video stream. In the present invention, those 3D models that are not previously stored in the user devices 21, 22, 23 are encoded in the left eye frame and the right eye frame. In order to achieve this function, the main program of the VR scene application 1100 transmits the VR scene information to the VR scene transmitter 1110 by calling the library through the API (path 1101 of FIG. 8), and the VR scene information includes Name, location, speed, attributes, orientation, and all the information needed to render the 3D model. After the VR scene transmitter 1110 receives such data, the following procedure can be performed.

Step (a): For all 3D models, sort all the 3D models to be rendered in the left eye frame, and the ordering manner may be, relative to a virtual position (such as a 3D projection surface or a user's left eye), from Sorted from near to far.

Step (b): For the 3D model, look for the first 3D model "M" that does not have a "Ready for Client" state from the closest point (the one closest to the user's left eye), in other words, the first 3D model. The state of M" is the "Not Ready" state, (after which, the "Not Ready" state is simply referred to as the NR state); of course, there may be no such 3D models (for example, all 3D models to be displayed) It is marked as "Ready for Client" status).

Step (c): For the 3D model, the 3D model M and all subsequent 3D models are rendered by the server 1, that is, all 3D models that are farther away from the left eye than M. (If there is no 3D model M, it is displayed on a black screen), the result of the coded rendering is a left eye frame of a 2D video stream, which is provided to the user for left eye viewing.

Step (d): repeating the above steps (a) to (c) for the right eye frame, that is, changing the operation of the left eye described in the above steps (a) to (c) to the right eye, borrowing This produces another 2D Another frame of the video stream is the right eye frame, which is provided to the user for viewing by the right eye.

Step (e): The following three pieces of information are transmitted to the left eye frame to the VR scene server 1120 (path 1112): [Info 1112-A], [Info 1112-B], and [Info 1112-C], and, right The eye shadow grid transmits the following three messages to the VR scene server 1120 (path 1113): [Info 1113-A], [Info 1113-B], and [Info 1113-C].

Step (f): The data packer 121 in the VR scene server 1120 will display the information of the left and right eyes ([Info 1112-A], [Info 1112-B], [Info 1112-C], [Info 1113] -A], [Info 1113-B] and [Info 1113-C] are packaged into one information packet.

Step (g): The VR scenario server 1120 transmits the information packet generated in the step (f) to the VR scenario client 1170 (path 1122) in the user devices 21, 22, 23.

[Info 1112-A] is the status information (or interpretation data) of all 3D models before the 3D model M. Note that there may be no such models exist. Such model systems all have a "Ready for Client" state, meaning that these models are preloaded on the user device, and the VR scene client (program) 1170 on the user devices 21, 22, 23 can already render these models themselves. In order to reduce the data transmission bandwidth, the VR scene transmitter 1110 does not have to transmit all the state information, as long as the difference between the current rendering and the last rendering is transmitted in the state information.

[Info 1112-B] If the server finds the 3D model M and its pre-stored state in the user device is "Not Ready", the server will change its user status to "Loading" and send a 3D model M. The download instruction asks the user device to download the 3D model M; if the user status is already "Loading", then do not send any indication, because the download instruction has been sent.

[Info 1112-C] is the video stream frame of the left eye encoded in step (c), that is, the left eye frame.

[Info 1113-A], [Info 1113-B], and [Info 1113-C] are basically the same as [Info 1112-A], [Info 1112-B], and [Info 1112-C], respectively, except that [Info 1113-A], [Info 1113-B], and [Info 1113-C] are about the right eye frame.

Each time the main program of the VR scene application 1100 updates the new VR scene data to the VR scene transmitter 1110, steps (a) to (g) are repeated. Usually, the main program of the VR scene application 1100 will be used every time. This type of material is updated during the rendering cycle.

Once the VR scene client 1170 receives the aforementioned data, it performs the rendering described later. program.

Step (i): Decode the video frames (including both the left eye frame and the right eye frame) in [Info 1112-C and Info 1113-C] and transfer the two frames to the frame binder 1171.

Step (ii): The frame combiner 1171 combines the two frames (including both the left eye frame 1711 and the right eye frame 1712) into a combined VR frame 1713 (see Fig. 9) as a background for subsequent 3D model rendering. .

Step (iii): Render all 3D models in [Info 1112-A and Info 1113-A] on the merged VR frames after step (ii) encoding. In order to reduce the network bandwidth occupation, the VR scene client 1170 will store this [Info 1112-A and Info 1113-A] information into the memory, so the next VR scene transmitter 1110 can only transmit the next rendering and The difference in the status of [Info 1112-A and Info 1113-A] between the renderings does not require the transmission of all status information.

Step (iv): outputting the rendering result in step (iii) as a rendered hybrid VR frame in the output video stream containing the VR scene, that is, the final output video stream result (path 1176) . In this embodiment, the user device is a glasses or a helmet-shaped electronic device, and includes two display screens respectively located in front of the left eye and the right eye of the user; wherein the left screen is displayed for the user to the left The image that is viewed by the eye (frame), and the screen on the right is used to display the image (frame) for the user's right eye. The mixed VR frame in the output video stream is played on the two screens of the user device in such a manner that the left half of each pixel in each line of the mixed VR frame is displayed on the left eye screen. The right half of each pixel in each line of the hybrid VR frame is displayed on the right eye screen to provide a visual reality of the user's virtual reality (VR).

If the status of [Info 1112-B] and [Info 1113-B] is provided, it indicates that the 3D model M needs to be prepared by the VR scene client 1170. At this time, the VR scene client 1170 will process the 3D model M according to the following procedure. .

Step (I): Search VR Scene Cache 1190 (Path 1174), VR Scene Cache 1190 contains the 3D model database previously downloaded and stored in user devices 21, 22, 23.

Step (II): If there is already a 3D model M in the VR scene cache 1190, step (V) is directly executed.

Step (III): If there is no 3D model M in the VR scene cache 1190, the VR scene client 1170 will send a download request to the VR scene server 1120 (path 1172), and the VR scene server 1120 will return The data of the 3D model M is given to the VR scene client 1170 (path 1124).

Step (IV): After the 3D model is completely downloaded, the VR scene client 1170 is about to be stored in the VR scene cache 1190 (path 1194), so that when there is a similar demand next time, there is no need to download again.

Step (V): The VR scene client 1170 will extract the 3D model M (path 1192) from the VR scene cache 1190.

Step (VI): Once the download is completed (or has been downloaded earlier), the VR scene client 1170 can extract the 3D model M; the VR scene client 1170 will send a "3D model is ready on client" (the 3D model is already in the user) The message on the device is sent to the VR Scene Server 1120 (path 1115), and the VR Scene Server 1120 will forward the message to the VR Scene Transmitter 1110 (path 1114).

Step (VII): After receiving the message, the VR scene transmitter 1110 changes the state of the 3D model M from "Loading" to "Ready for Client".

Step (VIII): In the next rendering, the VR scene transmitter 1110 will know that the 3D model M is now preloaded in the user device, so the VR scene client 1170 will be rendered by itself, therefore, the server 1 will not It is then necessary to render this 3D model M.

At the very beginning, there are no 3D models in the user devices 21, 22, 23, so the VR scene transmitter 1110 will render all 3D models and encode the results into 2D video including the left eye frame and the right eye frame. Streaming. The VR Scene Transmitter 1110 will process the download requirements [Info 1112-B] and [Info 1113-B] of the 3D model from the closest 3D projection surface (or the user's left or right eye). The VR scene client 1170 will download each 3D model from the VR scene server 1120 or extract it from the VR scene cache 1190 one by one. When more 3D models are available to the VR scene client 1170, the VR scene transmitter 1110 will automatically notify the VR scene client 1170 to render the models and sounds themselves, and reduce the 3D model rendered by the VR scene transmitter 1110. Quantity. Thereby, the 3D model in the encoded 2D video stream with the left eye frame and the right eye frame will be less and less until the last VR scene client 1170 can obtain all the 3D models; and then, In this stage, only the black screen is encoded by the server 1, in other words, the server 1 does not need to transmit the 2D video stream to the user devices 21, 22, 23, and the server 1 and the user devices 21, 22 The communication bandwidth of 23, can also be greatly reduced.

When a new 3D model N appears in the VR scene, the VR scene transmitter 1110 will (1) notify the VR scene client 1170 that only all of the new 3D models N are rendered. 3D model (relative to the user's left or right eye), (2) notifying the VR scene client 1170 to download the new 3D model N, and (3) the VR scene transmitter 1110 will render the new 3D model N and all the models that follow it, and encode the result as a 2D video stream containing the left eye frame and the right eye frame. Then, the 2D video stream including the left eye frame and the right eye frame is transmitted to the VR scene client 1170. Therefore, the VR scene client 1170 can still reproduce the 3D image rendering result of the VR scene application 1100 before the 3D model N is prepared on the user device.

FIG. 10 is a schematic diagram of a second embodiment of a system architecture of a VR scene system of the present invention. In the second embodiment shown in FIG. 10, most of the elements and functions are substantially the same or similar to the first embodiment disclosed in FIG. 8, except that the frame combiner 1111 is located in the VR scene transmitter 1110. The same or similar elements in Fig. 10 will be given the same reference numerals as in Fig. 8, and the details thereof will not be described.

As shown in FIG. 10, the main program of the VR scene application 1100 transmits the VR scene information to the VR scene transmitter 1110 by calling the library through the API. The VR scene information includes name, location, speed, Properties, orientation, and all other 3D model rendering materials. After the VR scene transmitter 1110 receives such data, the following procedure can be performed.

Step (a): For all 3D models, sort all the 3D models to be rendered in the left eye frame, and the ordering manner may be, relative to a virtual position (such as a 3D projection surface or a user's left eye), from Sorted from near to far.

Step (b): For the 3D model, look for the first 3D model "M" that does not have a "Ready for Client" state from the closest point (the one closest to the user's left eye), in other words, the first 3D model. The state of M" is the "Not Ready" state, (after which, the "Not Ready" state is simply referred to as the NR state); of course, there may be no such 3D model.

Step (c): rendering the 3D model "M" and all subsequent 3D models in the server 1 (if the 3D model "M" is not present, a black screen is directly generated) and then stored in the memory In the body.

Step (d): repeating the above steps (a) to (c) for the right eye frame, that is, changing the operation of the left eye described in the above steps (a) to (c) to the right eye, borrowing This produces a right eye frame that is provided to the user for viewing in the right eye.

Step (e): The rendered left eye frame and the right eye frame are combined by the frame combiner 1111 into a combined VR frame in a 2D video stream.

Step (e): The following three pieces of information are transmitted to the VR scene server 1120 (path 1112) for the left eye frame and the right eye frame: [Info 1112-A], [Info 1112-B], and [Info 1112-C], The VR scene server 1120 then transmits it to the VR scene client 1170 (path 1122) in the user devices 21, 22, 23.

[Info 1112-A] is the status information (or interpretation data) of all 3D models before the 3D model M. Note that there may be no such models exist. Such model systems all have a "Ready for Client" state, meaning that these models are preloaded on the user device, and the VR scene client (program) 1170 on the user devices 21, 22, 23 can already render these models themselves. In order to reduce the data transmission bandwidth, the VR scene transmitter 1110 does not have to transmit all the state information, as long as the difference between the current rendering and the last rendering is transmitted in the state information.

[Info 1112-B] If the server finds the 3D model M and its pre-stored state in the user device is "Not Ready", the server will change its user status to "Loading" and send a 3D model M. The download instruction asks the user device to download the 3D model M; if the user status is already "Loading", then do not send any indication, because the download instruction has been sent.

[Info 1112-C] is the merged VR frame in the video stream frame that has been rendered in step (e) and contains the left eye frame and the right eye frame.

Once the VR scene client 1170 receives the aforementioned data, the rendering procedure described later is performed.

Step (i): Decode the merged VR frame in [Info 1112-C] and use it as the background for rendering the subsequent 3D model.

Step (ii): Render all 3D models in [Info 1112-A] on the merged VR frame. In order to reduce the network bandwidth occupation, the VR scene client 1170 will store the [Info 1112-A] information into the memory, so the next VR scene transmitter 1110 can only transmit the next rendering and the current rendering. [Info 1112-A] Difference in status, no need to transfer all status information

Step (iii): outputting the rendering result in step (ii) as a rendered mixed VR frame in the output video stream containing the VR scene, that is, the final output video stream result (path 1176) .

11 is a schematic diagram of a third embodiment of a system architecture of a VR scenario system of the present invention. In the third embodiment shown in FIG. 11, most of the elements and functions are substantially the same or similar to the first embodiment disclosed in FIG. 8, except that the third embodiment no longer has a combination of frames. Therefore, the same or similar elements in Fig. 11 will be given the same reference numerals as in Fig. 8, and the details thereof will not be described.

As shown in FIG. 11, the VR scene server 1120 is a server computer software executed on a server 1 having a VR scene application 1100 for generating a virtual VR 3D environment including a plurality of 3D models. The VR scenario server 1120 is a server program executed on the server 1 together with the application program 1100, and is used as the VR scene transmitter 1110 of the server 1 and the VR scene client of the user devices 21, 22, and 23. 1170, the relay station for message transmission. The VR scene server 1120 also serves as a file download server for the VR scene client 1170 of the user devices 21, 22, 23 to download the necessary 3D model from the server 1. The VR Scene Transmitter 1110 maintains a list listing all 3D models and whether each 3D model has been stored in the state of the user device. This state is used to indicate that the status of each 3D model in the user device is (1) "Not One of Ready (not ready), (2) "Loading", and (3) "Ready for Client".

Server 1 will check the status of these 3D models to determine which 3D models need to be encoded in a left eye frame of a 2D video stream, and which 3D models need to be encoded in a right eye shadow of the 2D video stream. In the present invention, those 3D models that are not previously stored in the user devices 21, 22, 23 are encoded in the left eye frame and the right eye frame. In order to achieve this function, the main program of the VR scene application 1100 transmits the VR scene information to the VR scene transmitter 1110 by calling the library through the API (the path 1101 of FIG. 11), and the VR scene information includes Name, location, speed, attributes, orientation, and all the information needed to render the 3D model. After the VR scene transmitter 1110 receives such data, the following procedure can be performed.

Step (a): For all 3D models, sort all the 3D models to be rendered in the left eye frame, and the ordering manner may be, relative to a virtual position (such as a 3D projection surface or a user's left eye), from Sorted from near to far.

Step (b): For the 3D model, look for the first 3D model "M" that does not have a "Ready for Client" state from the closest point (the one closest to the user's left eye), in other words, the first 3D model. The state of M" is the "Not Ready" state, (after which, the "Not Ready" state is simply referred to as the NR state); of course, there may be no such 3D models (for example, all 3D models to be displayed) It is marked as "Ready for Client" status).

Step (c): For the 3D model, the 3D model M is rendered by the server 1 and thereafter All 3D models, that is, all 3D models that are farther away from the left eye than M. (If there is no 3D model M, it is displayed on a black screen), the result of the coded rendering is a left eye frame of a 2D video stream, which is provided to the user for left eye viewing.

Step (d): repeating the above steps (a) to (c) for the right eye frame, that is, changing the operation of the left eye described in the above steps (a) to (c) to the right eye, borrowing Another frame that produces another 2D video stream, the right eye frame, is provided to the user for viewing by the right eye.

Step (e): The following three pieces of information are transmitted to the left eye frame to the VR scene server 1120 (path 1112): [Info 1112-A], [Info 1112-B], and [Info 1112-C], and, right The eye shadow grid transmits the following three messages to the VR scene server 1120 (path 1113): [Info 1113-A], [Info 1113-B], and [Info 1113-C].

Step (f): The data packer 121 in the VR scene server 1120 will display the information of the left and right eyes ([Info 1112-A], [Info 1112-B], [Info 1112-C], [Info 1113] -A], [Info 1113-B] and [Info 1113-C] are packaged into one information packet.

Step (g): The VR scenario server 1120 transmits the information packet generated in the step (f) to the VR scenario client 1170 (path 1122) in the user devices 21, 22, 23.

[Info 1112-A] is the status information (or interpretation data) of all 3D models before the 3D model M. Note that there may be no such models exist. Such model systems all have a "Ready for Client" state, meaning that these models are preloaded on the user device, and the VR scene client (program) 1170 on the user devices 21, 22, 23 can already render these models themselves. In order to reduce the data transmission bandwidth, the VR scene transmitter 1110 does not have to transmit all the state information, as long as the difference between the current rendering and the last rendering is transmitted in the state information.

[Info 1112-B] If the server finds the 3D model M and its pre-stored state in the user device is "Not Ready", the server will change its user status to "Loading" and send a 3D model M. The download instruction asks the user device to download the 3D model M; if the user status is already "Loading", then do not send any indication, because the download instruction has been sent.

[Info 1112-C] is the video stream frame of the left eye encoded in step (c), that is, the left eye frame.

[Info 1113-A], [Info 1113-B], and [Info 1113-C] are basically the same as [Info 1112-A], [Info 1112-B], and [Info 1112-C], respectively, except that [Info 1113-A], [Info 1113-B], and [Info 1113-C] are related to the right eye frame.

Each time the main program of the VR scene application 1100 updates the new VR scene data to the VR scene transmitter 1110, steps (a) to (g) are repeated. Usually, the main program of the VR scene application 1100 will be used every time. This type of material is updated during the rendering cycle.

Once the VR scene client 1170 receives the aforementioned data, the rendering procedure described later is performed.

Step (i): Decode the video frames (including both the left eye frame and the right eye frame) in [Info 1112-C and Info 1113-C] and store the two frames in different memory spaces.

Step (ii): Render all 3D models included in [Info 1112-A and Info 1113-A] on the decoded left eye frame and right eye frame respectively (if this 3D model exists). In order to reduce the network bandwidth occupation, the VR scene client 1170 will store this [Info 1112-A and Info 1113-A] information into the memory, so the next VR scene transmitter 1110 can only transmit the next rendering and The difference in the status of [Info 1112-A and Info 1113-A] between the renderings does not require the transmission of all status information.

Step (iii): outputting the rendering result in step (ii) as a rendered left-eye frame and a mixed right-eye frame in the output video stream including the VR scene, that is, the final output video Streaming results (path 1176). Wherein, the mixed left eye frame and the mixed right eye frame can be combined to be referred to as a hybrid VR frame as mentioned before.

In this embodiment, the user device is a glasses or a helmet-shaped electronic device, and includes two display screens respectively located in front of the left eye and the right eye of the user; wherein the left screen is displayed for the user to the left The image that is viewed by the eye (frame), and the screen on the right is used to display the image (frame) for the user's right eye. The mixed VR frame in the output video stream is played on the two screens of the user device in such a manner that each of the mixed left VR frames in the mixed VR frame is displayed on the left eye screen, and the Each of the mixed right eye frames in the hybrid VR frame is displayed on the right eye screen to provide a visual experience of the user's virtual reality (VR).

In another embodiment, the video stream outputted on one of the user devices is displayed on the same screen in turn to display the mixed left eye frame and the mixed right eye frame. The user can wear a glasses-shaped electronic device, which can sequentially open and close the left-eye window and the right-eye window in turn according to the mixed left-eye frame and the mixed right-eye frame displayed on the screen to provide The visual perception of the user's virtual reality (VR).

The above-mentioned embodiments are not intended to limit the scope of application of the present invention, and the scope of the present invention should be based on the technical spirit defined by the content of the patent application scope of the present invention and the scope thereof. It is to be understood that the scope of the present invention is not limited by the spirit and scope of the present invention, and should be considered as a further embodiment of the present invention.

1‧‧‧Server

21, 22, 23‧‧‧ User equipment

1100‧‧‧Application

1110‧‧‧Scenario Transmitter

1120‧‧‧Scenario Server

121‧‧‧data packer

1170‧‧‧ Scene Client

1171‧‧1 frame combiner

1190‧‧‧ Scene cache

Paths of 1101, 1112, 1113, 1114, 1122, 1172, 1124, 1115, 1174, 1192, 1194, 1176‧‧

Claims (15)

A method for transmitting media over a network, the media comprising a plurality of images, comprising: step (A): executing a virtual reality (VR) application on a server to generate a virtual image comprising a plurality of 3D models a VR 3D environment, each of the 3D models being associated with a state indicating whether the 3D model is pre-existing in a user device; and step (B): the server checks the states of the 3D models to determine which one The 3D model is to be encoded as a left eye frame and a right eye frame included in a 2D video stream, and is encoded by encoding the 3D models that are not pre-existing in the user device to the left eye frame and the In the right eye frame; step (C): the server transmits at least the left eye frame and the right eye frame of the 2D video stream to the user device through the network; wherein the server also not The 3D models in the user device are transmitted to the user device in a predetermined order; and when the user device receives the 3D models transmitted by the server, the user device stores the 3D models And send a message to the a server to change the states of the 3D models and indicating that the 3D models are now pre-existing in the user device; and step (D): the user device receives the left eye shadow received from the server And the right eye frame is decoded, and the left eye frame and the right eye frame are used as a background for rendering the 3D model of the pre-existing user device but not included in the left eye frame and the right eye frame. A picture to thereby generate a hybrid VR frame as an output video stream containing one of the VR scenes. The method for transmitting media over a network as described in claim 1, wherein in the step (B), the states of the 3D models are obtained by the server in a position closest to a virtual location. The order of the other point farthest from the virtual position is checked; and, in the verification, when the first 3D model in the user device is not pre-stored, whether or not the 3D models located behind it are pre-existing In the user device, all remaining 3D models including the discovered 3D model are encoded into the left eye frame and the right eye frame. A method for transmitting media over a network as described in claim 2, wherein when a new 3D model appears in the VR 3D environment, whether or not the 3D models located thereafter are pre-existing in the user device , will include all subsequent 3D of the new 3D model The model is encoded into the left eye frame and the right eye frame. The method for transmitting media over a network as described in claim 2, wherein the virtual location is a 3D projection plane; and, in the step (D), the user device receives the server from the server After the left eye frame and the right eye frame are decoded, the user device further merges the left eye frame and the right eye frame into a combined VR frame, and then uses the combined VR frame as the background picture to render the The 3D models pre-stored in the user device but not included in the left eye frame and the right eye frame are thereby generated as the hybrid VR frame containing the output video stream of the VR scene. The method for transmitting media over a network as described in claim 1, wherein: in the step (C), the server is configured to transmit the 3D models not pre-existing in the user device to the The predetermined sequence of the user device is in the order of being closest to one of the virtual locations to another point farthest from the virtual location; in the step (C), the server will not be encoded to the left eye frame and The status information of the 3D model in the right eye frame is transmitted to the user device. When receiving and verifying the status information, the user device performs the following manner: if any 3D model system in the status information is received If the device is not pre-existing, the user device sends a request to the server to download the 3D model; wherein the status information includes each left eye shadow that is not encoded into the 2D video stream. And the at least one interpretation data of the 3D model in the right eye frame, the interpretation data of each of the 3D models includes a name, a position, a speed, a direction, and an attribute of the 3D model. A system for transmitting media over a network includes a server for executing a virtual reality (VR) application to generate a virtual VR 3D environment including a plurality of 3D models, each of the 3D models Corresponding to a state indicating whether the 3D model is pre-existing in a user device; and the user device is connected to the server through a network for obtaining at least some of the 3D models generated by the VR application The media includes a plurality of images, and the plurality of images are transmitted by: step (B): the server checks the states of the 3D models to determine which 3D model needs to be encoded. a left eye frame and a right eye shadow for a 2D video stream a coding method for encoding the non-pre-existing of the 3D models in the user device into the left-eye frame and the right-eye frame (C): the server at least streams the 2D video stream to the left The eye shadow grid and the right eye frame are transmitted to the user device through the network; wherein the server also transmits the 3D models in the non-pre-existing user device to the user device in a predetermined order; When the user device receives the 3D models sent by the server, the user device stores the 3D models and sends a message to the server to change the states of the 3D models. And indicating that the 3D models are now pre-existing in the user device; step (D): the user device decodes the left eye frame and the right eye frame received from the server, and the left eye frame And combining the right eye frame into a merged VR frame, and then using the merged VR frame as a background image for rendering the pre-existing user devices but not included in the merged VR frame, thereby generating a One of the VR scenes A mixed stream VR Movies frame; and a step (E): the user equipment comprises the mixing the output VR of the output image frame of the video stream of the scene VR. A system for transmitting media over a network as described in claim 6 wherein, in the step (B), the states of the 3D models are selected by the server to be closest to a virtual location. In the order of the other point farthest from the virtual location, it is checked that, in the verification, when the first 3D model in the user device is not pre-stored, whether the 3D model located behind it pre-stores the user In the device, all remaining 3D models including the discovered 3D model are encoded into the left eye frame and the right eye frame. A system for transmitting media over a network as described in claim 7, wherein when a new 3D model appears in the VR 3D environment, whether the 3D model is located behind the user device is pre-existing All of the remaining 3D models including the new 3D model are encoded into the left eye frame and the right eye frame. A system for transmitting media over a network as described in claim 6 wherein: in the step (C), the server is configured to transmit the 3D models not pre-existing in the user device to the user The predetermined sequence of devices is one closest to the virtual The order of the 3D model of the position to the other 3D model farthest from the virtual position; in the step (C), the server will also not be encoded into the left eye frame and the 3D in the right eye frame One state information of the model is transmitted to the user device; when receiving and verifying the status information, the user device performs the following manner: if any of the 3D models received in the status information are non-pre-existing in the device The user device sends a request to the server to download the 3D model; wherein the status information includes each of the 3D models that are not encoded into the left eye frame and the right eye frame. The interpretation data, the interpretation data of each of the 3D models includes a name, a position, a speed, a direction, and an attribute of the 3D model. The system for transmitting media over the network as described in claim 6, wherein the server further comprises: a VR scene transmitter, which is compiled in the VR application or dynamically linked to the VR during execution time. a library on the application; wherein the VR scene transmitter maintains a list of all of the states of the 3D model and each of the 3D models, the state is used to indicate that the state of the 3D model is "Not Ready" One of (not prepared), "Loading" and "Ready for Client"; and a VR scene server for executing the VR application on the server a server program; wherein the VR scene server serves as a relay station for the VR scene transmitter and the message transmission between the user devices, and the VR scene server also serves as the 3D necessary for the user device to download from the server. One of the models downloads the server program. The system for transmitting media over the network according to claim 10, wherein the user device further comprises: a VR scene client, which is a program running on the user device, for generating the output video string. Streaming and communicating with the server through the network; a frame combiner for combining the left eye frame and the right eye frame into the combined VR frame; A VR scene cache is used to store at least one 3D model previously downloaded from the server. A method of transmitting media over a network, the media comprising a plurality of images, the method comprising: step (A): executing a virtual reality (VR) application on a server to generate a plurality of 3D models a virtual VR 3D environment, each of the 3D models being associated with a state indicating whether the 3D model is pre-existing in a user device; and (B): the server checks the states of the 3D models to Determining which one of the 3D model sounds to be encoded as a 2D video stream and a right eye frame, the encoding method is to encode the 3D models that are not pre-existing in the user device to the 2D video Streaming the left eye frame with the right eye frame; then, the server merges the left eye frame and the right eye frame into a combined VR frame of the 2D video stream; step (C): the servo Transmitting at least the merged VR frame of the 2D video stream to the user device through a network; wherein the server also uses the 3D models in the non-pre-existing user device in a predetermined order, Transmitted to the user device; and when the user When receiving the 3D models transmitted by the server, the user device stores the 3D models and sends a message to the server to change the states of the 3D models and indicate the Some 3D models are now pre-existing in the user device; and step (D): the user device decodes the merged VR frame received from the 2D video stream of the server and utilizes the merged VR The frame is used as a background image for rendering the 3D model of the pre-existing user device but not included in the merged VR frame, thereby generating a hybrid VR frame containing one of the VR scenes and outputting the video stream. . The method for transmitting media over a network according to claim 12, wherein: in the step (B), the state of the 3D models is selected by the server to be closest to a virtual location. In the order of another point away from the virtual location, it is checked whether, in the verification, the first 3D model in the user device is not pre-existing, whether the 3D models located behind it are pre-existing in the user device All put the bag All remaining 3D models of the discovered 3D model are encoded into the left eye frame of the 2D video stream and the right eye frame; in the step (C), the server also stores the user in the left eye frame. a 3D model in the device, transmitted to the user device in a predetermined order from one of the closest to the virtual location to another point farthest from the virtual location; and when the user device receives the transmission from the server In the 3D models, the user device stores the 3D models and sends a message to the server to change the states of the 3D models, and indicates that the 3D models are now pre-existing the user. In the device. A method for transmitting media over a network as described in claim 13 wherein when a new 3D model appears in the VR 3D environment, whether or not the 3D models located thereafter are pre-existing in the user device All subsequent 3D models including the new 3D model are encoded into the left eye frame and the right eye frame; wherein the virtual position is a 3D projection surface. A method for transmitting media over a network as described in claim 12, wherein in the step (C), the server is also not encoded into the left eye frame and the 3D in the right eye frame One of the model status information is transmitted to the user device, and the user device receives and checks the status information in the following manner: if any of the 3D models in the status information are received, the device is non-pre-existing in the device. The user device sends a request to the server to download the 3D model; wherein the status information includes each left eye frame and the right eye frame that are not encoded into the 2D video stream. An interpreting material of the 3D model, the interpreting data including a name, a position, a speed, a direction, and an attribute of the 3D model.