WO2023273675A1

WO2023273675A1 - Method for processing video scene of free visual angle, and client and server

Info

Publication number: WO2023273675A1
Application number: PCT/CN2022/093592
Authority: WO
Inventors: 江平; 赵俊哲; 高元仲
Original assignee: 中兴通讯股份有限公司
Priority date: 2021-06-28
Filing date: 2022-05-18
Publication date: 2023-01-05
Also published as: CN115604523A

Abstract

A method for processing a video scene of a free visual angle, and a client and a server. The method, which is applied to a client, comprises: a client acquiring an index file (101); parsing video frame information and camera position values of all camera positions according to the index file (102); and acquiring a camera position value of a switched visual angle, and downloading a video frame according to the video frame information and the camera position value of the switched visual angle (103).

Description

Processing method, client and server of free-view video scene

Cross References to Related Applications

This application is based on a Chinese patent application with application number 202110722259.7 and a filing date of June 28, 2021, and claims the priority of this Chinese patent application. The entire content of this Chinese patent application is hereby incorporated by reference into this application.

technical field

The embodiments of the present application relate to the field of multimedia technologies, and in particular, relate to a processing method, a client, and a server for a free-view video scene.

Background technique

With the advent of the 5G era and the growth of user entertainment needs, single-view video experience can no longer meet user experience needs, while multi-view video can only provide fewer exciting perspectives, and user interaction options are limited. The viewing angle can be arbitrarily selected through the free viewing angle of the 360-degree full viewing angle, which can give users a customized experience. Now the free viewing angle is widely used in sports events, education and training, and entertainment performances, providing new video scenarios for 5G applications.

In some cases, when users experience free viewing angles, they have high requirements for the delay of viewing angle interaction and the smoothness of screen switching. Now the mainstream free viewing angles in the industry use splicing or real-time synthesis, and the splicing viewing angle scheme is transmitted. The bandwidth usage is high, and the image quality of the original video frame is lost, while the viewing angle effect of real-time synthesis cannot be guaranteed, and it consumes a lot of performance.

Contents of the invention

The following is an overview of the topics described in detail in this article. This summary is not intended to limit the scope of the claims.

Embodiments of the present application provide a method for processing a free-view video scene, a client, and a server.

In the first aspect, the embodiment of the present application provides a method for processing a free-view video scene, which is applied to a client, and the method includes: obtaining an index file, the index file includes video frame information; analyzing according to the index file Obtain the video frame information and camera position values of all camera positions; obtain the camera position values for switching viewing angles, and download video frames according to the video frame information and the camera position values for switching viewing angles.

In the second aspect, the embodiment of the present application provides a method for processing a free-view video scene, which is applied to a server. The method includes: performing slice packaging on a media file containing multiple code streams to obtain slices, and the slices Including video frame information; generating an index file corresponding to the slice; extracting the video frame information in the slice to the index file.

In a third aspect, an embodiment of the present application provides a client, including: a memory, a processor, and a computer program stored on the memory and operable on the processor, and the processor implements the above-mentioned first method when executing the computer program. In one aspect, the method for processing a free viewing angle video scene.

In a fourth aspect, an embodiment of the present application provides a server, including: a memory, a processor, and a computer program stored on the memory and operable on the processor, and the processor implements the second above when executing the computer program. The processing method of the free viewing angle video scene described in the aspect.

In the fifth aspect, the embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium stores a computer-executable program, and the computer-executable program is used to make the computer perform the above-mentioned first aspect. A processing method for a free-view video scene, or a processing method for a free-view video scene as described in the second aspect above.

Additional features and advantages of the application will be set forth in the description which follows, and, in part, will be obvious from the description, or may be learned by practice of the application. The objectives and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

Description of drawings

The accompanying drawings are used to provide a further understanding of the technical solution of the present application, and constitute a part of the specification, and are used together with the embodiments of the present application to explain the technical solution of the present application, and do not constitute a limitation to the technical solution of the present application.

Fig. 1 is the main flow chart (client side) of the processing method of a kind of free viewing angle video scene provided by one embodiment of the present application;

FIG. 2 is a sub-flow chart of a method for processing a free-view video scene provided by an embodiment of the present application;

FIG. 3 is a subflow chart of a method for processing a free-view video scene provided by an embodiment of the present application;

FIG. 4 is a camera position diagram of a free viewing angle provided by an embodiment of the present application;

FIG. 5 is a switching frame diagram of a free viewing angle provided by an embodiment of the present application;

Fig. 6 is a bullet time switching frame diagram of a free perspective provided by an embodiment of the present application;

FIG. 7 is a main flow chart (server side) of a method for processing a free-view video scene provided by an embodiment of the present application;

Fig. 8 is a flow chart of a live broadcast of a free viewing angle provided by an embodiment of the present application;

Fig. 9 is an on-demand flow chart of a free viewing angle provided by an embodiment of the present application;

FIG. 10 is a schematic structural diagram of a client provided by an embodiment of the present application;

Fig. 11 is a schematic diagram of a server structure provided by an embodiment of the present application.

detailed description

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, not to limit the present application.

It should be understood that in the description of the embodiments of the present application, multiple (or multiple) means more than two, greater than, less than, exceeding, etc. are interpreted as not including the original number, and above, below, within, etc. are understood as including the original number. If there are descriptions such as "first", "second", etc., it is only for the purpose of distinguishing technical features, and it cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or implicitly indicating the indicated The sequence relationship of the technical characteristics.

In some cases, when users experience free viewing angles, they have high requirements for the delay of viewing angle interaction and the smoothness of screen switching. Now the mainstream free viewing angles in the industry use splicing or real-time synthesis, and the splicing viewing angle scheme is transmitted. The bandwidth usage is high, and the image quality of the original video frame is lost, and the viewing angle effect of real-time synthesis cannot be guaranteed, and it consumes performance

The embodiment of the present application provides a processing method, a client and a server for a free viewing angle video scene. The client obtains an index file, analyzes the camera position values of the video frame information of all camera positions according to the index file, and obtains the position value of the switching angle of view. Camera position value, download the video frame according to the video frame information and the camera position value for switching the viewing angle. Based on this, it can not only realize the low latency of viewing angle interaction, but also ensure the smoothness of screen switching. By introducing the auxiliary file of index file, it can reduce unnecessary downloads while ensuring the quality of the screen, and at the same time, it is easy to expand other files later. Angle information.

As shown in FIG. 1 , FIG. 1 is a flowchart of a method for processing a free-view video scene provided by an embodiment of the present application. The method for processing a free-view video scene can be applied to a client, and the method for processing a free-view video scene includes, but is not limited to, the

following steps

101 , 102 and 103 .

Step 101, acquire an index file.

Step 102, analyze the video frame information and camera position values of all camera positions according to the index file.

Step 103, acquire the camera position value for switching the viewing angle, and download the video frame according to the video frame information and the camera position value for switching the viewing angle.

It is understandable that the client obtains the index file, which includes video frame information and camera position values of all camera positions, and parses out the video frame information and camera position values of all camera positions according to the index file, and obtains the switch The camera position value of the angle of view, download the video frame according to the video frame information and the camera position value of the switching angle of view, so as to realize the switching of the free angle of view. Based on this, it can not only realize the low latency of viewing angle interaction, but also ensure the smoothness of screen switching. By introducing the auxiliary file of index file, it can reduce unnecessary downloads while ensuring the quality of the screen, and at the same time, it is easy to expand other files later. Angle information.

It should be noted that the video frame information includes, but is not limited to, video frame starting position information, video frame size, and camera position value corresponding to the video frame.

Taking live broadcast as an example, the client can download the index file from the server and parse out the video frame information, and download the next frame of the current camera frame by frame according to the video frame information in the index file, and decode and render the downloaded frame , when the user performs a viewing angle switching operation, modify the current camera position value through the client, and download the next frame of the switched camera position by frame, and decode and render the downloaded frame, and so on until the viewing angle switching operation ends. Based on this, if the user switches the viewing angle, the client responds to the interaction, modifies the camera location value information, and then downloads frames according to the modified camera location stream. Since the download is performed in units of frames, the playback can achieve low-latency interaction of viewing angles, and switching camera positions does not affect picture rendering or cause camera position jumps, which can ensure the smoothness of screen switching.

It should be noted that after step 103, it may further include but not limited to the following sub-steps.

Decode and render video frames.

It can be understood that, for the downloaded video frame, the client needs to be decoded and rendered before it can be opened, so that the user can see the picture of the video frame on the client.

As shown in FIG. 2 , step 101 may include but not limited to the following sub-steps: step 1011 , step 1012 and step 1013 .

Step 1011, acquire the media presentation description file, the media presentation description file is generated by the server.

Step 1012, obtain the information of the index file according to the media presentation description file.

Step 1013, download the index file from the server according to the information of the index file.

It can be understood that, for the client to obtain the index file, for example, by obtaining the media presentation description (Media Presentation Description, MPD) file generated by the server based on DASH (Dynamic Adaptive Streaming over HTTP, adaptive streaming media transmission), The information of the index file is obtained according to the media presentation description file, and then the index file is downloaded from the server according to the information of the index file.

As shown in FIG. 3 , step 103 may include but not limited to the following sub-steps: step 1031 to step 1034 .

Step 1031, obtain the current camera position value.

Step 1032, determine the current viewing angle video frame to be downloaded according to the video frame information and the current camera position value.

Step 1033, acquire the position value of the target camera.

Step 1034, determine the target viewing angle video frame to be downloaded according to the video frame information and the position value of the target camera.

It can be understood that, as shown in FIG. 4 , in order to achieve a free viewing angle, multiple cameras are required to surround the object to be photographed in 360°, and the corresponding video frames are found and downloaded by switching camera positions. Specifically, each camera position corresponds to a camera position value, and the user realizes viewing angle switching by modifying the camera position value on the client, for example, by obtaining the current camera position value, and determining the current camera position value according to the current camera position value. camera position, and determine the current viewing angle video frame to be downloaded according to the video frame information and the current camera position; for another example, when the user performs a viewing angle switching operation, the current camera position value can be modified to the target camera position value, and the client obtains the The target camera position value input by the user, determine the target camera position according to the target camera position value, so as to realize switching from the current camera position to the target camera position, and determine the target to be downloaded according to the video frame information and the target camera position Angle of view video frame, so as to realize the switching of free angle of view.

It can be understood that, as shown in Figure 5, taking the live broadcast as an example, according to the video frame information in the index file, the next frame of the current camera position is downloaded frame by frame. From the camera position value to the target camera position value, download and switch the next frame of the target camera position by frame, and repeat until the viewing angle switching operation ends; as shown in Figure 5, taking VOD as an example, according to the video in the index file Frame information, download the next frame of the current camera position by frame, when the user performs an angle switching operation, modify the current camera position value to the target camera position value through the client, and download the next frame of the switching target camera position by frame Frame, repeat until the end of the viewing angle switching operation; as shown in Figure 6, taking bullet time as an example, according to the video frame information in the index file, download the next frame of the current camera position by frame, when the user performs the bullet time operation , modify the current camera position value through the client, increment by 1 to the target camera position value, and download the same frame for switching the target camera position by frame, and repeat this until the viewing angle switching operation ends.

As shown in FIG. 7 , FIG. 7 is a flowchart of a method for processing a free-view video scene provided by an embodiment of the present application. The method for processing a free-view video scene may be applied to a server, and the method for processing a free-view video scene includes, but is not limited to, the following

steps

201 , 202 and 203 .

Step 201: Slicing and encapsulating a media file containing multiple code streams to obtain fragments, which include video frame information.

Step 202, generating an index file corresponding to the slice.

Step 203, extract the video frame information in the slice to the index file.

It can be understood that the server slices and encapsulates the media files containing multiple code streams to obtain fragments. The fragments include video frame information, and generate index files corresponding to the fragments, and extract the video frame information in the fragments to Index file, so that the client can download the index file from the server. Based on this, it can not only realize the low latency of viewing angle interaction, but also ensure the smoothness of screen switching. By introducing the auxiliary file of index file, it can reduce unnecessary downloads while ensuring the quality of the screen, and at the same time, it is easy to expand other files later. Angle information.

It can be understood that the server may generate a media presentation description file based on the DASH protocol, where the media presentation description file includes information about fragments and information about index files. Based on this, the client can acquire the media presentation description file generated by the server, obtain the information of the index file according to the media presentation description file, and then download the index file from the server according to the information of the index file.

Transcode and merge multi-camera video streams into a unified stream for live broadcast or recording for on-demand. The server obtains the live stream or on-demand stream, performs slice encapsulation based on the DASH protocol, and extracts the frame information in the slice to the index file. Describe the segment information and index file information to the media presentation description file. The client obtains the media presentation description file, downloads the index file according to the custom fields of the index file, and parses the video frame information in the index file. The client downloads frames in units of frames according to the video frame information in the index file, and decodes and renders the downloaded frames. If the user switches the viewing angle, the client responds to the interaction, modifies the camera position value information, and then downloads frames according to the modified camera position stream. Since the download is performed in units of frames, the playback can achieve low-latency interaction of viewing angles, and switching camera positions does not affect picture rendering or cause camera position jumps, which can ensure the smoothness of screen switching.

The method for processing a free-view video scene provided by the present application will be further described below in conjunction with the accompanying drawings and specific embodiments.

As shown in Figure 8, taking a live broadcast as an example, the video capture module collects multi-camera video streams, and the server synchronizes the video frames of the multi-camera video streams, and merges the synchronized multiple video streams into a single code stream. The merged streams are DASH sliced, and the corresponding frame index files are generated synchronously. The server slices and encapsulates the media files containing multiple streams and generates corresponding index files. The index files mark the information of all frames in the corresponding slices. , the server generates a media presentation description file, the client downloads the media presentation description file and parses the index file, video fragmentation, and audio fragmentation information in it, the client downloads the index file and parses the video frame information, and the client Video frame information, download the next frame of the current camera position by frame, the client will decode and render the downloaded frame, the user performs the viewing angle switching operation, the client modifies the value of the current camera position, and downloads the next frame of the switched camera position by frame One frame, the client decodes and renders the downloaded frame, and repeats this until the viewing angle switching operation ends.

As shown in Figure 9, taking video-on-demand as an example, the server performs DASH slices on the recorded combined streams and generates corresponding frame index files synchronously. The server slices and encapsulates media files containing multiple streams and generates corresponding indexes. file, the index file marks the information of all frames in the corresponding segment, the server generates the media presentation description file, the client downloads the media presentation description file and parses the index file, video segment, and audio segment information, and the client downloads the index file and parse the video frame information, the client downloads the next frame of the current camera frame by frame according to the video frame information in the index file, the client decodes and renders the downloaded frame, the user switches the viewing angle, and the client modifies the current The camera position value, and download the next frame of the switched camera position by frame, and the client will decode and render the downloaded frame, and repeat this until the viewing angle switching operation ends.

Taking Bullet Time as an example, the server slices and encapsulates the media files containing multiple streams and generates corresponding index files. The index files mark the information of all frames in the corresponding slices, the server generates media presentation description files, and the client downloads the media Present the description file and analyze the index file, video segment and audio segment information in it. The client downloads the index file and parses the video frame information. One frame, the client decodes and renders the downloaded frame, the user performs the bullet time operation, the client modifies the current position value, increments by 1, and downloads the same frame of the switched position by frame, and the client downloads the frame Perform decoding and rendering, and repeat until the end of the bullet time operation.

As shown in FIG. 10 , the embodiment of the present application also provides a client.

Specifically, the terminal includes: one or more processors and memories, and one processor and memories are taken as an example in FIG. 10 . The processor and the memory may be connected through a bus or in other ways, and connection through a bus is taken as an example in FIG. 10 .

As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs, such as the method for processing free-view video scenes in the above-mentioned embodiments of the present application. The processor executes the non-transitory software program and the program stored in the memory, so as to realize the processing method of the free-view angle video scene in the above-mentioned embodiment of the present application.

The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application program required by at least one function; the data storage area may store the processing method for executing the free-view video scene in the above-mentioned embodiments of the present application required data etc. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may include memory located remotely from the processor, and these remote memories may be connected to the terminal through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The non-transitory software programs and programs required to realize the processing method of the free-view video scene in the above-mentioned embodiments of the present application are stored in the memory. The processing method of the perspective video scene, for example, execute the method steps 101 to 103 in Figure 1 described above, the method steps 1011 to 1013 in Figure 2, and the method steps 1031 to 1034 in Figure 3, and the client obtains the index file, analyze the video frame information and camera position values of all camera positions according to the index file, obtain the camera position values for switching viewing angles, and download video frames according to the video frame information and camera position values for switching viewing angles. Based on this, it can not only realize the low latency of viewing angle interaction, but also ensure the smoothness of screen switching. By introducing the auxiliary file of index file, it can reduce unnecessary downloads while ensuring the quality of the screen, and at the same time, it is easy to expand other files later. Angle information.

As shown in FIG. 11 , the embodiment of the present application also provides a server.

Specifically, the electronic device includes: one or more processors and memories, and one processor and memories are taken as an example in FIG. 11 . The processor and the memory may be connected through a bus or in other ways, and connection through a bus is taken as an example in FIG. 11 .

As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs, such as the method for processing free-view video scenes in the above-mentioned embodiments of the present application. The processor executes the non-transitory software program and the program stored in the memory, so as to implement the method for processing the free-view video scene in the above-mentioned embodiments of the present application.

The non-transitory software programs and programs required to realize the processing method of the free-view video scene in the above-mentioned embodiments of the present application are stored in the memory. The processing method of the perspective video scene, for example, executes steps 201 to 203 of the method in Figure 7 described above, and the server slices and encapsulates the media file containing multiple code streams to obtain fragments. The fragments include video frame information, And generate an index file corresponding to the segment, and extract the video frame information in the segment to the index file, so that the client can download the index file from the server. Based on this, it can not only realize the low latency of viewing angle interaction, but also ensure the smoothness of screen switching. By introducing the auxiliary file of index file, it can reduce unnecessary downloads while ensuring the quality of the screen, and at the same time, it is easy to expand other files later. Angle information.

In addition, the embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores a computer-executable program, and the computer-executable program is executed by one or more control processors, for example, shown in FIG. 11 Execution by one of the processors can cause the above-mentioned one or more processors to execute the processing method of the free-view video scene in the above-mentioned embodiment of the present application, for example, execute the method steps 101 to 103 in FIG. 1 described above, FIG. From step 1011 to step 1013 of the method in 2, and from step 1031 to step 1034 of the method in FIG. 3, the client obtains the index file, parses out the video frame information and camera position values of all camera positions according to the index file, and obtains the switching angle of view Camera position value, download the video frame according to the video frame information and the camera position value for switching the viewing angle. Based on this, it can not only realize the low latency of viewing angle interaction, but also ensure the smoothness of screen switching. By introducing the auxiliary file of index file, it can reduce unnecessary downloads while ensuring the quality of the screen, and at the same time, it is easy to expand other files later. Angle information. Or, execute steps 201 to 203 of the method in FIG. 7 described above, the server slices and encapsulates the media file containing multiple code streams to obtain slices, and the slices include video frame information, and generate corresponding to the slices. Index file, extract the video frame information in the segment to the index file, so that the client can download the index file from the server. Based on this, it can not only realize the low latency of viewing angle interaction, but also ensure the smoothness of screen switching. By introducing the auxiliary file of index file, it can reduce unnecessary downloads while ensuring the quality of the screen, and at the same time, it is easy to expand other files later. Angle information.

The embodiment of the present application includes: the client obtains the index file, parses the video frame information and camera position values of all camera positions according to the index file, obtains the camera position value of the switching angle of view, and obtains the camera position value according to the video frame information and the camera position of the switching angle of view. Bit value to download video frames. Based on this, it can not only realize the low latency of viewing angle interaction, but also ensure the smoothness of screen switching. By introducing the auxiliary file of index file, it can reduce unnecessary downloads while ensuring the quality of the screen, and at the same time, it is easy to expand other files later. Angle information.

Those skilled in the art can understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware and an appropriate combination thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit . Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those of ordinary skill in the art, the term computer storage media includes both volatile and nonvolatile media implemented in any method or technology for storage of information, such as computer readable programs, data structures, program modules, or other data. permanent, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridges, tape, magnetic disk storage or other magnetic storage devices, or can Any other medium used to store desired information and which can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable programs, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .

The above is a specific description of several implementations of the present application, but the application is not limited to the above-mentioned implementations, and those skilled in the art can also make various equivalent deformations or replacements without violating the spirit of the application. These equivalent modifications or replacements are all within the scope defined by the claims of the present application.

Claims

A method for processing a free-view video scene, applied to a client, wherein the method includes:

Get the index file;

Analyzing video frame information and camera position values of all camera positions according to the index file;

Obtain the camera position value for switching the viewing angle, and download the video frame according to the video frame information and the camera position value for switching the viewing angle.
The method according to claim 1, wherein said obtaining the index file comprises:

Acquire a media presentation description file, where the media presentation description file is generated by a server;

Obtain the information of the index file according to the media presentation description file;

The index file is downloaded from the server according to the information of the index file.
The method according to claim 2, wherein the acquiring the camera position value for switching the viewing angle, and downloading the video frame according to the video frame information and the camera position value for switching the viewing angle include:

Get the current camera position value;

Determine the current viewing angle video frame to be downloaded according to the video frame information and the current camera position value;

Obtain the position value of the target camera;

Determine the target viewing angle video frame to be downloaded according to the video frame information and the target camera position value.
The method according to claim 3, wherein, after said obtaining the camera position value for switching the viewing angle, and downloading the video frame according to the video frame information and the camera position value for switching the viewing angle, further comprising :

Decode and render the video frame.
The method according to any one of claims 1 to 4, wherein the video frame information includes any one of the following:

Video frame start position information;

video frame size;

The video frame corresponds to the camera position value.
A method for processing a free-view video scene, applied to a server, wherein the method includes:

Slicing and encapsulating the media file containing multiple code streams to obtain fragments, the fragments including video frame information;

Generate an index file corresponding to the fragmentation;

extracting the video frame information in the slice to the index file.
The method of claim 6, further comprising;

Generate a media presentation description file, where the media presentation description file includes the information of the fragment and the information of the index file.
A client, comprising: a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein, when the processor executes the computer program, any one of claims 1 to 5 is realized The processing method of the free viewing angle video scene.
A server, comprising: a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein, when the processor executes the computer program, the computer program described in any one of claims 6 to 7 is implemented. The processing method of the free-view video scene described above.
A computer-readable storage medium, wherein the computer-readable storage medium stores a computer-executable program, and the computer-executable program is used to enable a computer to execute the free-view video according to any one of claims 1 to 5. The processing method of the scene, or the processing method of the free viewing angle video scene according to any one of claims 6 to 7.