CN111372145B - Viewpoint switching method and system for multi-viewpoint video - Google Patents
Viewpoint switching method and system for multi-viewpoint video Download PDFInfo
- Publication number
- CN111372145B CN111372145B CN202010297270.9A CN202010297270A CN111372145B CN 111372145 B CN111372145 B CN 111372145B CN 202010297270 A CN202010297270 A CN 202010297270A CN 111372145 B CN111372145 B CN 111372145B
- Authority
- CN
- China
- Prior art keywords
- viewpoint
- terminal
- time point
- video
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/21805—Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
- H04N21/23106—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion involving caching operations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
- H04N21/4331—Caching operations, e.g. of an advertisement for later insertion during playback
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/65—Transmission of management data between client and server
- H04N21/658—Transmission by the client directed to the server
- H04N21/6587—Control parameters, e.g. trick play commands, viewpoint selection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8547—Content authoring involving timestamps for synchronizing content
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Databases & Information Systems (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention discloses a viewpoint switching method of a multi-viewpoint video, which comprises the following steps: after receiving the viewpoint switching request, the terminal checks whether an IDR frame exists in the preset duration of the local cache content of the terminal at the current viewpoint; if the IDR frame exists in the preset duration, adding the time point of the IDR frame into a viewpoint switching request, and reporting to a server; if the IDR frame does not exist in the preset time length and the cache content is greater than the preset time length, adding the time point after the preset time length into the viewpoint switching request and reporting the viewpoint switching request to the server; if the IDR frame does not exist in the preset time length and the cache content is less than the preset time length, adding the time point at the end of the cache into the viewpoint switching request and reporting the viewpoint switching request to the server; and emptying the local cache of the terminal after the time point in the server viewpoint switching request is reported. The invention realizes the alignment of the timestamps of different viewpoints in the multi-viewpoint video switching, so that the connection of the front viewpoint and the rear viewpoint is synchronous in time. The invention also discloses a corresponding viewpoint switching system of the multi-viewpoint video.
Description
Technical Field
The invention belongs to the technical field of video playing, and particularly relates to a viewpoint switching method and system for a multi-viewpoint video.
Background
The multi-view video is obtained by shooting the same scene or object from different positions and angles through a camera array formed by a plurality of cameras, as shown in fig. 1, compared with the single-view video, the multi-view video can provide multi-angle information of the same scene or object, so that people can obtain richer visual experience. The multi-view video data collected by the camera array is encoded by the encoding server and then transmitted to a Content Delivery Network (CDN) server, and the CDN server performs streaming processing on the video and transmits the video to a user terminal (e.g., a television set-top box, a Personal Computer (PC), a mobile phone, and the like). When a user watches through the terminal, the view points in the multi-view video can be freely switched in modes such as direction keys, dragging and the like to watch the shot object at a changed angle, so that better watching experience compared with the traditional single-view video is achieved.
The coding server codes and compresses the multi-view original video code stream collected by the camera array to form a compressed multi-path video code stream, and the compressed video frames are generally divided into three types of frames I/P/B. The I frame is a key frame, and is a first frame constituting a Group of pictures (GOP); p-frames are future uni-predictive frames that store only the difference from a previously decompressed picture, and B-frames are bi-predictive frames that, in addition to referencing a previously decompressed picture, also reference picture information in a subsequent frame. The I frame includes IDR (Instantaneous Decoding Refresh) frame, the IDR frame is refreshed immediately, so that the error will not propagate, and a new sequence is calculated from the IDR frame to start encoding, which has random access capability not possessed by normal I frame, and the player can always play from an IDR frame in the random access video stream.
The CDN server completes streaming packaging of multi-view video content, extracts a video code stream of a certain view point according to user requirements, and sends the video code stream to the terminal. In the sending process, the CDN server generally has a cache area for temporarily storing video data, and the terminal also has a cache area for temporarily storing video data, and the total length of the video content in the CDN server and the terminal cache area may reach the level of seconds or even the level of ten seconds, so that the content played by the terminal and the content currently processed by the CDN server have a large delay at a time point.
Disclosure of Invention
In view of the above drawbacks or needs for improvement in the prior art, an object of the present invention is to provide a method and a system for switching viewpoints of a multi-viewpoint video, which process video broadcasts of a current viewpoint and a next viewpoint at a switching time point when switching viewpoints of the multi-viewpoint video, and achieve natural connection of videos of different viewpoints at the same time point to improve the viewpoint switching effect.
To achieve the above object, according to an aspect of the present invention, there is provided a viewpoint switching method of a multi-viewpoint video, including:
after receiving a user viewpoint switching request, the terminal checks whether an IDR frame exists in a preset duration of terminal local cache contents of a current viewpoint;
if the IDR frame exists in the preset duration of the local cache content of the terminal at the current viewpoint, adding the time point of the IDR frame into the viewpoint switching request, and reporting to the server;
if the IDR frame does not exist in the preset time length of the local cache content of the terminal at the current viewpoint and the cache content is greater than the preset time length, adding the time point after the preset time length into the viewpoint switching request and reporting the viewpoint switching request to the server;
if the IDR frame does not exist in the preset time length of the local cache content of the terminal at the current viewpoint and the cache content is less than the preset time length, adding the time point at the tail end of the local cache of the terminal into the viewpoint switching request and reporting the viewpoint switching request to a server;
and emptying the local cache of the terminal after the time point in the server viewpoint switching request is reported.
In an embodiment of the present invention, after reporting the viewpoint switching request to the server, the terminal further receives a viewpoint-switched video stream sent by the server, and the specific steps are as follows:
a server receives a viewpoint switching request sent by a terminal;
if the IDR frame exists in the preset duration of the local cache content of the terminal, finding out the IDR frame corresponding to the time point directly from the video of the next viewpoint to be switched, which is stored in the server, emptying the cache of the server, and then carrying out streaming packaging from the time point and sending the IDR frame to the terminal;
if the local cache content of the terminal does not have an IDR frame before the time point in the viewpoint switching request is reported, combining the video frame of the corresponding time point of the next viewpoint to be switched on the server with the related video frame of the GOP where the time point is located, calculating and synthesizing an IDR frame, replacing the original non-IDR frame of the time point, emptying the cache of the server, and then carrying out streaming packaging from the time point and sending the streaming packaging to the terminal.
In an embodiment of the present invention, in the multi-view video, each video has an IDR frame at each time node of n seconds, the IDR frames of the videos are aligned at a time point, and n is a preset value.
In an embodiment of the present invention, the calculating and synthesizing a video frame at a time point corresponding to a next viewpoint to be switched to on the server with a relevant video frame of a GOP at the time point to synthesize an IDR frame specifically includes: decompressing all the coding frames in the GOP to which the time point belongs to generate original video data, intercepting the original video data according to the offset of the time point relative to the GOP starting point time, performing compression coding on the intercepted original video data by taking the time point as a starting point, and generating an IDR frame from a first frame to replace the coding frame in the original GOP.
In one embodiment of the present invention, an extension field is added before each frame of original video data of video content for viewpoint switching, and the extension field includes camera viewpoint information and timestamp information.
In one embodiment of the present invention, the camera viewpoint information includes: the number of the current viewpoint, the total number of the viewpoints, the frame information in the message and the frame type.
In an embodiment of the present invention, the value of n is 2 seconds, and the preset duration is 500 milliseconds.
According to another aspect of the present invention, there is also provided a viewpoint switching system of a multi-viewpoint video, including a terminal and a server, wherein:
the terminal is used for checking whether an IDR frame exists in the preset duration of the local cache content of the terminal at the current viewpoint after receiving a user viewpoint switching request;
if the IDR frame exists in the preset duration of the local cache content of the terminal at the current viewpoint, adding the time point of the IDR frame into the viewpoint switching request, and reporting to the server;
if the IDR frame does not exist in the preset time length of the local cache content of the terminal at the current viewpoint and the cache content is greater than the preset time length, adding the time point after the preset time length into the viewpoint switching request and reporting the viewpoint switching request to the server;
if the IDR frame does not exist in the preset time length of the local cache content of the terminal at the current viewpoint and the cache content is less than the preset time length, adding the time point at the tail end of the local cache of the terminal into the viewpoint switching request and reporting the viewpoint switching request to a server;
and emptying the local cache of the terminal after the time point in the server viewpoint switching request is reported.
In an embodiment of the present invention, after reporting the viewpoint switching request to the server, the terminal further receives a viewpoint-switched video stream sent by the server, and the specific steps are as follows:
the server receives a viewpoint switching request sent by a terminal;
if the IDR frame exists in the preset duration of the local cache content of the terminal, finding out the IDR frame corresponding to the time point directly from the video of the next viewpoint to be switched, which is stored in the server, emptying the cache of the server, and then carrying out streaming packaging from the time point and sending the IDR frame to the terminal;
if the local cache content of the terminal does not have an IDR frame before the time point in the viewpoint switching request is reported, combining the video frame of the corresponding time point of the next viewpoint to be switched on the server with the related video frame of the GOP where the time point is located, calculating and synthesizing an IDR frame, replacing the original non-IDR frame of the time point, emptying the cache of the server, and then carrying out streaming packaging from the time point and sending the streaming packaging to the terminal.
In an embodiment of the present invention, in the multi-view video, each video has an IDR frame at each time node of n seconds, the IDR frames of the videos are aligned at a time point, and n is a preset value.
Generally, compared with the prior art, the technical scheme of the invention has the following beneficial effects:
(1) the invention realizes the alignment of time stamps of different viewpoints in the multi-viewpoint video switching, so that the connection of front and rear viewpoints is synchronous in time;
(2) according to the invention, through judgment and processing of the IDR frame, the conditions of screen splash, screen blackout and the like can be ensured not to occur in the viewpoint switching connection process, and the user experience of viewpoint switching is improved;
(3) according to the invention, the playing fluency in the process of connecting the viewpoints before and after switching can be improved by processing the video caching conditions of the CDN server and the set top box terminal.
Drawings
FIG. 1 is a schematic diagram of a multi-view video acquired by a camera array;
fig. 2 is a service scene architecture diagram of multi-view video playing in the embodiment of the present invention;
FIG. 3 is a diagram of a multi-view video coding IDR/P/B frame;
FIG. 4 is a diagram illustrating a viewpoint switching method for multi-viewpoint video according to an embodiment of the present invention;
fig. 5 is a flow chart of a terminal and a CDN server side in switching a multi-view video view according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a viewpoint switching system of a multi-viewpoint video in an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
The technical problem to be solved by the invention is as follows:
in the process of playing the multi-view video, a user controls a terminal to switch views (for example, by using a direction key of a television set-top box remote controller, etc.), a CDN server responds to a request of the user to switch a picture played from a current view to a picture played from a next view, and the user expects that the current view and the next view played by the terminal are aligned at the time point of the switching.
In video coding, there are multiple types (I/P/B) of image frames, where only the IDR frame in the I frame is a type independent without referring to other frames, when a user requests viewpoint switching, a picture played by a next viewpoint corresponding to a switching time point may be a frame type that needs to be associated with other frames, and the next viewpoint playing frame has no association with a previous viewpoint frame, resulting in screen omission and screen blackness.
In addition, due to the cache function of the set-top box terminal and the server, when a user sends a switching command, the video content of the current viewpoint is still stored in the cache for a period of time, and the video of the next viewpoint can be displayed after a long time.
Due to the video frame characteristics and the factors of equipment cache, switching and playing of front and rear viewpoints cannot be correctly connected at the same time point, and the viewpoint switching effect is influenced.
Example 1
Fig. 2 is a service scene architecture diagram of multi-view video playing in an embodiment of the present invention, which adopts a multi-view video end-to-end system, and is divided into 2 parts, namely live broadcast and on-demand broadcast, wherein live broadcast uses a camera array to acquire multi-view video content of an object in real time, compression encoding is performed by an encoding server, and then streaming packaging is performed by a CDN server, and the streaming packaging is sent to a terminal; and the on-demand content is directly injected into the CDN server in a file mode, is processed by the CDN server, is subjected to streaming packaging and is sent to the terminal.
In the live broadcast process, the camera array collects original video data collected by a total of x cameras, when the coding server performs compression coding, as shown in fig. 3, an IDR frame is generated for each path of video code stream in a period of n seconds, that is, each path of video has an IDR frame at each time node of n seconds, that is, the gop (group of pictures) length of the coded video is n seconds, the IDR frames of each path of video are aligned at a time point, a total of x paths of coded video code streams are generated, n is a preset value, and it is more appropriate that the value of n is 2 seconds through test verification.
The requested multi-view video file is injected into the CDN server and stored by the CDN server, and because the coding modes of the video files obtained from the outside are possibly different, the injected video file needs to be checked and transcoded by the CDN server, IDR frames are generated by taking n seconds as a period, and the IDR frames of all paths of videos are kept aligned at a time point.
And generating the compressed video content of the IDR frame periodically, and performing streaming packaging on the video of each viewpoint by using a streaming media protocol by the CDN server. In the packaging process, the CDN server adds an extension field of 8 bytes, including camera viewpoint information and timestamp information, in front of the original video data of each frame of video content, for subsequent viewpoint switching use. The camera viewpoint information may be: the number of the current viewpoint, the total number of the viewpoints, the frame information in the message and the frame type.
For example, as shown in table 1, 1 byte is used to record the number of the current camera (viewpoint); 1 byte for recording the total number of cameras (viewpoints); the FP field occupies 4 bits, the value of 1 represents that a frame is in the message, the value of 2 represents the head of the frame, the value of 3 represents the middle section of the frame, and the value of 4 represents the end of the frame; the FT field occupies 4 bits, the frame is an IDR frame when the value is 1, the frame is a P frame when the value is 2, the frame is a B frame when the value is 3, and the frame is an I frame except the IDR frame when the value is 4; the last 4 bytes are used to record the time stamp in ms. When the terminal plays the video, the content of the extension field needs to be stripped off and then the video frame is played.
TABLE 1
During the process of playing a certain viewpoint video, a terminal (such as a television set-top box, a PC, a mobile phone and the like) initiates viewpoint switching through a control mode such as a direction key and the like, so as to switch the currently played viewpoint to other viewpoints to play the video. In the process of switching viewpoints, because the front and back contents of videos of different viewpoints have no reference relationship, the video frame played by a new viewpoint after switching is required to be an IDR frame, so that the new viewpoint played video is not related to the video played by the previous viewpoint, and the conditions of screen blooming and screen blacking cannot be caused. Meanwhile, at the time point of switching, it is necessary to ensure that the front and rear viewpoints can be normally connected at the time point, otherwise, poor viewing experience is caused by time hopping of the front and rear viewpoints, which requires processing data of the front viewpoint in the cache including the CDN server and the terminal.
Example 2
As shown in fig. 4, a viewpoint switching method for a multi-viewpoint video in an embodiment of the present invention includes:
s1, after receiving the user viewpoint switching request, the terminal checks whether the IDR frame exists in the preset duration of the terminal local cache content of the current viewpoint;
s2, if the IDR frame exists in the preset duration of the local cache content of the terminal of the current viewpoint, adding the time point of the IDR frame into the viewpoint switching request, and reporting to the server;
s3, if the IDR frame does not exist in the preset time length of the local cache content of the terminal of the current viewpoint and the cache content is greater than the preset time length, adding the time point after the preset time length into the viewpoint switching request and reporting the viewpoint switching request to the server;
s4, if the IDR frame does not exist in the preset duration of the local cache content of the terminal of the current viewpoint and the cache content is less than the preset duration, adding the time point at the end of the local cache of the terminal into the viewpoint switching request and reporting the request to the server;
and S5, emptying the local buffer of the terminal after the time point in the server viewpoint switching request is reported.
It should be noted that: the steps S2, S3, and S4 are parallel, not sequential, i.e., one of the steps S2, S3, and S4 may be executed according to the determination condition.
In addition, when the CDN receives the handover request, the CDN may store the video content of the previous view in a local cache of the CDN, and this part of the content does not need to be sent to the terminal again, so the CDN is also required to empty the local cache when the handover request is received.
Example 3
When a terminal receives a command of switching viewpoints of a user, checking the content of local cache of the terminal for the extended field timestamp information of each frame, if an IDR frame exists in the preset duration (for example, m milliseconds) of the content of the local cache of the terminal of the current viewpoint, adding the time point of the IDR frame into a viewpoint switching request, and reporting the viewpoint switching request to a CDN server; if the IDR frame does not exist in m milliseconds of the local cache content of the terminal of the current viewpoint and the cache content is greater than m milliseconds, adding the time point after m milliseconds into the viewpoint switching request and reporting the viewpoint switching request to the CDN server; and if the IDR frame does not exist in the preset duration of the local cache content of the terminal of the current viewpoint and the cache content is less than m milliseconds, adding the time point at the tail end of the local cache of the terminal into the viewpoint switching request and reporting the viewpoint switching request to the CDN server. And emptying the local cache of the terminal after the time point of the report server.
The CDN server receives a viewpoint switching request sent by the terminal, if the reported terminal cache has an IDR frame within m milliseconds, the IDR frame corresponding to a time point is found out directly from the next viewpoint to be switched stored in the CDN server according to the timestamp information of the extension field, and after the local cache is emptied, the IDR frame is subjected to streaming packaging from the time point and sent to the terminal; if no IDR frame exists before the reported terminal switching time point is sent, the stored video frame of the time point corresponding to the next viewpoint to be switched on the CDN server is combined with the relevant video frame of the GOP where the time point is located to calculate and synthesize an IDR frame, replace the original non-IDR frame of the time point, and the IDR frame is streamed and packaged from the time point and sent to the terminal.
The method for calculating and synthesizing an IDR frame may be: decompressing all the coded frames in the GOP (n seconds) to which the time point belongs to generate n seconds of original video data, intercepting the original video data according to the offset of the time point relative to the GOP starting time, performing compression coding on the intercepted original video data by taking the time point as a starting point, generating an IDR frame from a first frame, and replacing the coded frames in the original GOP.
When the terminal switches viewpoints, the longest m-millisecond content of the original viewpoint is played from the switching time point, then the video content of the new viewpoint is played, and the front viewpoint and the rear viewpoint are aligned at the switching time point. Tests prove that the value of m milliseconds is more suitable to be 500 milliseconds.
It should be noted that: when receiving the handover request, the CDN may store the video content of the previous view in a local cache of the CDN, and this part of the content does not need to be sent to the terminal again, so the CDN is also required to empty the local cache when receiving the handover request.
Example 4
Fig. 5 is a flow chart of a terminal and a CDN server side in switching a multi-view video view according to an embodiment of the present invention; the embodiment of the invention describes the processing of the terminal and the CDN server for the connection and playing of the front view and the rear view when the multi-view video is switched.
Wherein, on the terminal side:
s101: a terminal (such as a television set-top box, a mobile phone, a PC and the like) is playing video content of a multi-view video view A;
s102: a user initiates viewpoint switching operation by means of terminal direction keys and the like (such as direction keys of a set top box remote controller, mobile phone sliding and PC mouse dragging);
s103: the terminal analyzes the operation request of the user and judges that the next viewpoint to be switched is B;
s104: the terminal checks the local cache content aiming at the current viewpoint A and searches whether the video frame in m milliseconds has an IDR frame;
s105: the terminal finds that the video frame content of the viewpoint A cached locally at the terminal has an IDR frame within m milliseconds, reports the time point information of the IDR frame and a switching request for switching the viewpoint to the viewpoint B to the CDN server, and simultaneously clears the IDR frame time point and the content behind the IDR frame time point in the local cache at the terminal;
s106: if the terminal does not find the IDR frame in m milliseconds of the local cache of the terminal, checking whether the content length of the local cache of the terminal is greater than m milliseconds;
s107: when the terminal finds that the length of the local content is greater than m milliseconds, reporting time point information of a video frame at the m milliseconds, an IDR frame generation request and a viewpoint switching request to a CDN server, and meanwhile emptying the content of the local cache at the m milliseconds and later;
s108: when the terminal finds that the local content length is less than m milliseconds, reporting the time point information of the terminal local cache end, the IDR frame generation request and the viewpoint switching request to the CDN server;
s109: and the terminal receives and plays the content of the next viewpoint B after switching.
On the CDN server side:
s201: the CDN server checks the multi-view video content, ensures that the video of each view takes n seconds as a period, and an IDR frame is fixed, if the video does not meet the period, transcoding is required, and the IDR frame with n seconds as the period is generated to replace the original frame;
s202: when the CDN server is in streaming packaging, before each video frame of each viewpoint, adding the viewpoint and an extension field of timestamp information in a description mode in the scheme of embodiment 1 of the invention;
s203: after receiving a switching request of a terminal for switching a viewpoint to a B and time point information of an IDR frame cached by the terminal, the CDN server empties a local cache of the CDN server and directly provides a video code stream of a corresponding time point of a next viewpoint B according to the time point information without calculation and synthesis, wherein the first frame is the IDR frame;
s204: and when the CDN server receives a switching request for switching the viewpoint of the terminal to B, an IDR frame generation request and generation time point information, clearing the local cache, converting a video frame of the next viewpoint B at the time point into an IDR frame through synthesis calculation, replacing an original frame of the viewpoint B at the time point, simultaneously sending the IDR frame as a first frame to the terminal, and continuously sending a video code stream of the viewpoint B.
Example 5
Fig. 6 is a schematic structural diagram of a viewpoint switching system of a multi-viewpoint video according to an embodiment of the present invention. The system comprises a terminal and a server, wherein:
the terminal is used for checking whether an IDR frame exists in the preset duration of the local cache content of the terminal at the current viewpoint after receiving a user viewpoint switching request;
if the IDR frame exists in the preset duration of the local cache content of the terminal at the current viewpoint, adding the time point of the IDR frame into the viewpoint switching request, and reporting to the server;
if the IDR frame does not exist in the preset time length of the local cache content of the terminal at the current viewpoint and the cache content is greater than the preset time length, adding the time point after the preset time length into the viewpoint switching request and reporting the viewpoint switching request to the server;
if the IDR frame does not exist in the preset time length of the local cache content of the terminal at the current viewpoint and the cache content is less than the preset time length, adding the time point at the tail end of the local cache of the terminal into the viewpoint switching request and reporting the viewpoint switching request to a server;
and emptying the local cache of the terminal after the time point in the server viewpoint switching request is reported.
Further, after reporting the viewpoint switching request to the server, the terminal also receives a viewpoint-switched video stream sent by the server, specifically:
the server receives a viewpoint switching request sent by a terminal;
if the IDR frame exists in the preset duration of the local cache content of the terminal, finding out the IDR frame corresponding to the time point directly from the video of the next viewpoint to be switched, which is stored in the server, emptying the cache of the server, and then carrying out streaming packaging from the time point and sending the IDR frame to the terminal;
if the local cache content of the terminal does not have an IDR frame before the time point in the viewpoint switching request is reported, combining the video frame of the corresponding time point of the next viewpoint to be switched on the server with the related video frame of the GOP where the time point is located, calculating and synthesizing an IDR frame, replacing the original non-IDR frame of the time point, emptying the cache of the server, and then carrying out streaming packaging from the time point and sending the streaming packaging to the terminal.
Furthermore, in the multi-view video, each path of video has an IDR frame at each time node of n seconds, the IDR frames of each path of video are aligned at a time point, and n is a preset value.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
In some of the flows described in the present specification and claims and in the above-described figures, a number of operations are included that occur in a particular order, but it should be clearly understood that these operations may be performed out of order or in parallel as they occur herein, the order of the operations merely being used to distinguish between the various operations, and the order of execution does not itself represent any order of execution. Additionally, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel.
It will be understood by those skilled in the art that the foregoing is only a preferred embodiment of the present invention, and is not intended to limit the invention, and that any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A viewpoint switching method for a multi-viewpoint video, comprising:
after receiving a user viewpoint switching request, the terminal checks whether an IDR frame exists in a preset duration of terminal local cache contents of a current viewpoint;
if the IDR frame exists in the preset duration of the local cache content of the terminal at the current viewpoint, adding the time point of the IDR frame into the viewpoint switching request, and reporting to the server;
if the IDR frame does not exist in the preset time length of the local cache content of the terminal at the current viewpoint and the cache content is greater than the preset time length, adding the time point after the preset time length into the viewpoint switching request and reporting the viewpoint switching request to the server;
if the IDR frame does not exist in the preset time length of the local cache content of the terminal at the current viewpoint and the cache content is less than the preset time length, adding the time point at the tail end of the local cache of the terminal into the viewpoint switching request and reporting the viewpoint switching request to a server;
and emptying the local cache of the terminal after the time point in the server viewpoint switching request is reported.
2. The viewpoint switching method of a multi-viewpoint video according to claim 1, wherein after the terminal reports the viewpoint switching request to the server, the terminal further receives a viewpoint-switched video stream sent by the server, and the method specifically includes:
a server receives a viewpoint switching request sent by a terminal;
if the IDR frame exists in the preset duration of the local cache content of the terminal, finding out the IDR frame corresponding to the time point directly from the video of the next viewpoint to be switched, which is stored in the server, emptying the cache of the server, and then carrying out streaming packaging from the time point and sending the IDR frame to the terminal;
if the local cache content of the terminal does not have an IDR frame before the time point in the viewpoint switching request is reported, combining the video frame of the corresponding time point of the next viewpoint to be switched on the server with the related video frame of the GOP where the time point is located, calculating and synthesizing an IDR frame, replacing the original non-IDR frame of the time point, emptying the cache of the server, and then carrying out streaming packaging from the time point and sending the streaming packaging to the terminal.
3. The viewpoint switching method of a multi-viewpoint video according to claim 1 or 2, wherein in the multi-viewpoint video, each video path has one IDR frame at each time node of n seconds, the IDR frames of the video paths are aligned at a time point, and n is a preset value.
4. The viewpoint switching method of a multi-viewpoint video according to claim 2, wherein the calculating and synthesizing an IDR frame by combining a video frame at a time point corresponding to a next viewpoint to be switched to on the server with a video frame related to a GOP where the time point is located is specifically: decompressing all the coding frames in the GOP to which the time point belongs to generate original video data, intercepting the original video data according to the offset of the time point relative to the GOP starting point time, performing compression coding on the intercepted original video data by taking the time point as a starting point, and generating an IDR frame from a first frame to replace the coding frame in the original GOP.
5. The viewpoint switching method of multi-view video according to claim 1 or 2, wherein an extension field is added for viewpoint switching before each frame of original video data of video contents, the extension field including camera viewpoint information and time stamp information.
6. The viewpoint switching method of multi-viewpoint video according to claim 5, wherein the camera viewpoint information includes: the number of the current viewpoint, the total number of the viewpoints, the frame information in the message and the frame type.
7. The viewpoint switching method of a multi-viewpoint video according to claim 3, wherein the value of n is 2 seconds, and the preset duration value is 500 milliseconds.
8. A viewpoint switching system of a multi-viewpoint video, comprising a terminal and a server, wherein:
the terminal is used for checking whether an IDR frame exists in the preset duration of the local cache content of the terminal at the current viewpoint after receiving a user viewpoint switching request;
if the IDR frame exists in the preset duration of the local cache content of the terminal at the current viewpoint, adding the time point of the IDR frame into the viewpoint switching request, and reporting to the server;
if the IDR frame does not exist in the preset time length of the local cache content of the terminal at the current viewpoint and the cache content is greater than the preset time length, adding the time point after the preset time length into the viewpoint switching request and reporting the viewpoint switching request to the server;
if the IDR frame does not exist in the preset time length of the local cache content of the terminal at the current viewpoint and the cache content is less than the preset time length, adding the time point at the tail end of the local cache of the terminal into the viewpoint switching request and reporting the viewpoint switching request to a server;
and emptying the local cache of the terminal after the time point in the server viewpoint switching request is reported.
9. The viewpoint switching system of a multi-viewpoint video according to claim 8, wherein after the terminal reports the viewpoint switching request to the server, it further receives a viewpoint-switched video stream sent by the server, and specifically:
the server receives a viewpoint switching request sent by a terminal;
if the IDR frame exists in the preset duration of the local cache content of the terminal, finding out the IDR frame corresponding to the time point directly from the video of the next viewpoint to be switched, which is stored in the server, emptying the cache of the server, and then carrying out streaming packaging from the time point and sending the IDR frame to the terminal;
if the local cache content of the terminal does not have an IDR frame before the time point in the viewpoint switching request is reported, combining the video frame of the corresponding time point of the next viewpoint to be switched on the server with the related video frame of the GOP where the time point is located, calculating and synthesizing an IDR frame, replacing the original non-IDR frame of the time point, emptying the cache of the server, and then carrying out streaming packaging from the time point and sending the streaming packaging to the terminal.
10. The viewpoint switching system of a multi-viewpoint video according to claim 8 or 9, wherein in the multi-viewpoint video, each video has one IDR frame at each time node of n seconds, the IDR frames of the respective videos are aligned at a point in time, and n is a preset value.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010297270.9A CN111372145B (en) | 2020-04-15 | 2020-04-15 | Viewpoint switching method and system for multi-viewpoint video |
| PCT/CN2020/111970 WO2021208321A1 (en) | 2020-04-15 | 2020-08-28 | Multi-view video view switching method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010297270.9A CN111372145B (en) | 2020-04-15 | 2020-04-15 | Viewpoint switching method and system for multi-viewpoint video |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111372145A CN111372145A (en) | 2020-07-03 |
| CN111372145B true CN111372145B (en) | 2021-07-27 |
Family
ID=71212591
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010297270.9A Active CN111372145B (en) | 2020-04-15 | 2020-04-15 | Viewpoint switching method and system for multi-viewpoint video |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111372145B (en) |
| WO (1) | WO2021208321A1 (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111372145B (en) * | 2020-04-15 | 2021-07-27 | 烽火通信科技股份有限公司 | Viewpoint switching method and system for multi-viewpoint video |
| CN114035672A (en) * | 2020-07-20 | 2022-02-11 | 华为技术有限公司 | Video processing method and related equipment for virtual reality VR scene |
| CN111800653B (en) * | 2020-07-29 | 2021-06-11 | 北京大学深圳研究生院 | Video decoding method, system, device and computer readable storage medium |
| CN112135198A (en) * | 2020-09-11 | 2020-12-25 | 烽火通信科技股份有限公司 | Method, system and server for multiplexing multi-view switching video |
| CN112261418B (en) * | 2020-09-18 | 2022-09-30 | 网宿科技股份有限公司 | Method for transmitting live video data and live broadcast acceleration system |
| CN111866525A (en) * | 2020-09-23 | 2020-10-30 | 腾讯科技（深圳）有限公司 | Multi-view video playing control method and device, electronic equipment and storage medium |
| CN114584769A (en) * | 2020-11-30 | 2022-06-03 | 华为技术有限公司 | Visual angle switching method and device |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101459837A (en) * | 2009-01-09 | 2009-06-17 | 清华大学 | Method for control delay in interactive multiple vision point video stream media service |
| CN102055967A (en) * | 2009-10-28 | 2011-05-11 | 中国移动通信集团公司 | Visual angle switching and encoding method and device of multi-viewpoint video |
| CN102982159A (en) * | 2012-12-05 | 2013-03-20 | 上海创图网络科技发展有限公司 | Three-dimensional webpage multi-scenario fast switching method |
| WO2013162258A1 (en) * | 2012-04-23 | 2013-10-31 | 삼성전자 주식회사 | Multiview video encoding method and device, and multiview video decoding mathod and device |
| WO2014010537A1 (en) * | 2012-07-09 | 2014-01-16 | Mitsubishi Electric Corporation | Method and system for processing multiview videos for view synthesis using motion vector predictor list |
| CN107615756A (en) * | 2015-07-10 | 2018-01-19 | 华为技术有限公司 | Realize the multi-view point video Streaming Media of quick and smooth viewpoint switch |
| CN109218848A (en) * | 2017-07-06 | 2019-01-15 | 阿里巴巴集团控股有限公司 | View angle switch method, apparatus, equipment and the computer storage medium of video flowing |
| CN109698949A (en) * | 2017-10-20 | 2019-04-30 | 腾讯科技（深圳）有限公司 | Method for processing video frequency, device and system based on virtual reality scenario |
| CN110351607A (en) * | 2018-04-04 | 2019-10-18 | 优酷网络技术（北京）有限公司 | A kind of method, computer storage medium and the client of panoramic video scene switching |
| CN110800306A (en) * | 2017-05-19 | 2020-02-14 | 萨基姆宽带连接公司 | Immersive video transmission method |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1331359C (en) * | 2005-06-28 | 2007-08-08 | 清华大学 | Transmission method for video flow in interactive multi-viewpoint video system |
| CN100438632C (en) * | 2006-06-23 | 2008-11-26 | 清华大学 | Method for encoding interactive video in multiple viewpoints |
| US8787726B2 (en) * | 2012-02-26 | 2014-07-22 | Antonio Rossi | Streaming video navigation systems and methods |
| CN102970540B (en) * | 2012-11-21 | 2016-03-02 | 宁波大学 | Based on the multi-view video rate control of key frame code rate-quantitative model |
| CN105933343B (en) * | 2016-06-29 | 2019-01-08 | 深圳市优象计算技术有限公司 | A kind of code stream caching method for 720 degree of panoramic video netcasts |
| CN108319362B (en) * | 2018-01-02 | 2020-06-23 | 联想(北京)有限公司 | Panoramic information display method, electronic equipment and computer storage medium |
| CN111372145B (en) * | 2020-04-15 | 2021-07-27 | 烽火通信科技股份有限公司 | Viewpoint switching method and system for multi-viewpoint video |
-
2020
- 2020-04-15 CN CN202010297270.9A patent/CN111372145B/en active Active
- 2020-08-28 WO PCT/CN2020/111970 patent/WO2021208321A1/en active Application Filing
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101459837A (en) * | 2009-01-09 | 2009-06-17 | 清华大学 | Method for control delay in interactive multiple vision point video stream media service |
| CN102055967A (en) * | 2009-10-28 | 2011-05-11 | 中国移动通信集团公司 | Visual angle switching and encoding method and device of multi-viewpoint video |
| WO2013162258A1 (en) * | 2012-04-23 | 2013-10-31 | 삼성전자 주식회사 | Multiview video encoding method and device, and multiview video decoding mathod and device |
| WO2014010537A1 (en) * | 2012-07-09 | 2014-01-16 | Mitsubishi Electric Corporation | Method and system for processing multiview videos for view synthesis using motion vector predictor list |
| CN102982159A (en) * | 2012-12-05 | 2013-03-20 | 上海创图网络科技发展有限公司 | Three-dimensional webpage multi-scenario fast switching method |
| CN107615756A (en) * | 2015-07-10 | 2018-01-19 | 华为技术有限公司 | Realize the multi-view point video Streaming Media of quick and smooth viewpoint switch |
| CN110800306A (en) * | 2017-05-19 | 2020-02-14 | 萨基姆宽带连接公司 | Immersive video transmission method |
| CN109218848A (en) * | 2017-07-06 | 2019-01-15 | 阿里巴巴集团控股有限公司 | View angle switch method, apparatus, equipment and the computer storage medium of video flowing |
| CN109698949A (en) * | 2017-10-20 | 2019-04-30 | 腾讯科技（深圳）有限公司 | Method for processing video frequency, device and system based on virtual reality scenario |
| CN110351607A (en) * | 2018-04-04 | 2019-10-18 | 优酷网络技术（北京）有限公司 | A kind of method, computer storage medium and the client of panoramic video scene switching |
Non-Patent Citations (1)
| Title |
|---|
| 多视点视频系统用户交互技术研究;胡彬彬;《华中科技大学学报(自然科学版)》;20121215;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111372145A (en) | 2020-07-03 |
| WO2021208321A1 (en) | 2021-10-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111372145B (en) | Viewpoint switching method and system for multi-viewpoint video | |
| US8914835B2 (en) | Streaming encoded video data | |
| JP5788101B2 (en) | Network streaming of media data | |
| US7613381B2 (en) | Video data processing method and video data processing apparatus | |
| JP3801984B2 (en) | Multicast transmission system including bandwidth scaler | |
| US20070103558A1 (en) | Multi-view video delivery | |
| CN107566918A (en) | A kind of low delay under video distribution scene takes the neutrel extraction of root | |
| CN101917613A (en) | Acquiring and coding service system of streaming media | |
| KR20140126372A (en) | data, multimedia and video transmission updating system | |
| WO2009103343A1 (en) | Method and apparatus for distributing media over a communications network | |
| CN112153401B (en) | Video processing method, communication device and readable storage medium | |
| WO2015020069A1 (en) | Data processing device, data processing method, program, recording medium, and data processing system | |
| CN110798713B (en) | Time-shifted television on-demand method, terminal, server and system | |
| CN115883855B (en) | Playing data processing method, device, computer equipment and storage medium | |
| WO2009080114A1 (en) | Method and apparatus for distributing media over a communications network | |
| KR100606722B1 (en) | Method of providing video streamin service having higher times of rate | |
| CN116055810A (en) | Audio and video playing method, device and system, electronic equipment and storage medium | |
| WO2009080116A1 (en) | Method and apparatus for distributing media over a communications network | |
| WO2009080111A1 (en) | Method and apparatus for distributing media over a communications network | |
| WO2009080117A1 (en) | Method and apparatus for distributing media over a communications network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |