CN112492348A - Multi-code stream processing and output method, equipment and storage medium - Google Patents
Multi-code stream processing and output method, equipment and storage medium Download PDFInfo
- Publication number
- CN112492348A CN112492348A CN202011126892.1A CN202011126892A CN112492348A CN 112492348 A CN112492348 A CN 112492348A CN 202011126892 A CN202011126892 A CN 202011126892A CN 112492348 A CN112492348 A CN 112492348A
- Authority
- CN
- China
- Prior art keywords
- code stream
- stream data
- data
- path
- output
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 230000005540 biological transmission Effects 0.000 claims abstract description 91
- 238000003672 processing method Methods 0.000 claims abstract description 15
- 238000004590 computer program Methods 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000009751 slip forming Methods 0.000 description 3
- 238000005538 encapsulation Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
- H04N21/23602—Multiplexing isochronously with the video sync, e.g. according to bit-parallel or bit-serial interface formats, as SDI
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/242—Synchronization processes, e.g. processing of PCR [Program Clock References]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/643—Communication protocols
- H04N21/6437—Real-time Transport Protocol [RTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/647—Control signaling between network components and server or clients; Network processes for video distribution between server and clients, e.g. controlling the quality of the video stream, by dropping packets, protecting content from unauthorised alteration within the network, monitoring of network load, bridging between two different networks, e.g. between IP and wireless
- H04N21/64784—Data processing by the network
- H04N21/64792—Controlling the complexity of the content stream, e.g. by dropping packets
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computer Security & Cryptography (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The invention relates to a multi-code stream processing and output method, equipment and a storage medium, wherein the processing method comprises the following steps: multiplexing the same UVC transmission channel to output multi-path code stream data. The same UVC transmission channel is multiplexed to output multi-path code stream data, the problem that the existing UVC protocol is limited to multi-code stream data transmission is solved, and meanwhile, the single code stream but the multi-path code stream data are output to the host end equipment, so that transcoding and recoding of the host end equipment to the code stream data are avoided, and the CPU consumption of the host end equipment is reduced.
Description
Technical Field
The present invention relates to the field of video coding technologies, and in particular, to a method, an apparatus, and a storage medium for processing and outputting multiple code streams.
Background
In a video conference scene or a video live broadcast scene, since a far-end network (i.e., a network condition or a network environment of a far-end device) has a large difference, a plurality of code streams are generally output for video data of the same picture so as to adapt to different far-end networks. The code stream (Data Rate) indicates the number of times that the communication port changes in high and low levels when transmitting Data within 1 second, and is also called the code Rate, and the more Data is transmitted per unit time, the more information is contained, and the higher the picture quality of the video is. The multi-code stream technology generates a plurality of stream media data with different code streams and resolutions simultaneously in the encoding process, and automatically allocates a solution scheme of relatively optimal decoding image quality for the stream media data according to the actual network bandwidth condition of a user. In the actual network live broadcast/video conference application, because the network environments of the visitors at different network positions are different, the transmission of the network live broadcast streaming media only with a certain fixed code stream resolution often causes the image quality seen by the user with higher network speed to be still not clear enough, the decoding time of the user with lower network speed is too long to make the picture not smooth enough, and in order to solve the contradiction between the two, the visitors browse the live broadcast content which is clear and smooth as much as possible, the multi-code stream technology is adopted to become the simplest and most effective method.
As shown in fig. 1, for a camera device (abbreviated as UVC device) performing data transmission based on a UVC protocol, in order to adapt to video conference/live broadcast clients in various network environments, generally, after video data of a large code stream (for example, 1080p video data) is obtained from the UVC device (such as a USB camera device), a host device is used to transcode the large code stream data and encode the large code stream data again, so as to form multiple paths of code stream data (generally, 3 paths) to adapt to each video conference/live broadcast client, if a network environment of the client is good, 1080p video data is output, if the network environment of the client is general, 360p video data is output, and so on. Therefore, code stream data input to the host end equipment from the UVC transmission channel needs to be coded after being transcoded by the host end equipment, but the transcoding and coding mode consumes a large amount of CPU at the host end and has high performance requirements. If the mode of transcoding and recoding is completed by the UVC device before the video data is transmitted to the host end device, the burden of the host device can be greatly reduced. Whether the UVC device is capable of transcoding and re-encoding the video data is mainly based on rules and standards of the UVC protocol, and the support of the UVC protocol on multi-code stream data transmission will be described in detail below:
the UVC protocol version 1.0 does not support transmission of multiple paths of code stream data, but can achieve output of multiple code stream data through a composite mode of multiple UVC devices, but has the disadvantages that each path of video requires 2 endpoint resources, but at present, the endpoint resources of host-side devices are all within 4, so most host-side devices cannot support transmission of more than two paths of code stream data, and for the host-side devices, multiple UVC devices are not easily controlled.
The UVC protocol version 1.5 supports transmission of multiple paths of code stream data, but still needs to increase consumption of endpoint resources, that is, each time transmission of one path of code stream data is increased, one endpoint resource needs to be added. As the endpoint resource needs to be matched with the UVC device itself, the mainstream UVC device (such as a common USB device) does not support the standard for a while, and the common transmission software of the host device does not support the standard, which causes a compatibility problem.
By combining the limitations of the two versions of the UVC protocol and the unsolved problems brought by the prior art, a novel multi-code-stream processing and transmitting method applied to the UVC device needs to be developed urgently, and the problems can be solved at the same time.
Disclosure of Invention
The present invention is directed to overcome at least one of the above-mentioned drawbacks of the prior art, and provides a method, an apparatus, and a storage medium for processing and outputting a multi-code stream, so as to solve the problem of limitation in multi-code stream data transmission of a UVC apparatus, avoid recoding of the multi-code stream data by a host side apparatus, and reduce CPU consumption of the host side apparatus.
The technical scheme adopted by the invention is as follows:
a multi-code stream processing method comprises the following steps: multiplexing the same UVC transmission channel to output multi-path code stream data.
As described in the background art, although the UVC protocol 1.5 version supports the multi-code stream coding technology, when outputting multi-channel code stream data, the consumption of endpoint resources needs to be increased, and when outputting one channel of code stream data, one endpoint resource is needed, and because the UVC protocol 1.5 version needs to increase a plurality of endpoint resources when transmitting multi-channel code stream data, a plurality of mainstream UVC devices and software of a host device do not support the protocol of the version. Compared with the version 1.5 of the UVC protocol, the multi-code stream processing method outputs the multi-channel code stream data by multiplexing the same UVC transmission channel, the multi-code stream data is only output through one endpoint resource, namely the same UVC transmission channel, and extra endpoint resources are not required to be added, so that both mainstream UVC equipment and host end equipment for transmitting data with the UVC equipment can be suitable for the method to transmit the multi-code stream data, and the problem of limitation of the existing UVC protocol on the multi-code stream data transmission is solved. Meanwhile, because the code stream output to the host end equipment is not a single code stream but multi-path code stream data, the transcoding and recoding of the host end equipment to the code stream data are avoided, and the CPU consumption of the host end equipment is reduced.
Further, before multiplexing the same UVC transmission channel to output multiple paths of code stream data, coding the video data simultaneously to form multiple paths of code stream data. And carrying out multi-path code stream coding on the video data, wherein the coding of the multi-path code stream data can be carried out simultaneously and are mutually independent. The process of video encoding can be summarized as: the video data is preprocessed, compressed and encoded, and the compressed data is encapsulated into data packets, each encapsulation finishes one data packet to represent that one path of code stream data corresponding to the data packet finishes encoding, under the condition of encoding multi-path code stream data, a plurality of data packets can be encapsulated at a certain moment or in a short time, and the plurality of data packets correspond to the multi-path code stream data which is encoded. Because the video data is continuous data, continuous code stream data is continuously formed in the encoding process, namely continuous data packets are continuously packaged, and the continuous data packets can correspond to all parts of the video data before encoding. Therefore, the method provided by the invention realizes the coding of the multi-path code stream data and can output the multi-path code stream data to the host end equipment through one channel, so that the host end equipment only needs to forward the multi-path code stream data to an opposite end or other equipment/modules, transcoding and recoding the code stream data are not needed, and the CPU consumption of the host end equipment is reduced.
Further, encoding the video data simultaneously to form multi-path code stream data, specifically: and simultaneously coding video data according to different coding parameters to form a plurality of paths of code stream data, and adding a corresponding unique identifier in each path of code stream data.
The method carries out multi-code stream coding on video data, and comprises the following specific processes: and coding the code stream data by referring to the coding parameters corresponding to the code stream according to the code stream data to be generated, wherein unique identifiers are required to be added in each path of code stream data before coding is finished in the coding process, the unique identifiers are used for marking each path of code stream data, and the unique identifiers belonging to the same path of code stream data are the same. After the continuous data containing the multi-path code stream data is obtained, the host end equipment can identify each path of code stream data in the continuous data through the unique identification of each path of code stream data, splits the continuous data into independent each path of code stream data, and then respectively sends the independent each path of code stream data to the opposite end or other equipment/modules according to the requirements of the opposite end or other equipment/modules, so that the opposite end or other equipment/modules under different network environments can obtain the code stream data conforming to the network environment of the host end equipment.
Further, the position of the unique identifier added to each path of code stream data is at the head of the path of code stream data. The unique identification of each path of code stream data is added at the head of the code stream data, and the host end equipment can immediately identify different paths of code stream data through the unique identification of the head of each path of code stream data after receiving continuous code stream data, so that each path of code stream data is correspondingly output to an opposite end or other equipment/modules. The unique identification is arranged at the head of each path of code stream data, so that the identification speed of the host end equipment can be increased, the whole transmission process is smoother, and the time delay is lower.
Further, multiplexing the same UVC transmission channel to output multi-path code stream data, specifically: multiplexing the same UVC transmission channel, and alternately transmitting the encoded multi-channel code stream data. And serially outputting the multi-channel code stream data which is coded within a certain time or a short time in the UVC transmission channel, namely alternately using the UVC transmission channel to transmit the multi-channel code stream data which is coded. After the transmission of the multi-path code stream data is finished, the multi-path code stream data which is encoded in the next moment or the next time period is transmitted alternately, so that the continuous video data is ensured to be transmitted to the host end equipment orderly and timely after the multi-code stream encoding is finished, the host end equipment can transmit each path of code stream data to the client or other equipment timely, and the real-time performance of the video data is ensured.
A multi-stream output method, comprising the steps of: receiving continuous data input from the same UVC transmission channel, splitting the continuous data into multiple paths of code stream data, and outputting at least one path or multiple paths of the code stream data to one or more other devices or modules.
After continuous data input by the same UVC transmission channel is received, the continuous data can be output to other equipment or modules only by splitting the continuous data into multiple paths of code stream data without transcoding and recoding the continuous data, and the loss of resources is reduced.
Further, splitting the continuous data into multiple paths of code stream data, specifically: and identifying the unique identifier in each path of code stream data in the continuous data, and splitting the continuous data into a plurality of paths of code stream data according to the unique identifier in each path of code stream data.
In the process of splitting continuous data, the unique identification of each of multiple paths of code stream data in the continuous data needs to be relied on, the continuous data comprises multiple paths of code stream data, each path of code stream data comprises at least one unique identification, the position of each path of code stream data in the continuous data can be marked by identifying the unique identification of each path of code stream data, so that the continuous data can be split into each path of code stream data according to the identification, and then the multiple paths of code stream data are output to one or more devices or modules according to the requirements of other devices or modules.
The technical scheme adopted by the invention is as follows:
a multi-bitstream processing device, comprising: and the multiplexing output module is used for multiplexing the same UVC transmission channel to output multi-path code stream data.
A multi-bitstream processing device, comprising: the split output module is used for receiving continuous data input from the same UVC transmission channel, splitting the continuous data into multiple paths of code stream data, and outputting at least one path or multiple paths of the code stream data to one or more other devices or modules.
The USB camera equipment processes multi-code stream data, executes the multi-code stream processing method when processing the multi-code stream data, and after acquiring video data, the camera equipment encodes the video data into multi-channel code stream data by using the multi-code stream processing method and then multiplexes the multi-channel code stream data to the same UVC transmission channel for output.
A computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements any of the data processing methods described above.
Compared with the prior art, the invention has the beneficial effects that:
(1) multiplexing the same UVC transmission channel to output multi-path code stream data, solving the problem of limitation of the existing UVC protocol on multi-code stream data transmission, and simultaneously avoiding transcoding and recoding the code stream data by the host end equipment and reducing the CPU consumption of the host end equipment because the single code stream but the multi-path code stream data is output to the host end equipment;
(2) unique identifiers are added in the multiple paths of code stream data respectively for marking the code stream data of each path, and the host end equipment can identify the code stream data of each path in the continuous data through the unique identifiers and correspondingly output the continuous data to an opposite end or other equipment/modules, so that the opposite end or other equipment/modules in different network environments can acquire the code stream data conforming to the network environment of the opposite end or other equipment/modules;
(3) the multi-channel code stream data after being encoded can be alternately transmitted in the UVC transmission channel, so that the continuous video data can be orderly and timely transmitted to the host end equipment after the multi-code stream encoding is completed, the host end equipment can timely transmit the code stream data of each channel to the opposite end or other equipment/modules, and the real-time performance of the video data is ensured.
Drawings
Fig. 1 is a schematic diagram illustrating the principle of code stream data processing in the prior art.
Fig. 2 is a schematic diagram illustrating the principle of data processing of multiple code streams in embodiment 1.
Fig. 3 is a schematic structural diagram of alternate transmission of multiple paths of code stream data in a UVC transmission channel in embodiment 1.
FIG. 4 is a flowchart illustrating steps S1 to S3 in example 2.
Detailed Description
The drawings are only for purposes of illustration and are not to be construed as limiting the invention. For a better understanding of the following embodiments, certain features of the drawings may be omitted, enlarged or reduced, and do not represent the size of an actual product; it will be understood by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.
Example 1
The present embodiment provides a method for processing multiple code streams, where the method includes the steps of:
t1: multiplexing the same UVC transmission channel to output multi-path code stream data;
specifically, the multi-path code stream data output in step T1 is output to the host device, which refers to any device/apparatus/module capable of receiving the code stream data. As shown in fig. 2, compared with the UVC protocol version 1.5 described in the background art, the multi-code stream processing method provided in this embodiment outputs multiple code stream data by multiplexing the same UVC transmission channel, and after the multi-code stream is encoded, the multiple code stream data can be output only by using one endpoint resource (i.e., the same UVC transmission channel), and no additional endpoint resource needs to be added, so that mainstream UVC devices, such as USB camera devices, are all suitable for data transmission by the method provided in this embodiment, and thus it can be seen that the method solves the problem of limitation of the existing UVC protocol version 1.5 on multi-code stream data transmission.
As shown in fig. 2, after receiving multiple paths of code stream data, the host device may directly output the multiple paths of code stream data to each peer or other devices/modules according to actual bandwidth conditions or actual requirements of the peers or other devices/modules, so that the peers or other devices/modules can view clear and smooth real-time content as much as possible, especially in a video conference or a video live broadcast scenario. Preferably, after the host device detects that the network environment of the opposite terminal or other devices/modules changes, it may select to switch and output the code stream data of the opposite terminal or other devices/modules. Therefore, the method avoids transcoding and recoding the code stream data by the host-end equipment and reduces the CPU consumption of the host-end equipment because the code stream output to the host-end equipment is not a single code stream but multi-path code stream data.
Specifically, the method can be implemented by adding a corresponding function to an extended protocol of the UVC protocol, that is, updating the original UVC protocol, so that the mainstream UVC device can continue to use the original UVC protocol, and the method of the present embodiment can be applied to process the transmission of the multi-code stream data only by updating the added function to the extended protocol of the original UVC protocol. The method can also be an independent protocol and is used for controlling data transmission by being matched with the UVC protocol, mainstream UVC equipment can also follow the original UVC protocol, and the independent protocol capable of realizing the method can be combined with the original UVC protocol to carry out data transmission.
Further, before performing step T1, the following steps are performed:
t0: and simultaneously coding the video data to form multi-path code stream data.
Step T0 realizes the function of multi-path code stream coding for video data, the coding of multi-path code stream data can be carried out simultaneously and independently, that is, the coding of multi-path code stream data is processed in parallel; specifically, the video data is generally continuous video data acquired by a camera/image sensor or the like, and may also be a transcoded video file. The video data can be real-time or non-real-time data, for example, in a video conference or a video live scene, the video data is real-time data; the process of video encoding can be summarized as: the video data is preprocessed, compressed and encoded, and the compressed data is encapsulated into data packets, each encapsulation finishes one data packet to represent that one path of code stream data corresponding to the data packet finishes encoding, under the condition of encoding multi-path code stream data, a plurality of data packets can be encapsulated at a certain moment or in a short time, and the plurality of data packets correspond to the multi-path code stream data which is encoded. Specifically, each path of encoded stream data, i.e., each packet, generally includes the sequence number of each frame, the time stamp of the encoded data when generated, the video data, and the video data length of the subsequent video. Because the video data is continuous data, the video data is encoded continuously to form continuous code stream data, that is, continuous data packets are encapsulated continuously, and the continuous data packets can correspond to various parts of the video data before encoding, for example, under the MPEG encoding standard, a plurality of segments of picture groups are formed after encoding the continuous video data, and each segment of picture group corresponds to each of the continuously generated data packets, or each segment of picture group is a main constituent part of each of the continuously generated data packets.
Specifically, the specific implementation procedure of step T0 is as follows: and simultaneously coding the video data according to different coding parameters to form multiple paths of code stream data, and adding a corresponding unique identifier in each path of code stream data.
And coding the multi-path code stream data according to coding parameters corresponding to the code streams, adding unique identifiers in each path of code stream data before the coding is finished, wherein the unique identifiers are used for marking each path of code stream data, and the unique identifiers belonging to the same path of code stream data are the same. Illustratively, in conjunction with the added unique identifier, the fields contained in each flow of code stream, i.e. each packet, are as follows:
magic: a typical value is 'STRM' for preventing packet loss;
streamId is the ID of the current data stream, namely the unique identifier of each path of code stream data;
seq is the serial number of each frame of the same path of code stream and is used for checking whether packet loss and disorder occur;
timestamp: the time stamp when the current coding data is generated is used for audio and video playing synchronization;
DataLen: identifying a length of the following video data;
data: real video data.
The encoded multi-path code stream data can be used as a section of continuous data to be multiplexed with the same UVC transmission channel to be transmitted to host end equipment, after the host end equipment acquires the continuous data containing the multi-path code stream data, each path of code stream data can be identified in the continuous data through the unique identifier of each path of code stream data, the continuous data is divided into independent each path of code stream data, and then the independent each path of code stream data is respectively transmitted to an opposite end or other equipment/modules according to the requirements of the opposite end or other equipment/modules, so that the opposite end or other equipment/modules under different network environments can acquire the code stream data meeting the network environment of the host end equipment.
Preferably, the unique identifier added to each flow of code stream data is located at the head of the flow of code stream data. The unique identifier of each path of code stream data is added at the head of the code stream data, namely, at the head of the data packet, and after receiving the continuous code stream data, the host end device can immediately identify different paths of code stream data through the unique identifier of the head of each path of code stream data (the head of the data packet), so that the different paths of code stream data are correspondingly output to the opposite end or other devices/modules. The unique identification is arranged at the head of the code stream data, so that the identification speed of the host end equipment can be increased, the whole transmission process is smoother, and the time delay is lower.
In combination with the process of the multi-path code stream data simultaneously encoded in step T0, the specific execution process of step T1 is:
t1: multiplexing the same UVC transmission channel, and alternately transmitting the encoded multi-channel code stream data.
Since step T0 mentions that continuous multiple paths of code stream data, i.e. multiple continuous data packets, are continuously formed in the video encoding process, it is also necessary to define how to transmit these continuously formed code stream data in step T1 for the transmission of multiple paths of code stream data, in this embodiment, step T1 adopts an alternate transmission mode for the transmission of multiple paths of code stream data, and the specific process is as follows: and serially outputting the multi-path code stream data (corresponding to a plurality of data packets which are packaged) which are encoded within a certain time or a short time period in the UVC transmission channel, namely alternately using the UVC transmission channel to transmit the multi-path code stream data which are encoded. After the transmission of the multi-channel code stream data coded in the time or the time period is finished, the multi-channel code stream data (corresponding to a plurality of data packets which are packaged and finished) coded in the next time/time period are alternately transmitted, so that the continuous video data are ensured to be orderly and timely transmitted to the host end equipment after the multi-channel code stream coding is finished, the host end equipment can timely transmit the multi-channel code stream data to an opposite end or other equipment/modules after the multi-channel code stream data are obtained through identification and splitting, and the real-time performance of the video data is ensured.
As shown in fig. 3, assuming that the multiple paths of code stream data to be transmitted include three paths of code stream data, code stream data 1, code stream data 2, and code stream data 3 (corresponding to three data packets that are completely encapsulated) that are completely encoded within a time t1 (which is a certain time or a shorter time period) are serially output in the UVC transmission channel, that is, the UVC transmission channel is alternately used to transmit the code stream data 1, the code stream data 2, and the code stream data 3. After the transmission of the code stream data 1, the code stream data 2 and the code stream data 3 which are encoded within the time t1 is finished, the code stream data 1, the code stream data 2 and the code stream data 3 which are encoded within the next time/time period t2 are alternately transmitted, so that the video data are ensured to be transmitted to the host-side equipment orderly and timely after the multi-code stream encoding is finished. For example, in fig. 3, the transmission sequence of each path of code stream data is code stream data 1, code stream data 2, and code stream data 3, but the sequence of actually transmitting the code stream data may be transmitted according to the sequence of completing the encoding of the code stream data, and if the code stream data 2 completes the encoding before the code stream data 1 and the code stream data 3 within time t1, the transmission sequence may be code stream data 2, code stream data 1, and code stream data 3.
In combination with the above further, specific or preferred scheme, the execution process of the multi-stream processing method provided in this embodiment is as follows:
step T0 is executed: simultaneously encoding video data according to different encoding parameters to form multi-path code stream data, and respectively adding corresponding unique identifiers at the heads of the multi-path code stream data in the encoding process; step T1 is executed: multiplexing the same UVC transmission channel, and alternately transmitting the encoded multi-channel code stream data until the encoded multi-code stream data are all output.
The method provided by the embodiment outputs the multi-path code stream data by multiplexing the same UVC transmission channel, solves the problem of limitation of the existing UVC protocol on multi-code stream data transmission, and simultaneously avoids transcoding and recoding the code stream data by the host end equipment because the single code stream but the multi-path code stream data is output to the host end equipment, thereby reducing the CPU consumption of the host end equipment.
Example 2
This embodiment provides a multi-code stream output method, as shown in fig. 4, the method includes the steps of:
s1: receiving continuous data input from the same UVC transmission channel;
s2: splitting the continuous data received in the step S1 into multi-path code stream data;
s3: and outputting the code stream data obtained by splitting the at least one or more paths of step S2 to one or more other devices or modules.
Step S1 is executed: receiving continuous data input from the same UVC transmission channel, wherein the continuous data transmission on the UVC transmission channel can be completed only by one endpoint resource without additional endpoint resources; step S2 is executed: splitting the continuous data into multi-path code stream data, wherein the received continuous data is composed of the multi-path code stream data by default, and the step S2 can only split the continuous data into the multi-path code stream data; specifically, the received continuous data is composed of not only multiple paths of code stream data but also multiple paths of code stream data alternately transmitted in the same UVC transmission channel, and thus it can be seen that the multiple paths of code stream data transmitted in the UVC transmission channel in step S1 are alternately transmitted. Step S3 is executed: outputting at least one path or multiple paths of code stream data to an opposite terminal or other equipment/modules according to the requirements of the opposite terminal or other equipment/modules; for example, when it is detected that a network environment of a certain peer or other device/module changes, such as a network becomes poor, the code stream output to the peer or other device/module also needs to be correspondingly switched to a smaller one-way code stream data. Specifically, the opposite terminal or other devices/modules are any devices, modules or apparatuses capable of receiving code stream data, and electronic devices such as a mobile phone and a tablet computer are also included.
In the execution process of the method, continuous data does not need to be transcoded and encoded for the transmission of the multi-channel code stream data, only the continuous data needs to be split, and the split continuous data can be correspondingly output to an opposite terminal or other equipment/modules, so that any equipment or device executing the method can greatly reduce the resource loss of an internal CPU (central processing unit). Specifically, in a video conference or live broadcast scene, after receiving multi-code stream data input by the same UVC transmission channel, corresponding code stream data is transmitted to an opposite terminal or other equipment/modules according to the network environment or actual requirements of the opposite terminal or other equipment/modules, so that the actual network environment or actual requirements are met, clear and smooth video or live broadcast content is provided for the opposite terminal or other equipment/modules, and meanwhile, the CPU (central processing unit) loss of equipment or a device executing the method cannot be overlarge.
Specifically, the specific execution procedure of step S2 is as follows: the unique identifier in each path of code stream data is identified in the continuous data received in step S1, and the continuous data is split into multiple paths of code stream data according to the unique identifier in each path of code stream data.
In the process of splitting the continuous data, the unique identification of each path of code stream data in the continuous data is required to be relied on, the multiple paths of code stream data in the continuous data respectively comprise at least one unique identification, the position of each path of code stream data in the continuous data can be marked by identifying the unique identification of each path of code stream data, so that the continuous data can be split into the independent code stream data of each path according to the identification, and then one or more paths of code stream data are output to other equipment/modules according to the requirements of an opposite terminal or other equipment/modules.
As a preferred scheme, the position of the unique identifier in each path of code stream data is arranged at the head of each path of code stream data, and different paths of code stream data can be immediately identified through the unique identifier at the head of each path of code stream data (the head of a data packet) after continuous data are received, so that the code stream data can be rapidly split and correspondingly output to an opposite terminal or other devices/modules. The unique identification is arranged at the head of the code stream data, so that the identification speed and the splitting speed can be increased, the whole transmission process is smoother, and the time delay is lower.
Example 3
The present embodiment provides a multi-code stream processing apparatus, where the processing apparatus is configured to assist a UVC apparatus to perform data transmission, and the processing apparatus itself may also be a UVC apparatus, where a module in the processing apparatus assists the data transmission of the apparatus itself, and the UVC apparatus refers to any apparatus/device/module that performs data transmission based on a UVC protocol.
The processing equipment comprises a multiplexing output module which is used for multiplexing the same UVC transmission channel to output multi-path code stream data. The multiplexing output module multiplexes the multi-channel code stream data to the same UVC transmission channel for output, and the problem of limitation of the existing UVC protocol 1.5 version on multi-code stream data transmission is solved. Specifically, the multiplexed output module outputs the multi-path code stream data to the host device, which refers to any device/apparatus/module capable of receiving the code stream data. The host end equipment can directly transmit the code stream data to the opposite end or other equipment/modules after receiving the multi-code stream data, the multi-path code stream data directly output by the host end equipment can adapt to the actual network bandwidth conditions of each opposite end or other equipment/modules, and particularly under the video conference or video live broadcast scene, the opposite end or other equipment/modules can watch clear and smooth real-time content as far as possible. Meanwhile, the multiplexing output module outputs the code stream data to the host end equipment instead of a single code stream, so that transcoding and recoding of the host end equipment on the code stream data are avoided, and the CPU consumption of the host end equipment is reduced.
Furthermore, the processing device further comprises a parallel coding module for coding the video data simultaneously to form multiple paths of code stream data. The multi-path code stream data output by the multiplexing output module is the multi-path code stream data coded by the parallel coding module.
Specifically, the specific process of the parallel encoding module for simultaneously encoding the multi-path code stream data is as follows: and controlling the simultaneous encoding of the video data according to different encoding parameters to form multiple paths of code stream data, and adding corresponding unique identifiers in the multiple paths of code stream data respectively.
Preferably, the unique identifier added to each flow of code stream data is located at the head of the flow of code stream data.
Combining the specific and preferred working process of the parallel coding module, the specific process of the multiplexing output module for transmitting the multi-path code stream data is as follows: multiplexing the same UVC transmission channel, and alternately transmitting the encoded multi-channel code stream data.
The specific working process of the multi-code stream processing device provided by this embodiment is as follows: the parallel coding module is used for simultaneously coding the video data according to different coding parameter controls to form multi-path code stream data, and corresponding unique identifiers are respectively added at the heads of the multi-path code stream data in the coding process; and the multiplexing transmission module multiplexes the same UVC transmission channel and alternately transmits the multi-path code stream data coded by the parallel coding module until the multi-path code stream data coded by the parallel coding module is output.
The multi-code stream processing device provided by the embodiment outputs multi-path code stream data by multiplexing the same UVC transmission channel, so that the problem of limitation of the existing UVC protocol on multi-code stream data transmission is solved, and simultaneously, because the multi-path code stream data is output, namely the single code stream is not output to the host end device, transcoding and recoding of the host end device on the code stream data are avoided, and the CPU consumption of the host end device is reduced.
Example 4
The present embodiment provides a multi-code stream processing device, which is used to assist a host device to perform data transmission, but the processing device itself may also be a host device, and a module in the processing device assists data transmission of the host device, where the host device refers to any device/apparatus/module capable of receiving code stream data, and generally, a central processing unit CPU is disposed in the host device.
The processing equipment comprises a splitting output module which is used for receiving continuous data input from the same UVC transmission channel, splitting the continuous data into multiple paths of code stream data and outputting at least one path or multiple paths of code stream data to one or more other equipment or modules.
After receiving the continuous data, the splitting output module only needs to split the continuous data, and the split data can be output to an opposite terminal or other equipment/modules, so that the resource loss of a CPU in the processing equipment can be greatly reduced. After receiving multi-code stream data input by the same UVC transmission channel, the splitting output module transmits the corresponding code stream data to the opposite terminal or other equipment/modules according to the network environment or actual requirements of the opposite terminal or other equipment/modules, can adapt to the network environment or actual requirements of each opposite terminal or other equipment/modules, particularly provides clear and smooth video or live broadcast content for the opposite terminal or other equipment/modules in a video conference or live broadcast scene, and meanwhile does not cause the CPU loss of processing equipment to be overlarge.
Specifically, the specific execution steps of the splitting output module in executing the "splitting continuous data into multiple paths of code stream data" are as follows: and identifying the unique identifier in each path of code stream data in the received continuous data, and splitting the continuous data into multi-path code stream data according to the unique identifier in each path of code stream data.
The splitting output module needs to rely on the unique identifier of each path of code stream data in the continuous data in the process of splitting the continuous data, each path of code stream data in the continuous data comprises at least one unique identifier, the splitting output module can mark the position of each path of code stream data in the continuous data by identifying the unique identifier of each path of code stream data, so that the continuous data can be split into each path of code stream data according to the identifier, and one or more paths of code stream data are output to other equipment/modules according to the requirements of an opposite terminal or other equipment/modules.
As a preferred scheme, the position of the unique identifier in each path of code stream data is set at the head of each path of code stream data, and the splitting output module can immediately identify different paths of code stream data through the unique identifier of the head (the head of the data packet) of each path of code stream data after receiving the continuous data, so as to rapidly split the data and correspondingly output the data to the opposite terminal or other devices/modules. The unique identification is arranged at the head of the code stream data, so that the recognition speed and the splitting speed of the split output module can be increased, the whole transmission process is smoother, and the time delay is lower.
Example 5
The present embodiment provides a USB image pickup apparatus that executes the multi-stream processing method provided in embodiment 1 when processing and transmitting multi-stream data.
Example 6
The present embodiment provides a computer-readable storage medium on which a computer program is stored, the computer program implementing the data processing method in embodiment 1 or embodiment 2 when executed by a processor.
It should be understood that the above-mentioned embodiments of the present invention are only examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific embodiments of the present invention. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention claims should be included in the protection scope of the present invention claims.
Claims (11)
1. A method for multi-stream processing, the method comprising: multiplexing the same UVC transmission channel to output multi-path code stream data.
2. The method of claim 1, wherein the video data is encoded simultaneously to form multiple paths of said code stream data before multiplexing the same UVC transmission channel to output multiple paths of code stream data.
3. The multi-stream processing method of claim 2,
the method for encoding the video data simultaneously to form multi-path code stream data specifically comprises the following steps: and simultaneously coding video data according to different coding parameters to form a plurality of paths of code stream data, and adding a corresponding unique identifier in each path of code stream data.
4. The multi-stream processing method according to claim 3, wherein the position of the unique identifier added to each stream of the stream data is at the head of the stream data of the stream.
5. The multi-stream processing method according to any one of claims 2 to 4,
multiplexing the same UVC transmission channel to output multi-path code stream data, which specifically comprises the following steps: multiplexing the same UVC transmission channel, and alternately transmitting the encoded multi-channel code stream data.
6. A method for multi-stream output, comprising the steps of: receiving continuous data input from the same UVC transmission channel, splitting the continuous data into multiple paths of code stream data, and outputting at least one path or multiple paths of the code stream data to one or more other devices or modules.
7. The multi-stream output method according to claim 6, wherein the splitting of the continuous data into multiple streams of stream data is specifically:
and identifying the unique identifier in each path of code stream data in the continuous data, and splitting the continuous data into a plurality of paths of code stream data according to the unique identifier in each path of code stream data.
8. A multi-stream processing apparatus, comprising: and the multiplexing output module is used for multiplexing the same UVC transmission channel to output multi-path code stream data.
9. A multi-stream processing apparatus, comprising: the split output module is used for receiving continuous data input from the same UVC transmission channel, splitting the continuous data into multiple paths of code stream data, and outputting at least one path or multiple paths of the code stream data to one or more other devices or modules.
10. A USB camera device for processing multi-stream data, wherein the USB camera device executes the multi-stream processing method according to any one of claims 1 to 5 when processing the multi-stream data.
11. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data processing method of any one of claims 1 to 5 or 6 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011126892.1A CN112492348A (en) | 2020-10-20 | 2020-10-20 | Multi-code stream processing and output method, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011126892.1A CN112492348A (en) | 2020-10-20 | 2020-10-20 | Multi-code stream processing and output method, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112492348A true CN112492348A (en) | 2021-03-12 |
Family
ID=74926875
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011126892.1A Pending CN112492348A (en) | 2020-10-20 | 2020-10-20 | Multi-code stream processing and output method, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112492348A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100185795A1 (en) * | 2009-01-20 | 2010-07-22 | Quanta Computer Inc. | Digital video/audio capture device and method thereof |
| CN102088622A (en) * | 2011-01-13 | 2011-06-08 | 成都索贝数码科技股份有限公司 | Multimedia information processing terminal |
| CN102301730A (en) * | 2011-07-18 | 2011-12-28 | 华为技术有限公司 | Method, device and system for transmitting and processing multichannel AV |
| CN103763579A (en) * | 2014-01-22 | 2014-04-30 | 深圳国微技术有限公司 | Method for recognizing multichannel transmission streams based on USB interface |
| CN106534857A (en) * | 2016-11-02 | 2017-03-22 | 深圳市维海德技术股份有限公司 | Code stream transmission method and camera based on USB |
| CN106708763A (en) * | 2016-08-31 | 2017-05-24 | 北京亮亮视野科技有限公司 | Head-mounted display device and data transmission system of intelligent host |
| CN110958434A (en) * | 2019-12-13 | 2020-04-03 | 厦门亿联网络技术股份有限公司 | Multi-image splicing method and system and computer readable storage medium |
-
2020
- 2020-10-20 CN CN202011126892.1A patent/CN112492348A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100185795A1 (en) * | 2009-01-20 | 2010-07-22 | Quanta Computer Inc. | Digital video/audio capture device and method thereof |
| CN102088622A (en) * | 2011-01-13 | 2011-06-08 | 成都索贝数码科技股份有限公司 | Multimedia information processing terminal |
| CN102301730A (en) * | 2011-07-18 | 2011-12-28 | 华为技术有限公司 | Method, device and system for transmitting and processing multichannel AV |
| CN103763579A (en) * | 2014-01-22 | 2014-04-30 | 深圳国微技术有限公司 | Method for recognizing multichannel transmission streams based on USB interface |
| CN106708763A (en) * | 2016-08-31 | 2017-05-24 | 北京亮亮视野科技有限公司 | Head-mounted display device and data transmission system of intelligent host |
| CN106534857A (en) * | 2016-11-02 | 2017-03-22 | 深圳市维海德技术股份有限公司 | Code stream transmission method and camera based on USB |
| CN110958434A (en) * | 2019-12-13 | 2020-04-03 | 厦门亿联网络技术股份有限公司 | Multi-image splicing method and system and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2737728C (en) | Low latency video encoder | |
| US11856329B2 (en) | Dynamic advertisement stream replacement | |
| US10911763B2 (en) | System and method for electronic data communication | |
| KR102324326B1 (en) | Streaming multiple encodings encoded using different encoding parameters | |
| CN104735470B (en) | A kind of streaming media data transmission method and device | |
| CN109150823B (en) | Raw video transmission and reception using scalable frame rate | |
| US10616383B2 (en) | System and method for electronic data communication | |
| US10594977B2 (en) | System and method for electronic data communication | |
| US20100064054A1 (en) | Remote fast forward and rewind functionality for client devices | |
| CN107147916B (en) | Method for transmitting H.265 coding video data on transmission layer | |
| CN101742289B (en) | Method, system and device for compressing video code stream | |
| CN105635636A (en) | Video conference system and method for realizing transmission control of video image | |
| US10523895B2 (en) | System and method for electronic data communication | |
| US20090259764A1 (en) | Intro outro merger with bit rate variation support | |
| KR20110058841A (en) | Moving-picture image data-distribution method | |
| FI105634B (en) | Procedure for transferring video images, data transfer systems and multimedia data terminal | |
| TW201138464A (en) | Network device, information processing apparatus, stream switching method, information processing method, program, and content distribution system | |
| CN112492348A (en) | Multi-code stream processing and output method, equipment and storage medium | |
| CN108124183B (en) | Method for synchronously acquiring video and audio to perform one-to-many video and audio streaming | |
| Lei et al. | Video transcoding gateway for wireless video access | |
| CN111049624B (en) | High fault tolerance feedback-free link image transmission method and system based on sliding window | |
| CN201352834Y (en) | High-definition conference system based on digital television HDTV platform | |
| CN101521790A (en) | Methods for realizing high definition conference system based on digital television | |
| CN115085790B (en) | Channel environment self-adaption method and narrow bandwidth conference system | |
| CN101814969A (en) | Method and system for reducing bit stream and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210312 |