WO2022198605A1

WO2022198605A1 - Data stream transmission method and device

Info

Publication number: WO2022198605A1
Application number: PCT/CN2021/083112
Authority: WO
Inventors: 陶岚; 江小华; 李林炳
Original assignee: 华为技术有限公司
Priority date: 2021-03-25
Filing date: 2021-03-25
Publication date: 2022-09-29
Also published as: CN116711298A

Abstract

The present application relates to the field of video stream transmission, and discloses a data stream transmission method and device. The present application solves the problem in the prior art of high transmission delay of a video stream. The method is performed by a video acquisition device, the video acquisition device is connected to a video display device by means a first line and a second line separately, and the first line is used for transmitting control instructions of the video acquisition device and the video display device. The method comprises: the video acquisition device generates a plurality of application video streams according to an original video stream acquired by the video acquisition device, and splices the plurality of application video streams to obtain a data stream to be transmitted; and then, the video acquisition device sends said data stream to the video display device by means of the second line. The transmission bandwidth required by said data stream is less than or equal to the maximum transmission bandwidth of the second line.

Description

A data stream transmission method and device

technical field

The present application relates to the field of data transmission, and in particular, to a data stream transmission method and device.

Background technique

With the development of smart services, TVs can use artificial intelligence (AI) technology to provide users with various types of services, such as multi-party video calls, human perception, gesture control, sports fitness, and remote home monitoring on the TV. Usually, in the above business, the camera transmits the control command and the original video stream captured by the camera to the TV through the universal serial bus (USB), and the TV converts the original video stream according to the control command to obtain the resolution Different multiple application video streams, and further, the TV displays the multiple application video streams, and the user can view various video information obtained according to the original video stream through the TV.

The conversion of the original video stream will occupy most of the computing resources of the TV main chip. Due to the low performance of the TV main chip, it is easy to cause the playback of the application video stream to freeze, which reduces the user's video viewing experience. In the current technical solution, in order to prevent the playback of the video stream from being stuck, the camera converts the original video stream collected by the camera to obtain multiple application video streams, and the camera transmits the multiple application video streams to the TV through USB, reducing the number of application video streams. The computing resources occupied by the TV main chip to convert the raw video stream. However, the transmission capacity of USB is small, which results in high latency when USB transmits multiple video streams.

Therefore, how to reduce the transmission delay of the video stream is an urgent problem to be solved at present.

SUMMARY OF THE INVENTION

The present application provides a data stream transmission method and device, which solve the problem of high transmission delay of video streams.

In order to achieve the above purpose, the present application adopts the following technical solutions.

In a first aspect, the present application provides a data stream transmission method. The method is executed by a video capture device, or the method can be applied to a terminal device that can support the implementation of the method. For example, the terminal device includes a chip system. In a possible design, the video capture device is connected to the video display device through a first line and a second line respectively, and the first line is used to transmit control instructions of the video capture device and the video display device. The method includes: a video capture device A plurality of application video streams are generated according to the original video streams collected by the video capture device, and the plurality of application video streams are spliced to obtain the data stream to be transmitted, and then the video capture device sends the data stream to be transmitted to the video display device through the second line. Wherein, the transmission bandwidth required by the data stream to be transmitted is less than or equal to the maximum transmission bandwidth of the second line.

In this way, the video capture device transmits the control command through the first line, and sends the data stream to be transmitted to the video display device through the second line, which realizes the separate transmission of the control command and the data stream, and avoids using the same line to transmit the control command and the data stream. When the control instruction occupies the bandwidth of the line for transmitting the data stream, the problem of high transmission delay of the data stream is caused. In the data stream transmission method provided by the embodiment of the present application, the data stream is transmitted through a dedicated second line, which reduces the transmission delay for transmitting the data stream to be transmitted, and improves the transmission rate for transmitting the data stream to be transmitted; In the process of sending the data stream to be transmitted to the video display device through the second line, the control command sent by the video display device can also be received through the first line. The video capture device sends the control command, which effectively reduces the transmission delay of the control command. In addition, compared with the prior art, which can only transmit the video stream separately (after the transmission of one video stream is completed, another video stream is transmitted), in the data stream transmission method used in the embodiment of the present application, the video capture device can The multiple application video streams are spliced to obtain the data stream to be transmitted, and the to-be-transmitted data stream is put into the second line for transmission, so that the video capture device does not need to transmit each application video stream separately, and the transmission delay of the video stream is reduced. In addition, since the transmission bandwidth required for the data stream to be transmitted is less than or equal to the maximum transmission bandwidth of the second line, the video capture device can transmit the data stream to be transmitted according to the maximum transmission bandwidth of the second line, avoiding the transmission of the second line. Transmission delay caused by insufficient bandwidth.

Wherein, the above-mentioned maximum transmission bandwidth of the second line indicates the maximum data amount of the video stream that can be transmitted by the second line in a unit time. Here, take the video stream needing to transmit N frames of images with the same resolution in a unit time as an example, the maximum data amount = the resolution of a single frame of image × N, N is a positive integer, and the resolution of the image can be expressed in pixels , for example, the resolution of the image is 1080P (1920×1080).

The above-mentioned transmission bandwidth required for the data stream to be transmitted indicates the amount of data required to be transmitted for all application video streams and description information in the data stream to be transmitted in a unit time, and the description information can be used to describe the resolution information of the application video stream. , exposure information, white balance information or focus information, etc.

In a possible example, the multiple application video streams included in the data stream to be transmitted may be different types of video streams. For example, the multiple application video streams include at least one video stream among preview streams, photographing streams, depth data streams, face streams, object detection streams, monitoring streams, time-of-flight ranging streams, or gesture control streams.

The user can use the video display device to view different video information of the original video stream, for example, the user can view the preview stream and the photo stream in the video call, and view the face recognition information obtained according to the original video stream during the video call (such as Classify the face shape of the face), thereby improving the user's experience of watching videos.

In an optional implementation manner, the first line is a USB, and the second line is a high definition multimedia interface (high definition multimedia interface, HDMI) line.

Compared with the prior art, the same transmission line (USB) is multiplexed between the camera and the TV to transmit the video stream and the control command. In the process of sending the video stream to the TV through the USB, the TV cannot send the control command to the camera through the USB, resulting in a higher transmission delay of the control command in the opposite direction to the transmission direction of the video stream. Using the data stream transmission method provided by the embodiment of the present application, the video capture device transmits the control command through USB, and transmits the to-be-transmitted data stream through HDMI, which realizes the separate transmission of the data stream and the control command, and the video display device does not need to transmit the data stream to be transmitted. Only after the transmission process is over, the control command is sent to the video capture device, that is to say, in the process of the video capture device sending the data stream to be transmitted to the video display device through HDMI, the video display device can send the control command to the video capture device through USB. command, reducing the transmission delay of the control command. In addition, the transmission bandwidth of USB in the prior art can generally only reach 400MB/s (megabytes per second), while the HDMI cable used for transmitting data streams in this application has a relatively large data stream transmission capacity, such as 10.2 gigabits Gpbs/second (giga bits per second, Gpbs)=1275MB/s, therefore, the data stream transmission method provided by the embodiment of the present application uses HDMI to replace the USB in the prior art, which can improve the transmission speed of the data stream and reduce the transmission of the data stream. time delay.

In another optional implementation manner, if the second line is an HDMI line, the format of the application video stream may be the YUV format. Wherein, the Y information indicates the brightness (Luminance) of each frame image in the application video stream, and the U and V information indicates the color information (Chrominance) of each frame image in the application video stream. Since the video stream in the YUV format is a lossless video stream, and the transmission bandwidth of the USB in the prior art is relatively small, the transmission delay of the YUV format is relatively high. In the data stream transmission method provided by the embodiment of the present application, the second line is an HDMI line, and the transmission bandwidth of the HDMI line is larger than that of the USB, which can reduce the transmission delay of the video capture device sending the data stream to be transmitted to the video display device.

In addition, in the process of transmitting multiple application video streams by the camera in the prior art, the camera will encode the multiple application video streams, thereby compressing the multiple application video streams to obtain the data stream to be transmitted. Including encoding information, the encoding information indicates the encoding method of the application video stream by the camera; since the video stream in the data stream to be transmitted is compressed, this will cause the video stream included in the data stream to be transmitted. The video stream has losses, and further, the video stream played on the TV is not completely consistent with the above-mentioned application video stream, which reduces the playback effect of the video stream. However, in the data stream transmission method provided by the embodiment of the present application, when the second line is an HDMI cable and the format of the application video stream is the YUV format, the video capture device does not need to encode the application video stream, and thus does not generate Encoding information, which reduces the fields used to describe the format of the application video stream in the data stream to be transmitted, reduces the redundancy of the data stream to be transmitted, and reduces the amount of data that the video capture device sends to the video display device using HDMI to the data stream to be transmitted , and further, the transmission speed of the to-be-transmitted data stream is increased, and the transmission delay of the to-be-transmitted data stream is reduced. In addition, since the video capture device in the embodiment of the present application can use the dedicated second line to transmit the data stream in the YUV format, there is no difference between the video stream played by the video display device and the application video stream generated by the video capture device, which improves the The video display device plays the effect of the video stream, thereby improving the user's experience of watching the video.

In another optional implementation manner, the video capture device may also receive a capture command sent by the video display device through the first line, and capture the original video stream according to the capture command; the video capture device also processes the above-mentioned capture command according to the capture command. The original video stream gets multiple application video streams. For example, the video capture device may generate multiple application video streams based on multiple capture instructions. For example, the video display device generates multiple capture commands according to the user's gesture information, and the multiple capture commands include face recognition commands and preview commands. , the video display device sends the multiple video capture instructions to the video capture device, so that the video capture device generates a face recognition stream according to the face recognition command, generates a preview stream according to the preview command, and the video capture device combines the preview stream with the human After the face recognition streams are spliced to obtain the to-be-transmitted data stream, the video acquisition device sends the to-be-transmitted data stream to the video display device, and the user can view the above-mentioned preview stream and face recognition stream through the video display device.

In an optional implementation manner, the data stream to be transmitted may include channel information, the channel information includes the number of channels in the second line, and bandwidth information of each channel, and the channel information also includes applications in the data stream to be transmitted The number of video streams, and which channel each application video stream is on. In order to facilitate the video display device to determine the information of each channel in the to-be-transmitted data stream and the application video stream based on the channel information in the to-be-transmitted data stream after receiving the to-be-transmitted data stream, it avoids that the video display device only relies on each application video The stream information of the stream is divided into the data stream to be transmitted, which improves the speed of parsing the data stream to be transmitted by the video display device. It should be understood that in this embodiment of the present application, the bandwidth information included in the channel information may be characterized by the maximum resolution of the video stream that can be transmitted by the channel. Exemplarily, the bandwidth information included in the channel information is 3840*540 , the bandwidth information of channel 2 is 3840*540, and the bandwidth information of channel 3 is 3840*240, respectively indicating that the resolution of the video stream that can be transmitted by channel 1 can be up to 3840*540, and the resolution of the video stream that can be transmitted by channel 2. The maximum can be 3840*540, and the maximum resolution of the video stream that can be transmitted by channel 3 can be 3840*240.

In another optional implementation manner, the above-mentioned channel information includes resolution information of each application video stream, so that after the video display device acquires the channel information, it can be based on the resolution information of each application video stream as The application allocates computing resources for the playback of the video stream, which can improve the utilization rate of computing resources of the video display device and avoid the video display device from being stuck; in the process of splitting the data stream to be transmitted by the video display device, the video display device can not only The channel where each application video stream is located determines the position of the application video stream, and the channel information including the resolution information of the application video stream can also be used to determine the width and height information of the application video stream, which improves the resolution of the video display device in parsing the data stream to be transmitted. accuracy.

In another optional implementation manner, the above-mentioned channel information may be set in the header of the data stream to be transmitted. The video acquisition device sets the channel information in the header of the data stream to be transmitted, and sends the data stream to be transmitted to the video display device through the second line. In the prior art, the video display device can only split the data stream to be transmitted according to the stream information of each application video stream, which requires a lot of computing resources, resulting in a freeze when the video display device plays the video stream, and the present In the scheme, after the video display device receives the data stream to be transmitted, since the channel information is in the header of the data stream to be transmitted, the video display device can preferentially obtain the channel information in the data stream to be transmitted, and determine based on the channel information. The information of each channel in the data stream to be transmitted and the application video stream improves the speed of parsing the data stream to be transmitted by the video display device.

In another optional implementation manner, the data stream to be transmitted may further include stream information, where the stream information indicates attribute information of each application video stream in the data stream to be transmitted, and the attribute information includes exposure information of the application video stream , at least one of focus information or white balance information. The video capture device sets the stream information in the data stream to be transmitted, and sends the data stream to be transmitted to the video display device through the second line. After the video display device receives the data stream to be transmitted, the video display device can determine the number, width and height of each channel in the data stream to be transmitted, and the number, location and resolution of the application video stream according to the channel information in the data stream to be transmitted. Further, the video display device can also determine the attribute information of the application video stream according to the stream information in the data stream to be transmitted, thereby realizing the identification of different types of application video streams, and further improving the ability of the video display device to parse the data stream to be transmitted. accuracy.

In another optional implementation manner, the above-mentioned multiple application video streams include a first application video stream and a second application video stream, and the resolution of the first application video stream and the resolution of the second application video stream are different. The embodiment of the present application can transmit multiple video streams with different resolutions at the same time, and the video capture device generates multiple application video streams by using the original video stream, which can avoid the video display device from converting the original video stream and release the computing resources of the video display device. , to prevent the video display device from being stuck when playing the video stream.

In another optional implementation manner, the data stream to be transmitted further includes a data area, the data area includes multiple channels, and the video capture device splices multiple application video streams to obtain the data stream to be transmitted, including: the video capture device combines multiple The application video streams are respectively set in their corresponding target channels, wherein the target channel corresponding to the application video stream is a channel whose maximum transmission bandwidth is greater than or equal to the transmission bandwidth required by the application video stream. According to the bandwidth information of each channel in the data area and the resolution information of each application video stream in the channel information, the video capture device takes the channel whose maximum transmission bandwidth is greater than or equal to the transmission bandwidth required by the application video stream as the target channel of the application video stream , and further, the video capture device sets the application video stream in the target channel corresponding to the application video stream, so as to realize the transmission of the application video stream based on the transmission capability of the channel. That is to say, each application video stream can be put into the target channel whose transmission bandwidth is greater than or equal to the transmission bandwidth required by the application video stream, preventing the video capture device from putting the application video stream with larger resolution into the bandwidth. In the channel with smaller information, the transmission delay caused by insufficient transmission bandwidth of the channel is reduced.

In another optional implementation manner, the data stream to be transmitted includes a data area, the data area includes multiple channels, and the video capture device splices multiple application video streams to obtain the data stream to be transmitted, including: if the first application video stream The required transmission bandwidth is greater than the transmission bandwidth of any one of the multiple channels, and is less than or equal to the maximum transmission bandwidth of the second line, then the video capture device uses the first application video stream as the first data stream to be transmitted, and the third An application video stream is any one of the multiple application video streams; and, if the transmission bandwidth required by the remaining data streams matches the transmission bandwidth possessed by the multiple channels, the video capture device splices the multiple application videos in the remaining data streams The second to-be-transmitted data stream is obtained from the stream, wherein the remaining data streams include other application video streams except the first application video stream among the multiple application video streams obtained according to the original video stream. The video acquisition device sending the data stream to be transmitted to the video display device through the second line includes: the video acquisition device alternately sends the first data stream to be transmitted and the second data stream to be transmitted to the video display device through the second circuit.

In the prior art, the camera transmits the application video stream with a larger resolution to the TV, and then sends other application video streams to the TV, resulting in a high transmission delay of the other application video streams. According to the data stream transmission method, the video capture device can use the first application video stream as the first data stream to be transmitted, and the first application video stream is a plurality of application video streams. The required transmission bandwidth is greater than any one of the multiple channels. The application video stream of the transmission bandwidth, and the remaining data stream that matches the transmission bandwidth of the multiple channels is used as the second data stream to be transmitted, and then the video capture device alternately sends the first data stream to be transmitted through the second line. and the second data stream to be transmitted, so as to realize the time-sharing and alternate transmission of a single application video stream with a larger resolution and multiple application video streams with a smaller resolution. In other words, using the data stream transmission method provided by the embodiments of the present application, the video capture device does not need to send the second data stream to be transmitted to the video display device after the transmission of the first data stream to be transmitted is completed, which avoids the need for separate video streams. The disadvantage of transmission (after the transmission of one video stream is completed, another video stream is transmitted) reduces the transmission delay of the video stream.

In a second aspect, the present application provides a data stream transmission method, the method is executed by a video display device, or the method can be applied to a terminal device that can support the implementation of the method, for example, the terminal device includes a chip system. In a possible design, the video capture device is connected to the video display device through a first line and a second line respectively, and the first line is used to transmit control instructions of the video capture device and the video display device. The method includes: a video display device The data stream to be transmitted sent by the video capture device is received through the second line, and the video display device also processes the data stream to be transmitted and plays at least one application video stream. The above-mentioned data stream to be transmitted is obtained by splicing multiple application video streams, and the transmission bandwidth required for the data stream to be transmitted is less than or equal to the maximum transmission bandwidth of the second line. In this way, the video display device transmits the control command through the first line, and receives the data stream to be transmitted sent by the video capture device through the second line, so as to realize the separate transmission of the control command and the data stream, and avoid using the same line to transmit the control command and the data stream. When the control instruction occupies the bandwidth of the line for transmitting the data stream, the problem of high transmission delay of the data stream is caused. In the data stream transmission method provided by the embodiment of the present application, the data stream is transmitted through a dedicated second line, which reduces the transmission delay of transmitting the data stream to be transmitted, and improves the transmission rate of transmitting the data stream to be transmitted; During the process of receiving the data stream to be transmitted sent by the video capture device through the second line, the control command sent by the video capture device can also be received through the first line. Sending a control command to the video capture device effectively reduces the transmission delay of the control command. In addition, compared with the prior art, which can only transmit the video stream separately (after the transmission of one video stream is completed, another video stream is transmitted), in the data stream transmission method used in the embodiment of the present application, the video capture device can The multiple application video streams are spliced to obtain the data stream to be transmitted, and the to-be-transmitted data stream is put into the second line for transmission, so that the video capture device does not need to transmit each application video stream separately, and the transmission delay of the video stream is reduced. In addition, since the transmission bandwidth required for the data stream to be transmitted is less than or equal to the maximum transmission bandwidth of the second line, the video display device can transmit the data stream to be transmitted according to the maximum transmission bandwidth of the second line, avoiding the need for the transmission of the second line. Transmission delay caused by insufficient bandwidth.

In addition, since the data stream to be transmitted includes multiple application video streams, after the video display device receives the data stream to be transmitted, the video display device can process and play the data stream to be transmitted, and the user can view the original video through the video display device. Various video information of the stream, such as the user can view the preview stream and photo stream in the video call, and view the face recognition information obtained from the original video stream during the video call (such as classifying the face shape of the face) , which improves the user's viewing experience.

In a possible example, the data stream to be transmitted includes channel information, the channel information includes the number of channels in the second line, and the bandwidth information of each channel, and the channel information further includes the number of application video streams in the data stream to be transmitted , and the channel on which each application video stream is located. After the video display device receives the to-be-transmitted data stream, the video display device can determine each channel in the to-be-transmitted data stream and the information of the application video stream based on the channel information in the to-be-transmitted data stream, so as to prevent the video display device from The stream information of the application video stream is split into the data stream to be transmitted, which improves the speed of parsing the data stream to be transmitted by the video display device.

In another possible example, the channel information includes resolution information of each application video stream. After the video display device obtains the channel information, it can allocate computing resources for the playback of the application video stream according to the resolution information of each application video stream, which can improve the utilization rate of the computing resources of the video display device and avoid the occurrence of the video display device. When the video display device splits the data stream to be transmitted, the video display device can not only determine the position of the application video stream according to the channel where each application video stream is located, but also use the channel information including the application video stream. The resolution information determines the width and height information of the application video stream, which improves the accuracy of parsing the data stream to be transmitted by the video display device.

In another possible example, the channel information is set in the header of the data stream to be transmitted. In the prior art, the video display device can only split the data stream to be transmitted according to the stream information of each application video stream, which requires a lot of computing resources, resulting in the video display device being stuck in playing the video stream. In the scheme, after the video display device receives the data stream to be transmitted, since the channel information is in the header of the data stream to be transmitted, the video display device can preferentially obtain the channel information in the data stream to be transmitted, and determine based on the channel information. The information of each channel in the data stream to be transmitted and the application video stream improves the speed of parsing the data stream to be transmitted by the video display device.

In an optional implementation manner, the video display device processes the data stream to be transmitted and plays at least one application video stream, including: the video display device splits the data stream to be transmitted according to the channel information to obtain all the application video streams; video display The device selects at least one application video stream among all the application video streams to play. After the video display device receives the data stream to be transmitted, the video display device can determine the number, width and height of each channel in the data stream to be transmitted, and the number, location and resolution of the application video stream according to the channel information in the data stream to be transmitted. Further, the video display device can also determine the attribute information of the application video stream according to the stream information in the data stream to be transmitted, thereby realizing the identification of different types of application video streams, and improving the accuracy of the video display device in parsing the data stream to be transmitted. sex.

In a possible example, the application video stream played by the video display device is the AI data obtained after the video display device splits the data stream to be transmitted according to the channel information, and performs AI processing on the split application video stream. Flow, the AI processing can be, but is not limited to: object detection, face recognition, object classification, color recognition, object counting, etc. For example, the video display device includes a processor, which can be used to perform the operation corresponding to the above AI processing, and display the result of the operation on the display screen of the video display device, so that the user can watch the video through the display screen of the video display device. Various video information to the original video stream, for example, users can view the preview stream and photo stream in the video call, and view the face shape recognition information obtained from the original video stream during the video call (such as the face shape of the face). classification), which improves the user's experience of watching videos.

In another optional implementation manner, before the video display device receives the data stream to be transmitted sent by the video capture device through the second line, the data stream transmission method may further include: the video display device responds to acting on the video display device The operation information is generated, and at least one acquisition instruction is generated. The video display device can send the acquisition instruction to the video acquisition device, so that the video acquisition device generates multiple application video streams according to the acquisition instruction, and sends the data stream to be transmitted obtained by splicing the multiple application video streams to the video through the second line. The display device enables the user to see various video information (such as target detection information, face type classification, color recognition, etc.) of the original video stream through the display screen of the video display device, which improves the user's experience of watching videos.

In a third aspect, an embodiment of the present application provides a video capture device, and the beneficial effects can be found in the description of any aspect of the first aspect, which will not be repeated here. The video capture device has the function of implementing the behavior in the method example of any one of the above-mentioned first aspects. The functions can be implemented by hardware, or can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the video capture device is connected to the video display device through a first line and a second line respectively, the first line is used to transmit control instructions of the video capture device and the video display device, and the video capture device includes: a processing unit , which is used to generate multiple application video streams according to the original video streams collected by the video capture device; the splicing unit is used to splicing multiple application video streams to obtain the data stream to be transmitted, and the transmission bandwidth required for the data stream to be transmitted is less than or equal to the second The maximum transmission bandwidth of the line; the transceiver unit is used to send the data stream to be transmitted to the video display device through the second line.

In an optional implementation manner, the data stream to be transmitted includes channel information, the channel information includes the number of channels in the second line, and the bandwidth information of each channel, and the channel information also includes the application video stream in the data stream to be transmitted , and which channel each application video stream is on.

In another optional implementation manner, the channel information includes resolution information of each application video stream.

In another optional implementation manner, the channel information is set in the header of the data stream to be transmitted.

In another optional implementation manner, the multiple application video streams include a first application video stream and a second application video stream, and the resolution of the first application video stream and the resolution of the second application video stream are different.

In another optional implementation manner, the data stream to be transmitted further includes a data area, and the data area includes multiple channels, and a splicing unit is specifically used to set multiple application video streams in respective corresponding target channels; application The target channel corresponding to the video stream is a channel whose maximum transmission bandwidth is greater than or equal to the transmission bandwidth required by the application video stream.

In another optional implementation manner, the data stream to be transmitted includes a data area, the data area includes multiple channels, and the splicing unit is specifically used if the transmission bandwidth required by the first application video stream is greater than any one of the multiple channels If the transmission bandwidth of the channel is less than or equal to the maximum transmission bandwidth of the second line, the first application video stream is used as the first data stream to be transmitted, and the first application video stream is any one of multiple application video streams; The splicing unit is specifically used for splicing multiple application video streams in the remaining data stream to obtain the second data stream to be transmitted if the transmission bandwidth required by the remaining data stream matches the transmission bandwidth possessed by the multiple channels. Among the multiple application video streams obtained from the video stream, other application video streams except the first application video stream; the transceiver unit is specifically configured to alternately send the first data stream to be transmitted and the second data stream to be transmitted to the video display device through the second line flow.

In another optional implementation manner, the multiple application video streams include at least one of a preview stream, a photo stream, a depth data stream, a face stream, a target detection stream, a monitoring stream, a time-of-flight ranging stream, or a gesture control stream. A sort of.

In another optional implementation manner, the first line is a USB, and the second line is an HDMI line.

In a fourth aspect, an embodiment of the present application provides a video display device, and the beneficial effects can be found in the description of any aspect of the second aspect, which will not be repeated here. The video capture device has the function of implementing the behavior in the method example of any one of the above-mentioned second aspects. The functions can be implemented by hardware, or can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the video display device is connected to the video capture device through a first line and a second line respectively, and the first line is used to transmit control instructions of the video capture device and the video display device, and the video display device includes: a transceiver unit , used to receive the data stream to be transmitted sent by the video capture device through the second line, the data stream to be transmitted is obtained by splicing multiple application video streams, and the transmission bandwidth required for the data stream to be transmitted is less than or equal to the maximum transmission bandwidth of the second line bandwidth; a processing unit, configured to process the data stream to be transmitted to obtain multiple application video streams; and a display unit, configured to play at least one application video stream among the multiple application video streams.

In another optional implementation manner, the processing unit is specifically configured to split the data stream to be transmitted to obtain all the application video streams according to the channel information; the display unit is specifically configured to select at least one application video in all the application video streams stream to play.

In a fifth aspect, an embodiment of the present application provides a terminal device, including a processor and an interface circuit, the processor receives or sends data through the interface circuit, and the processor implements the first aspect and the first aspect through a logic circuit or executing code instructions The method in any one possible implementation of the aspect, or the operation steps of the method in any one of the possible implementations of the second aspect and the second aspect.

In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instruction is stored, and when the computer program or instruction is executed by a terminal device, any one of the first aspect and the first aspect is implemented The method in the possible implementation manner, or the operation steps of the method in any one possible implementation manner of the second aspect and the second aspect.

In a seventh aspect, an embodiment of the present application provides a computer program product, the computing program product includes instructions, and when the computer program product runs on a terminal device or a processor, the terminal device or the processor executes the instructions, so as to realize the first The aspect and the method in any one possible implementation of the first aspect, or the second aspect and the operation steps of the method in any one possible implementation of the second aspect.

In an eighth aspect, an embodiment of the present application provides a chip, including a memory and a processor, where the memory is used to store computer instructions, and the processor is used to call and run the computer instructions from the memory to execute the above-mentioned first aspect and its first aspect The method in any possible implementation manner, or the operation steps of the method in any possible implementation manner of the second aspect and the second aspect.

On the basis of the implementation manners provided by the above aspects, the present application may further combine to provide more implementation manners.

Description of drawings

1 is a schematic diagram of a video communication system provided by the application;

2 is a schematic flowchart of a data stream transmission method provided by the present application;

3 is a schematic diagram of a smart screen provided by the application;

4 is a schematic structural diagram of a data stream to be transmitted provided by the present application;

5 is a schematic flowchart of another data stream transmission method provided by the present application;

6 is a schematic diagram of alternate transmission of data streams provided by the present application;

7 is a schematic diagram of a data stream transmission system provided by the present application;

FIG. 8 is a schematic structural diagram of a terminal device provided by the present application.

Detailed ways

The terms "first", "second" and "third" in the description and claims of the present application and the above drawings are used to distinguish different objects, rather than to limit a specific order.

In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations or illustrations. Any embodiments or designs described in the embodiments of the present application as "exemplary" or "such as" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present the related concepts in a specific manner.

For the sake of clarity and conciseness in the description of the following embodiments, a brief introduction of related technologies is first given.

In the current technical solution, the camera transmits the video stream between the camera and the TV through the USB, and the transmission capability of the USB is relatively small, resulting in a relatively high delay when the USB transmits the video stream. For example, the transmission bandwidth required for a video stream of a three-dimensional (3-dimension, 3D) image is about 442MB/s, while the transmission bandwidth of USB 3.0 can generally only reach 400MB/s. delay.

In order to solve the above problems, the present application provides a data stream transmission method. The video capture device is connected to the video display device through a first line and a second line respectively, and the first line is used to transmit the control instructions of the video capture device and the video display device. The method includes: a video capture device is used to capture an original video stream, and generate a plurality of application video streams according to the original video stream, and splicing the plurality of application video streams to obtain a data stream to be transmitted, and further, the video capture device passes the second The line sends the data stream to be transmitted to the video display device; wherein, the transmission bandwidth required by the data stream to be transmitted is less than or equal to the maximum transmission bandwidth of the second circuit. The video display device is configured to process the to-be-transmitted data stream and play at least one display video stream after receiving the to-be-transmitted data stream. In this way, the video capture device transmits the control command through the first line, and sends the data stream to be transmitted to the video display device through the second line, which realizes the separate transmission of the control command and the data stream, and avoids using the same line to transmit the control command and the data stream. When the control instruction occupies the bandwidth of the line for transmitting the data stream, the problem of high transmission delay of the data stream is caused.

In the data stream transmission method provided by the embodiment of the present application, the data stream is transmitted through a dedicated second line, which reduces the transmission delay for transmitting the data stream to be transmitted, and improves the transmission rate for transmitting the data stream to be transmitted; In the process of sending the data stream to be transmitted to the video display device through the second line, the control command sent by the video display device can also be received through the first line. The video capture device sends the control command, which effectively reduces the transmission delay of the control command.

In addition, compared with the prior art, which can only transmit the video stream separately (after the transmission of one video stream is completed, another video stream is transmitted), in the data stream transmission method used in the embodiment of the present application, the video capture device can The multiple application video streams are spliced to obtain the data stream to be transmitted, and the to-be-transmitted data stream is put into the second line for transmission, so that the video capture device does not need to transmit each application video stream separately, and the transmission delay of the video stream is reduced. In addition, since the transmission bandwidth required for the data stream to be transmitted is less than or equal to the maximum transmission bandwidth of the second line, the video capture device can transmit the data stream to be transmitted according to the maximum transmission bandwidth of the second line, avoiding the transmission of the second line. Transmission delay caused by insufficient bandwidth.

The video capture device may be a camera or a camera, etc. The camera is used to capture still images or videos. The object is projected through the lens to generate an optical image onto the photosensitive element, which in turn generates the raw video stream. The photosensitive element can be a charge coupled device (Charge Coupled Device, CCD) or a complementary metal-oxide-semiconductor (Complementary Metal-Oxide-Semiconductor, CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to a graphics processor (Graphics Processing Unit, GPU) or an image signal processor (Image Signal Processor, ISP) to convert it into a digital image signal.

The video display device is used to display images, videos, etc. The video display device may be a television, a projector, a computer monitor, and the like. In some possible embodiments, the video display device may be other terminal devices with image display functions, such as wearable devices, vehicle-mounted devices, Augmented Reality (AR)/Virtual Reality (VR) devices, Notebook computer, Ultra-Mobile Personal Computer (UMPC), netbook, Personal Digital Assistant (PDA), etc.

By way of example, FIG. 1 is a schematic diagram of a video communication system provided by the present application. The video communication system includes a TV 110 , a retractable camera 120 , a camera 130 and a conference tablet 140 . The television 110 and the conference tablet can communicate via the network 150 . Optionally, the network 150 may include switches, servers and other devices.

As an optional implementation manner, the video capture device and the video display device can be integrated on a terminal device. As shown in FIG. 1 , the retractable camera 120 is arranged on the TV 110, and the device integrated with the TV function and the camera function can be It is called a smart screen.

In a possible example, multiple video capture devices may be set on the video display device, for example, multiple cameras may be set on the TV 110, and other shooting devices may also be connected through an interface.

As another optional implementation manner, the video capture device and the video display device may be set separately. As shown in FIG. 1 , the video capture device may be the camera 130 , and the video display device may be the conference tablet 140 .

The video display device may include one or more display screens, and the display screens are used to display images, videos, and the like. The display includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light). emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Mini-LED, Micro-LED, Micro-OLED, quantum dot light-emitting diode (quantum dot light emitting diodes, QLED), etc.

The video display device may further include a processor, for example, the processor may include an application processor (Application Processor, AP), a modem processor, a GPU, an ISP, a controller, a memory, a video codec, a digital signal processor ( Digital Signal Processor, DSP), baseband processor, and/or Neural-Network Processing Unit (NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors. Among them, the controller can be the nerve center and command center of the video display device. The controller can generate an operation control signal according to the instruction operation code and timing signal, and complete the control of fetching and executing instructions.

Memory may also be provided in the processor for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can hold instructions or data that the processor has just used or recycled. If the processor needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, the waiting time of the processor is reduced, and thus the efficiency of data flow transmission is improved.

In some embodiments, the processor may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transceiver (universal asynchronous transmitter) receiver/transmitter, UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, USB interface and/or HDMI interface, etc. Among them, the USB interface is an interface that conforms to the USB standard specification, and can specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like. In this embodiment of the present application, the USB interface is used to transmit control instructions.

As shown in FIG. 1 , the USB interface can be used to connect a charger to supply power to the TV 110 , and can also be used to transmit control commands between the TV 110 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The USB interface can also be used to connect other terminal devices, such as AR devices, retractable cameras 120 and the like.

It is worth noting that the video communication scenario shown in FIG. 1 is only a possible implementation provided by the embodiments of this application, and the data stream transmission method provided in this application is used for image transmission links, such as photography, videography, and monitoring. etc. involving the field of image transmission.

The implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

As shown in FIG. 2 , which is a schematic flowchart of a data stream transmission method provided by the present application, the video capture device 210 is connected to the video display device 220 through a first line and a second line, respectively. For example, the retractable camera 120 is the video capture device 210 and the TV 110 is the video display device 220 as an example, to illustrate the data stream transmission method provided in this application, the video capture device 210 can realize the scalable as shown in FIG. 1 . The functions of the camera 120 and the video display device 220 can realize the functions of the television 110 as shown in FIG. 1 .

The first line is used to transmit the control commands of the video capture device 210 and the video display device 220, and the second line is used for the video capture device 210 to transmit the data stream to the video display device 220, which realizes the separate transmission of the data stream and the control command. During the process that the acquisition device 210 sends the data stream to be transmitted to the video display device 220 through the second line, the video display device 220 may send a control instruction to the video acquisition device 210 through the first circuit, so that the video display device 220 does not need to transmit the data stream to be transmitted. The control command is sent to the video capture device 210 only after the transmission process of the video capture device is completed, which reduces the transmission delay of the control command.

Please continue to refer to FIG. 2 , the data stream transmission method provided by the embodiment of the present application may include the following steps.

S210, the video display device 210 generates a collection instruction in response to the operation information acting on the video display device.

Here, the video display device 220 is provided with a video capture device 210 as an example. As shown in FIG. 3 , FIG. 3 is a schematic diagram of a smart screen provided by the present application. The video display device 220 may be a smart screen 300 and the video capture device 210 It is the camera 310 on the smart screen 300.

For example, the operation information may be a remote control command received by the smart screen 300 and sent by the user through the remote controller of the smart screen 300.

For another example, the operation information is a control instruction sent by a remote device (such as a mobile phone) communicatively connected to the smart screen 300 through an application (application, APP).

For another example, the operation information is the voice information received by the audio input device of the smart screen 300, the audio input device may be a microphone set on the smart screen 300, and the audio input device may also be a remote control set on the smart screen 300 on the microphone.

For another example, the operation information may also be the gesture information of the user received by the smart screen 300. If the display screen 320 of the smart screen 300 is a touch screen, the gesture information may be the operation performed by the user on the touch screen; the gesture The information may also be a gesture operation in which the user performs a non-touch operation (such as an air-space operation) according to a certain action.

The above operation information is only a possible implementation manner provided by this application, and does not represent a limitation on this application. In some possible examples, the above-mentioned operation information may also be an instruction sent by a server wiredly connected to the video display device.

S220, the video display device 220 sends a capture instruction to the video capture device 210 through the first line.

In one possible example, the first line may be USB. For example, the processor of the video capture device 210 has a USB interface, and the processor of the video display device 220 also has a USB interface, and two ends of the first line are respectively connected to the two USB interfaces.

In another possible example, the first line may also be implemented through other interfaces of the video capture device 210 and the video display device 220. For example, the other interfaces may be, but not limited to: Mini USB interface, Micro USB interface interface, USB Type C interface, etc.

Using the first line to transmit the control command between the video capture device and the video display device avoids the transmission of the video stream and the control command in the same line, and further reduces the transmission of the video stream and the video stream during the one-way transmission process of the video stream. The transmission delay of the control command in the opposite direction.

S230 , the video collection device 210 generates a plurality of application video streams according to the original video streams collected by the video collection device 210 .

For example, the original video stream is a video data source obtained by the video capture device 210 photographing the target object or person.

In a possible example, the multiple application video streams may be multiple video streams with the same resolution, for example, each application video stream is a “1080P@60” video stream, and “1080P” refers to the video stream The resolution information of the video stream is 1920*1080 (width is 1920, height is 1080), and "@60" means that the frame rate information of the video stream is 60 frames/second.

In another possible example, the multiple application video streams may also be multiple video streams with different resolutions. For example, the multiple application video streams include a first application video stream and a second application video stream. The resolution of the video stream of one application is different from the resolution of the video stream of the second application. For example, the resolution information of the first application video stream is 1080P (width is 1920 and height is 1080), and the resolution information of the second application video stream is 720P (width is 1080 and height is 720).

The generation of multiple application video streams by the video capture device 210 may be realized based on multiple capture instructions. For example, the video display device 220 generates a plurality of capture commands according to the user's gesture information, and the plurality of capture commands includes a face recognition command and a preview command. , the video display device 220 sends the plurality of video capture instructions to the video capture device 210, so that the video capture device 210 generates a face recognition stream according to the face recognition command, generates a preview stream according to the preview command, and the video capture device 210 will After the preview stream and the face recognition stream are spliced to obtain the to-be-transmitted data stream, the video capture device 210 sends the to-be-transmitted data stream to the video display device 220, and the user can view the above-mentioned preview stream and face recognition stream through the video display device 220. Since the above-mentioned multiple application video streams are generated by the video capture device 210 according to the original video streams, the video display device 220 is prevented from converting the original video streams, and the computing resources of the video display device 220 are released, thereby enabling the video display device 220 There will be no lag when playing the video stream, which improves the user's experience of watching videos.

As an optional implementation manner, the multiple application video streams included in the data stream to be transmitted may be different types of video streams. For example, the multiple application video streams include at least one video stream among preview streams, photographing streams, depth data streams, face streams, object detection streams, monitoring streams, time-of-flight ranging streams, or gesture control streams. The user can use the video display device to view different video information of the original video stream, for example, the user can view the preview stream and the photo stream in the video call, and view the face recognition information obtained according to the original video stream during the video call (such as Classify the face shape of the face), thereby improving the user's experience of watching videos.

The preview stream refers to a video stream in which the video capture device previews the captured video data source. For example, the preview stream shown in Figure 3.

The camera stream refers to the video stream obtained by the video capture device taking pictures of the target object or person. For example, the camera flow shown in Figure 3.

The depth data stream refers to a group of image streams that take the distance (depth) from the video capture device to each point in the scene as the pixel value, which directly reflects the geometry of the visible surface of the scene. Depth image data can be calculated as point cloud data after coordinate transformation, and point cloud data with rules and necessary information can also be calculated as depth image data.

The face stream refers to the face information stream in the original video stream captured by the video capture device.

The target detection stream refers to the data stream that matches the target detection requirement obtained after the video capture device processes the original video stream. For example, the target detection requirement may be that the object category in the original video stream is a specified category, and the color of a person's clothes is a specified color. As shown in FIG. 3 , the target detection flow may be to perform skeleton recognition on the character in the original video stream to obtain skeleton data of the character, and the smart screen 300 may display the skeleton data stream.

The monitoring stream refers to the video stream obtained by the video capture device recording the monitoring scene. For example, the monitoring flow shown in Figure 3.

The time-of-flight ranging flow means that the video acquisition device continuously sends light pulses to the target, and then uses the sensor to receive the light returned from the object, and obtains the distance between the video acquisition device and the target by detecting the flight (round-trip) time of the light pulse, and then A data stream generated based on the distance of each point on the target.

The gesture control stream refers to the data stream obtained by the video capture device according to the gesture information of the target person in the original video stream. For example, the gesture control flow shown in Figure 3.

It is worth noting that FIG. 3 is only a possible implementation manner provided by the present application, and should not be construed as a limitation on the present application. In some possible embodiments, the video display device may also display more or fewer video streams.

Each application video stream can be a different type of video stream, and the user can use the video display device to view different video information according to the original video stream. Viewing the face shape identification information obtained according to the original video stream (for example, classifying the face shape of the face) improves the user's experience of watching videos.

As an optional implementation manner, the generation of multiple application video streams by the video collection apparatus may be implemented based on multiple collection instructions. As shown in Figure 3, for example, the user can click on a video call to generate a preview stream and a photo stream; another example, the user can generate a face recognition instruction through gesture operations, and then the video capture device generates a face stream according to the original video stream; , the user sends a fitness posture recognition instruction through the remote controller, and the video acquisition device generates a depth data stream (as shown in FIG. 3 ) according to the original video stream. The above examples are implemented with different instructions through different operation information. In some possible examples, the user may perform multiple operations within a period of time, and the video capture device may generate a collection instruction according to the multiple operations. The video capture device may be instructed to generate application video streams corresponding to the multiple operations.

Please continue to refer to FIG. 2 , the data stream transmission method provided by the present application further includes the following steps.

S240, the video collection device 210 splices the multiple application video streams to obtain a data stream to be transmitted.

The data stream to be transmitted may include channel information, and the channel information may include the number of application video streams in the data stream to be transmitted. Exemplarily, as shown in (a) of FIG. 4 , the multiple application video streams include application video stream 1 to application video stream 4; as shown in (c) of FIG. 4 , the channel information indicates that the data stream to be transmitted includes There are four application video streams, and the four application video streams are application video stream 1 to application video stream 4. The video capture device 210 may splicing application video stream 1 to application video stream 4 to obtain the data stream to be transmitted.

In a possible example, the channel information may further include the channel where each application video stream is located. Exemplarily, as shown in (c) of FIG. 4 , the application video stream 1 is set in the channel 1 .

The channel information may also include the number of channels in the second line, and bandwidth information for each channel. The bandwidth information refers to the width and height of the channel, and the bandwidth information of the channel determines the maximum resolution of the application video stream that can be transmitted by the channel. Illustratively, as shown in (b) of FIG. 4 , the data stream to be transmitted includes an information area 1, an information area 2 and a data area, and the data area includes multiple channels (channel 1 to channel 6), for example, channel 1 and The bandwidth information of channel 2 is "3840*540", and the bandwidth information of channel 3 to channel 6 is "3840*240". The maximum resolution of the application video stream that can be transmitted by channel 1 and channel 2 is 1080P (1920*1080 ), the maximum resolution of the application video stream that can be transmitted by channel 3 to channel 6 is 720P (1280*720).

As an optional implementation manner, the above S240 specifically includes: the video collection device 210 sets a plurality of application video streams in respective corresponding target channels. The target channel corresponding to the application video stream is a channel whose maximum transmission bandwidth is greater than or equal to the transmission bandwidth required by the application video stream. The transmission bandwidth required by the application video stream is determined by the video capture device according to the resolution information of the application video stream. Each application video stream can be put into the target channel whose transmission bandwidth is greater than or equal to the transmission bandwidth required by the application video stream. , which avoids the video capture device from putting the application video stream with larger resolution into the channel with smaller bandwidth information, thereby reducing the transmission delay caused by insufficient transmission bandwidth of the channel.

The transmission bandwidth of the channel indicates the maximum data volume of the video stream that can be transmitted by the channel in a unit time. Here, it is taken as an example that the video stream needs to transmit N frames of images with the same resolution in a unit time, the maximum data amount=the resolution of a single frame of image×N, where N is a positive integer. The above-mentioned transmission bandwidth required by the application video stream indicates the amount of data required to be transmitted by all image frames in the application video stream in a unit time.

In an optional implementation manner, the channel information may further include resolution information of each application video stream. For example, the resolution information of the application video streams 1 to 4 and the bandwidth information of the channels 1 to 6 shown in FIG. 4 are shown in Table 1.

Table 1

通道aisle	带宽bandwidth	视频流video stream	分辨率Resolution
通道1channel 1	38405403840540	应用视频流1Application Video Streaming 1	1080P(19201080)1080P(19201080)
通道2channel 2	38405403840540	应用视频流2App Video Streaming 2	1600120016001200

通道3channel 3	38402403840240	应用视频流3Application Video Streaming 3	720P(1280720)720P(1280720)
通道4Channel 4	38402403840240	应用视频流4Application Video Streaming 4	540P(960540)540P(960540)
通道5Channel 5	38402403840240
通道6Channel 6	38402403840240

Here, the same number of image frames transmitted per unit time of different application video streams is taken as an example. The comparison of transmission bandwidth is explained by taking the resolution as an example. The maximum transmission bandwidth of channel 1 and channel 2 is 3840*540. The bandwidth indicates that the maximum amount of data that can be transmitted by channel 1 and channel 2 is a video stream with a resolution of 1080P (1920*1080). As shown in Figure 4, since the transmission bandwidth required by application video stream 1 (with a resolution of 1080P) and application video stream 2 (with a resolution of 1600*1200) is less than or equal to the maximum transmission bandwidth of channel 1 and channel 2, the application video Stream 1 can be placed in channel 1, and application video stream 2 can be placed in channel 2.

The maximum transmission bandwidth of channels 3 to 6 is 3840*240, which indicates that the maximum amount of data that can be transmitted by channels 3 to 6 is a video stream with a resolution of 720P (1280*720). As shown in Figure 4, since the transmission bandwidth required by application video stream 3 (720P resolution) and application video stream 4 (540P resolution) is less than or equal to the maximum transmission bandwidth of channel 3 to channel 6, application video stream 3 It can be put into channel 3, and the application video stream 4 can be put into channel 4.

In a first possible example, the video capture device 210 setting the multiple application video streams in the respective corresponding target channels respectively includes: the video capture device 210 determines, according to the resolution information of the application video stream, the requirements of the application video stream. and put the application video stream into the target channel whose transmission bandwidth is greater than or equal to the transmission bandwidth required by the application video stream. The video capture device can determine the target channel based on the resolution information of the application video stream, avoid putting the application video stream with larger resolution into the channel with smaller bandwidth information, and reduce the transmission delay caused by insufficient transmission bandwidth of the channel.

In a second possible example, the multiple application video streams include a first application video stream and a second application video stream, and the video capture device 210 sets the multiple application video streams in respective corresponding target channels, and specifically includes: video The acquisition device determines the first transmission bandwidth according to the resolution information of the first application video stream, the video acquisition device determines the second transmission bandwidth according to the resolution information of the second application video stream, and further, the video acquisition device compares the first application video stream with the second transmission bandwidth. The second application video stream is set in the target channel, wherein the sum of the first transmission bandwidth and the second transmission bandwidth is less than or equal to the maximum transmission bandwidth of the target channel. Combining the contents shown in Table 1 and Figure 4, it is assumed that the first application video stream may be application video stream 3, the second application video stream may be application video stream 4, and the target channel may be channel 1. Since application video stream 3 and The total transmission bandwidth (1280*720+960*540=3840*375) required by the application video stream 4 is less than the maximum transmission bandwidth (3840*540) of the channel 1, and the video capture device 210 can combine the application video stream 3 and the application video stream 4 into channel 1. The video capture device can put multiple application video streams into the same target channel, and the total transmission bandwidth required by the multiple application video streams is less than or equal to the maximum transmission bandwidth of the target channel, which can effectively utilize each channel in the second line. The transmission capacity of the second line is avoided to avoid waste of the transmission capacity of the second line.

According to the bandwidth information of each channel in the data area and the resolution information of each application video stream in the channel information, the video capture device takes the channel whose maximum transmission bandwidth is greater than or equal to the transmission bandwidth required by the application video stream as the target channel of the application video stream , and further, the video capture device sets the application video stream in the target channel corresponding to the application video stream, so as to realize the transmission of the application video stream based on the transmission capability of the channel. That is to say, each application video stream can be put into the target channel whose transmission bandwidth is greater than or equal to the transmission bandwidth required by the application video stream, preventing the video capture device from putting the application video stream with larger resolution into the bandwidth. In the channel with smaller information, the transmission delay caused by insufficient transmission bandwidth of the channel is reduced.

The channel information provided in this application can be adjusted according to the application scenarios of the video capture device and the video display device. The application scenarios of the video capture device and the video display device are different, and the above channel information can also be different. The examples provided to describe the data streaming method should not be construed as a limitation on the present application. For example, the number of channels and bandwidth information in the above channel information may be pre-configured according to the application scenarios of the video capture device and the video display device. For example, in a video call scenario, the data area may include two channels, and the two channels They are used to transmit preview stream and photo stream respectively. For another example, the number of channels and the bandwidth information in the above channel information may be dynamically generated according to the number and resolution information of the application video streams.

S250, the video collection device sends the data stream to be transmitted to the video display device through the second line.

Wherein, the transmission bandwidth required by the data stream to be transmitted is less than or equal to the maximum transmission bandwidth of the second line. As shown in FIG. 4 , the total transmission bandwidth required by the application video stream 1 to the application video stream 4 is smaller than the maximum transmission bandwidth of the second line.

The maximum transmission bandwidth of the second line indicates the maximum data volume of the video stream that can be transmitted by the second line in a unit time; the above-mentioned transmission bandwidth required for the data stream to be transmitted indicates that all the data streams in the data stream to be transmitted in a unit time The amount of data to be transmitted by the application video stream and description information, which can be used to describe the resolution information, exposure information, white balance information or focus information of the application video stream, for example, the description information can include the above channel information.

In a possible example, the second line may be HDMI, or may be a VGA (video graphics array), a digital visual interface (digital visual interface, DVI) line, or a display port (display port, DP) line, or the like.

For example, if the second line is HDMI, the format of the application video stream may be the YUV format, wherein the Y information indicates the brightness (Luminance) of each frame image in the application video stream, and the U and V information indicates each frame in the application video stream. Color information (Chrominance) of a frame of image.

Since the video stream of the YUV format is a lossless video stream, and the transmission bandwidth of the USB in the prior art is small, the transmission delay of the YUV format is relatively high. In the data stream transmission method provided by the embodiment of the present application, the second line is an HDMI line, and the transmission bandwidth of the HDMI line is larger than that of the USB, which can reduce the transmission delay of the video capture device sending the data stream to be transmitted to the video display device.

It should be noted that the format of the application video stream is YUV format is only a possible example provided by the embodiments of the present application, and in some possible examples, the format of the application video stream may be lossless RGB (red green blue) format , the format of the application video stream may also be a lossy video format, such as an audio-video interleaved format (audio video interleaved, AVI) or a video file format (real audio variable bit, RMVB).

As an optional implementation manner, the channel information may be set in the header of the data stream to be transmitted. As shown in Figure 4, channel information (inner metadata) can be set in the information area 1.

The video acquisition device sets the channel information in the header of the data stream to be transmitted, and sends the data stream to be transmitted to the video display device through the second line. In the prior art, the video display device can only split the data stream to be transmitted according to the stream information of each application video stream, which requires a lot of computing resources, resulting in the video display device being stuck in playing the video stream. In the scheme, after the video display device receives the data stream to be transmitted, since the channel information is in the header of the data stream to be transmitted, the video display device can preferentially obtain the channel information in the data stream to be transmitted, and determine based on the channel information. The information of each channel in the data stream to be transmitted and the application video stream improves the speed of parsing the data stream to be transmitted by the video display device.

As another optional implementation manner, the data stream to be transmitted may further include stream information, the stream information indicates attribute information of each application video stream in the to-be-transmitted data stream, and the attribute information includes exposure information, focus information of the application video stream at least one of information or white balance information. Illustratively, the attribute information may be obtained by a video capture device using 3A (3-automatic) digital imaging technology, and the attribute information may include automatic exposure (AE) information, automatic focus (AF) information and automatic White balance (automatic white balance, AWB) information. For the relevant content of the 3A digital imaging technology, reference may be made to the relevant elaboration in the prior art, which will not be repeated here.

In some possible examples, as shown in FIG. 4 , the flow information may be provided in the information area 2 . It should be noted that the information area 2 and the data area shown in FIG. 4 are only examples given in this application to illustrate the above-mentioned embodiments. In some possible examples, the information area 2 may also be set after the data area, It can also be set between any 2 channels.

The video capture device sets the stream information in the data stream to be transmitted, and sends the data stream to be transmitted to the video display device through the second line. After the video display device receives the data stream to be transmitted, the video display device can determine the number, width and height of each channel in the data stream to be transmitted, and the number, location and resolution of the application video stream according to the channel information in the data stream to be transmitted. Further, the video display device can also determine the attribute information of the application video stream according to the stream information in the data stream to be transmitted, thereby realizing the identification of different types of application video streams, and improving the accuracy of the video display device in parsing the data stream to be transmitted. sex.

In a possible embodiment, the multiple application video streams may be different, and there may be a video stream larger than the maximum transmission bandwidth of the above-mentioned channel in the multiple application video streams, which will cause any channel in the second line to exist. The video stream cannot be streamed. In order to solve the above problem, the present application provides a possible implementation manner, as shown in FIG. 5 , which is a schematic flowchart of another data stream transmission method provided by the present application. The above S240 specifically includes S2401 and S2402.

S2401, if the transmission bandwidth required by the first application video stream is greater than the transmission bandwidth possessed by any one of the multiple channels, and is less than or equal to the maximum transmission bandwidth of the second line, the video capture device 210 collects the first application video stream as the first data stream to be transmitted.

The above-mentioned first application video stream is any one of multiple application video streams. For example, in combination with the content shown in Figure 4 and Table 1, if the resolution information of the first application video stream is 2K (2560*1440), and the maximum transmission bandwidth in channels 1 to 6 is 3840*540<2K , the video capture device 210 regards the first application video stream as the first data stream to be transmitted.

S2402 , if the transmission bandwidth required by the remaining data streams matches the transmission bandwidths of the multiple channels, the video capture device 210 splices multiple application video streams in the remaining data streams to obtain a second data stream to be transmitted.

The remaining data stream includes other application video streams except the first application video stream among the multiple application video streams obtained by the video collection device 210 according to the original video stream. For a related example of the second data stream to be transmitted, reference may be made to the related description of the above S240, and details are not repeated here.

Please continue to refer to FIG. 5 , the above S250 specifically includes S2501.

S2501, the video capture device 210 alternately sends the first data stream to be transmitted and the second data stream to be transmitted to the video display device 220 through the second line.

Exemplarily, FIG. 6 is a schematic diagram of alternate transmission of data streams provided by the application, where the second line is an HDMI line, and the maximum transmission bandwidth of the HDMI line can support 4K@60 (resolution 4096×2160, frame rate For example, the first application video stream is 4K@30 (resolution is 4096×2160, and the frame rate is 30 frames/second), and the remaining data streams include 4 application video streams ( For application video streams 1 to 4 shown in Table 1, the frame rates of which are all 30 frames per second), the video capture device 210 and the video display device 220 may alternately transmit the first data stream to be transmitted and the video display device 220 in units of frames. The second data stream to be transmitted. For example, the video capture device 210 first sends the first frame of image in the first data stream to be transmitted (4K@30 video stream), and then sends the first frame of image of each application video stream in the second data stream to be transmitted, and then The video capture device 210 sends the second frame image in the first data stream to be transmitted (4K@30 video stream), and then sends the second frame image of each application video stream in the second data stream to be transmitted, until the first frame image to be transmitted The transmission of each frame of image in the data stream and the second data stream to be transmitted is completed.

In the prior art, the camera transmits the application video stream with a larger resolution to the TV, and then sends other application video streams to the TV, resulting in a high transmission delay of the other application video streams. According to the data stream transmission method, the video capture device can use the first application video stream as the first data stream to be transmitted, and the first application video stream is a plurality of application video streams. The required transmission bandwidth is greater than any one of the multiple channels. The application video stream of the transmission bandwidth, and the remaining data stream that matches the transmission bandwidth of the multiple channels is used as the second data stream to be transmitted, and then the video capture device alternately sends the first data stream to be transmitted through the second line. and the second data stream to be transmitted, so as to realize the time-sharing and alternate transmission of a single application video stream with a larger resolution and multiple application video streams with a smaller resolution. In other words, using the data stream transmission method provided by the embodiment of the present application, the video capture device does not need to send the second data stream to be transmitted to the video display device after the transmission of the first data stream to be transmitted is completed, avoiding the need for the prior art The disadvantage of only being able to transmit the video stream separately (after the transmission of one video stream is completed, and then transmit the other video stream), reduces the transmission delay of the video stream.

Therefore, the video capture device 210 and the video display device 220 can perform partition transmission on the multiple application video streams or include time-sharing alternation of channel partitions based on the maximum transmission bandwidth of the second line and the bandwidth information of each channel in the data area. transmission, and further, reduce the transmission delay of the video stream and improve the transmission efficiency of the video stream. The partition transmission refers to splicing a plurality of application video streams to obtain the data stream to be transmitted, and then the video capture device sends the data stream to be transmitted to the video display device. For the content of the partition transmission, please refer to the relevant elaborations of the above S240 and S250, I won't go into details here. For the above-mentioned time-sharing alternate transmission including channel partitions, reference may be made to the related descriptions in FIG. 5 and FIG. 6 , and details are not repeated here.

In the above-mentioned embodiments of the present application, the time-sharing alternate transmission is described by taking the video capture device sending two data streams to be transmitted to one video display device as an example. The transport data streams can also be sent separately to different video display devices. For example, the video capture device sends the first data stream to be transmitted to the first video display device, and the first video display device plays the first application video stream; the video capture device sends the second data stream to be transmitted to the second video display device, The second video display device plays the application video stream included in the second data stream to be transmitted.

After the video display apparatus 220 receives the above-mentioned data stream to be transmitted, please continue to refer to FIG. 2 , the data stream transmission method provided by this embodiment of the present application may further include S260.

S260, the video display device 220 processes the data stream to be transmitted and plays at least one application video stream.

The above S260 specifically includes: the video display device 220 splits the data stream to be transmitted to obtain all the application video streams according to the channel information; further, the video display device 220 selects at least one application video stream among all the application video streams to play.

In a possible example, the application video stream played by the video display device 220 is obtained after the video display device 220 splits the data stream to be transmitted according to the channel information.

In another possible example, the application video stream played by the video display device 220 is: the video display device 220 splits the data stream to be transmitted according to the channel information, and performs AI processing on the split application video stream to obtain The AI data flow, the AI processing can be, but is not limited to: target detection, face recognition, object classification, color recognition, object counting, etc. For example, the video display device 220 includes a processor, and the processor can be configured to perform operations corresponding to the above AI processing, and display the results of the operations on the display screen of the video display device 220 .

Since the data stream to be transmitted includes channel information, the channel information includes the number of channels in the second line and the bandwidth information of each channel, the channel information also includes the number of application video streams in the data stream to be transmitted, and each application video stream The channel and resolution information of the stream. After the video display device 220 obtains the channel information, it can allocate computing resources for the playback of the application video stream according to the resolution information of each application video stream, which can improve the performance of the video display device. The utilization rate of the computing resources of the video display device 220 is avoided, and the video display device 220 is prevented from being stuck. In addition, in the process of splitting the data stream to be transmitted by the video display device 220, the video display device 220 can not only determine the position of the application video stream according to the channel where each application video stream is located, but also can use the channel information to include the application video stream. The resolution information of the stream determines the width and height information of the application video stream, which improves the accuracy of parsing the data stream to be transmitted by the video display device.

Since the data stream to be transmitted includes multiple application video streams, each application video stream can be a different type of video stream. After the video display device receives the data stream to be transmitted, the video display device can process and play the data stream to be transmitted. , the user can view various video information according to the original video stream through the video display device, for example, the user can view the preview stream and the photo stream in the video call, and view the face recognition obtained according to the original video stream during the video call. Information (such as classifying the face shape of a human face) improves the user's viewing experience.

It can be understood that, in order to realize the functions in the above embodiments, the video capture device and the video display device include corresponding hardware structures and/or software modules for performing each function. Those skilled in the art should easily realize that the units and method steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in hardware or in the form of a combination of hardware and computer software. Whether a function is performed by hardware or computer software-driven hardware depends on the specific application scenarios and design constraints of the technical solution.

The data stream transmission method provided according to this embodiment is described in detail above with reference to FIG. 1 to FIG. 6 , and the video capture device, video display device, and terminal provided according to this embodiment will be described below with reference to FIG. 7 and FIG. 8 . equipment.

FIG. 7 is a schematic diagram of a data stream transmission system provided by the present application. The data stream transmission system 700 can be used to implement the functions of the video capture device 210 and the video display device 220 in the above method embodiments, and thus can also achieve the beneficial effects of the above method embodiments. In this embodiment, the data stream transmission system 700 includes a video capture device 710 and a video display device 720 . The video capture device 710 and the video display device 720 are respectively connected through a first line and a second line, the first line is used to transmit the control instructions between the video capture device 710 and the video display device 720, and the second line is used to transmit the video capture The data stream to be transmitted sent by the device 710 to the video display device 720 .

The structure and function of the data stream transmission system 700 will be introduced below with reference to FIG. 7. The video capture device 710 can implement the functions of the video capture device 210 shown in FIG. 2, FIG. 5 and FIG. , the functions of the video display device 220 shown in FIGS. 5 and 6 . It should be understood that this embodiment only exemplarily divides the structures and functional modules of the video capture device 710 and the video display device 720, and the present application does not make any specific divisions.

As shown in FIG. 7 , the video capture device 710 includes a first transceiver unit 711 , a first processing unit 712 , a splicing unit 713 and a first storage unit 714 , and the above-mentioned units can be used to realize the above-mentioned ones shown in FIGS. 2 , 5 and 6 . methods corresponding to the respective operation steps performed by the video capture device 210.

When the video capture device 710 is used to implement the function of the video capture device 210 in the method embodiment shown in FIG. 2 , the first transceiver unit 711 is configured to execute S250, the first processing unit 712 is configured to execute S230, and the splicing unit 713 is configured to execute S230. Execute S240.

When the video capture device 710 is used to implement the function of the video capture device 210 in the method embodiment shown in FIG. 5 , the first transceiver unit 711 is configured to execute S2501, the first processing unit 712 is configured to execute S230, and the splicing unit 713 is configured to execute S230. S2401 and S2402 are executed.

The first storage unit 714 may be configured to store information such as multiple application video streams and data streams to be transmitted generated in the above steps.

As shown in FIG. 7 , the video display device 720 includes a second transceiver unit 721 , a second processing unit 722 , a display unit 723 and a second storage unit 724 , and the above-mentioned units can be used to implement the above-mentioned ones shown in FIGS. 2 , 5 and 6 . The methods corresponding to the respective operation steps performed by the video display device 220 are shown.

When the video display device 720 is used to implement the functions of the video display device 220 in the method embodiments shown in FIG. 2 and FIG. 5 , the second transceiver unit 721 is used to perform S220, the second processing unit 722 is used to perform S210, and the second The processing unit 722 and the display unit 723 are also used to perform S260.

The second storage unit 724 may be configured to store the collection instructions generated in the above steps, the multiple application video streams, the received data streams to be transmitted, and other information.

More detailed descriptions of the above-mentioned data stream transmission system 700 can be obtained directly by referring to the relevant descriptions in the embodiments shown in the above-mentioned FIG. 2 , FIG. 5 , and FIG. 6 , and details are not repeated here.

FIG. 8 is a schematic structural diagram of a terminal device provided by this application. The terminal device 800 includes a processor 810 and a communication interface 820 . The processor 810 and the communication interface 820 are coupled to each other. It is understood that the communication interface 820 can be a transceiver or an input-output interface. Optionally, the terminal device 800 may further include a memory 830 for storing instructions executed by the processor 810 or input data required by the processor 810 to run the instructions or data generated after the processor 810 runs the instructions.

When the terminal device 800 is used to implement the methods shown in FIG. 2 and FIG. 5 , the processor 810 , the communication interface 820 and the memory 830 may also cooperate to implement each of the data stream transmission methods performed by the video capture apparatus 210 or the video display apparatus 220 Steps. The terminal device 800 may also perform the functions of the video capture apparatus 710 and/or the video display apparatus 720 shown in FIG. 7 , which will not be described here.

The specific connection medium between the communication interface 820 , the processor 810 , and the memory 830 is not limited in the embodiments of the present application. In the embodiment of the present application, the communication interface 820, the processor 810, and the memory 830 are connected through a bus 840 in FIG. 8. The bus is represented by a thick line in FIG. 8, and the connection between other components is only for schematic illustration. , is not limited. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in FIG. 8, but it does not mean that there is only one bus or one type of bus.

The memory 830 may be used to store software programs and modules, such as program instructions/modules corresponding to the data stream transmission method provided in the embodiments of the present application, and the processor 810 executes various functions by executing the software programs and modules stored in the memory 830. applications and data processing. The communication interface 820 can be used for signaling or data communication with other devices. In this application, the terminal device 800 may have a plurality of communication interfaces 820 .

In a possible example, the above-mentioned memory 830 may be, but not limited to, a random access memory (Random Access Memory, RAM), a read only memory (Read Only Memory, ROM), a programmable read only memory (Programmable Read- Only Memory, PROM), Erasable Programmable Read-Only Memory (EPROM), Electric Erasable Programmable Read-Only Memory (EEPROM), etc.

The above-mentioned processor may be an integrated circuit chip with signal processing capability. The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; it may also be a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

The method steps in the embodiments of the present application may be implemented in a hardware manner, or may be implemented in a manner in which a processor executes software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM, flash memory, ROM, PROM, EPROM, EEPROM, registers, hard disk, removable hard disk, CD-ROM or any other form of storage medium well known in the art . An exemplary storage medium is coupled to the processor, such that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium may reside in an ASIC. Alternatively, the ASIC may be located in the end device. Of course, the processor and the storage medium may also exist in the terminal device as discrete components.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are executed in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, network equipment, user equipment, or other programmable apparatus. The computer program or instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instructions may be downloaded from a website site, computer, A server or data center transmits by wire or wireless to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server, data center, or the like that integrates one or more available media. The usable medium can be a magnetic medium, such as a floppy disk, a hard disk, and a magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); it can also be a semiconductor medium, such as a solid state drive (solid state drive). , SSD).

In the various embodiments of the present application, if there is no special description or logical conflict, the terms and/or descriptions between different embodiments are consistent and can be referred to each other, and the technical features in different embodiments are based on their inherent Logical relationships can be combined to form new embodiments.

In this application, "at least one" means one or more, and "plurality" means two or more. "And/or", which describes the association relationship of the associated objects, indicates that there can be three kinds of relationships, for example, A and/or B, which can indicate: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A, B can be singular or plural. In the text description of this application, the character "/" generally indicates that the related objects are a kind of "or" relationship; in the formula of this application, the character "/" indicates that the related objects are a kind of "division" Relationship. "At least one item(s) below" or similar expressions thereof refer to any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (a) of a, b or c, can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c" ", where a, b, c can be single or multiple. At least one (a) of a, b and c may also mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c" , where a, b, c can be single or multiple.

It can be understood that, the various numbers and numbers involved in the embodiments of the present application are only for the convenience of description, and are not used to limit the scope of the embodiments of the present application. The size of the sequence numbers of the above processes does not imply the sequence of execution, and the execution sequence of each process should be determined by its function and internal logic.

Claims

A data stream transmission method, characterized in that the method is performed by a video capture device, the video capture device is connected to a video display device through a first line and a second line respectively, and the first line is used to transmit the The video capture device and the control instruction of the video display device, the method includes:

generating multiple application video streams according to the original video streams collected by the video capture device;

splicing the multiple application video streams to obtain a data stream to be transmitted, the transmission bandwidth required for the data stream to be transmitted is less than or equal to the maximum transmission bandwidth of the second line;

The data stream to be transmitted is sent to the video display device through the second line.
The method according to claim 1, wherein the data stream to be transmitted includes channel information, the channel information includes the number of channels in the second line, and bandwidth information of each channel, the channel information The channel information further includes the number of application video streams in the data stream to be transmitted, and the channel where each of the application video streams is located.
The method according to claim 2, wherein the channel information includes resolution information of each of the application video streams.
The method according to claim 2 or 3, wherein the channel information is set in the header of the data stream to be transmitted.
The method according to any one of claims 1-4, wherein the plurality of application video streams comprise a first application video stream and a second application video stream, and the resolution of the first application video stream and the The resolutions of the second application video streams are different.
The method according to claim 5, wherein the data stream to be transmitted further includes a data area, the data area includes multiple channels, and the splicing of the multiple application video streams to obtain the data stream to be transmitted includes :

The multiple application video streams are respectively set in their corresponding target channels; the target channel corresponding to the application video stream is a channel whose maximum transmission bandwidth is greater than or equal to the transmission bandwidth required by the application video stream.
The method according to claim 5, wherein the data stream to be transmitted includes a data area, the data area includes multiple channels, and the splicing of the multiple application video streams to obtain the data stream to be transmitted includes:

If the transmission bandwidth required by the first application video stream is greater than the transmission bandwidth of any one of the multiple channels, and is less than or equal to the maximum transmission bandwidth of the second line, the first application video stream As the first data stream to be transmitted, the first application video stream is any one of the multiple application video streams;

If the transmission bandwidth required by the remaining data streams matches the transmission bandwidths of the multiple channels, splicing multiple application video streams in the remaining data streams to obtain a second data stream to be transmitted, the remaining data streams including Among the multiple application video streams obtained from the original video stream, other application video streams except the first application video stream;

The sending the data stream to be transmitted to the video display device through the second line includes:

The first data stream to be transmitted and the second data stream to be transmitted are alternately sent to the video display device through the second line.
The method according to any one of claims 1-7, wherein the multiple application video streams include a preview stream, a photo stream, a depth data stream, a face stream, a target detection stream, a monitoring stream, and a time-of-flight stream. At least one of ranging flow or gesture control flow.
The method according to any one of claims 1-8, wherein the first line is a universal serial bus USB, and the second line is a high-definition multimedia interface HDMI line.
A data stream transmission method, characterized in that the method is performed by a video display device, the video display device is connected to a video capture device through a first line and a second line respectively, and the first line is used to transmit the The video capture device and the control instruction of the video display device, the method includes:

The data stream to be transmitted sent by the video capture device is received through the second line, the data stream to be transmitted is obtained by splicing multiple application video streams, and the transmission bandwidth required for the data stream to be transmitted is less than or equal to the required transmission bandwidth. the maximum transmission bandwidth of the second line;

The data stream to be transmitted is processed and at least one application video stream is played.
The method according to claim 10, wherein the data stream to be transmitted includes channel information, the channel information includes the number of channels in the second line, and bandwidth information of each channel, the channel information The channel information further includes the number of application video streams in the data stream to be transmitted, and the channel where each of the application video streams is located.
The method according to claim 11, wherein the channel information comprises resolution information of each of the application video streams.
The method according to claim 11 or 12, wherein the channel information is set in the header of the data stream to be transmitted.
The method according to any one of claims 11-13, wherein the processing the data stream to be transmitted and playing at least one application video stream comprises:

According to the channel information, split the data stream to be transmitted to obtain all application video streams;

At least one application video stream among all the application video streams is selected to be played.
A video capture device, characterized in that the video capture device is connected to a video display device through a first line and a second line respectively, and the first line is used to transmit the data between the video capture device and the video display device. Control instructions, the video capture device includes:

a processing unit, configured to generate multiple application video streams according to the original video streams collected by the video capture device;

a splicing unit, configured to splicing the multiple application video streams to obtain a data stream to be transmitted, the transmission bandwidth required by the data stream to be transmitted is less than or equal to the maximum transmission bandwidth of the second line;

A transceiver unit, configured to send the data stream to be transmitted to the video display device through the second line.
The apparatus according to claim 15, wherein the data stream to be transmitted includes channel information, the channel information includes the number of channels in the second line, and bandwidth information of each channel, the channel information includes The channel information further includes the number of application video streams in the data stream to be transmitted, and the channel where each of the application video streams is located.
The apparatus according to claim 16, wherein the channel information includes resolution information of each of the application video streams.
The apparatus according to claim 16 or 17, wherein the channel information is set in the header of the data stream to be transmitted.
The apparatus according to any one of claims 15-18, wherein the plurality of application video streams comprise a first application video stream and a second application video stream, and the resolution of the first application video stream and the The resolutions of the second application video streams are different.
The device according to claim 19, wherein the data stream to be transmitted further includes a data area, the data area includes multiple channels, and the splicing unit is specifically configured to separate the multiple application video streams respectively. The target channel corresponding to the application video stream is a channel whose maximum transmission bandwidth is greater than or equal to the transmission bandwidth required by the application video stream.
The device according to claim 19, wherein the data stream to be transmitted comprises a data area, and the data area comprises a plurality of channels, and the splicing unit is specifically used for transmission if the first application video stream is required If the bandwidth is greater than the transmission bandwidth of any one of the multiple channels, and is less than or equal to the maximum transmission bandwidth of the second line, the first application video stream is used as the first data stream to be transmitted, and the The first application video stream is any one of the multiple application video streams;

The splicing unit is specifically configured to splicing multiple application video streams in the remaining data streams to obtain a second to-be-transmitted data stream if the transmission bandwidth required by the remaining data streams matches the transmission bandwidth possessed by the multiple channels. , the remaining data stream includes other application video streams other than the first application video stream among the multiple application video streams obtained according to the original video stream;

The transceiver unit is specifically configured to alternately send the first data stream to be transmitted and the second data stream to be transmitted to the video display device through the second line.
The device according to any one of claims 15-21, wherein the multiple application video streams include a preview stream, a photo stream, a depth data stream, a face stream, a target detection stream, a monitoring stream, and a time-of-flight stream. At least one of ranging flow or gesture control flow.
The device according to any one of claims 15-22, wherein the first line is a universal serial bus (USB), and the second line is a high-definition multimedia interface (HDMI) line.
A video display device, characterized in that the video display device is connected to a video capture device through a first line and a second line respectively, and the first line is used to transmit the data between the video capture device and the video display device. Control instructions, the video display device includes:

A transceiver unit, configured to receive the data stream to be transmitted sent by the video capture device through the second line, the data stream to be transmitted is obtained by splicing multiple application video streams, and the transmission data stream required by the data stream to be transmitted The bandwidth is less than or equal to the maximum transmission bandwidth of the second line;

a processing unit, configured to process the to-be-transmitted data stream to obtain multiple application video streams;

A display unit, configured to play at least one application video stream among the plurality of application video streams.
The apparatus according to claim 24, wherein the data stream to be transmitted includes channel information, the channel information includes the number of channels in the second line, and bandwidth information of each channel, the The channel information further includes the number of application video streams in the data stream to be transmitted, and the channel where each of the application video streams is located.
The apparatus according to claim 25, wherein the channel information comprises resolution information of each of the application video streams.
The apparatus according to claim 25 or 26, wherein the channel information is set in the header of the data stream to be transmitted.
The apparatus according to any one of claims 25-27, wherein the processing unit is specifically configured to split the data stream to be transmitted to obtain all the application video streams according to the channel information;

The display unit is specifically configured to select at least one application video stream among all the application video streams to play.
A terminal device, characterized in that it includes a processor and an interface circuit, the processor receives or transmits data through the interface circuit, and the processor is used for implementing the method as claimed in claims 1 to 9 through a logic circuit or executing code instructions The method of any one of, or the method of any one of claims 10 to 14.
A computer-readable storage medium, characterized in that a computer program is stored in the storage medium, and when the computer program is executed by a terminal device or a processor, the computer program according to any one of claims 1 to 9 is implemented. The method, or the method of any one of claims 10 to 14.
A computer program product, characterized in that the computer program product includes instructions, when the instructions are run on a terminal device or a processor, the method according to any one of claims 1 to 9 is implemented, or the right The method of any one of claims 10 to 14.