CN117528145A - Unified multi-picture method and system thereof - Google Patents

Abstract

The invention provides a unified multi-picture method and a system thereof. Wherein the method comprises: receiving configuration parameters obtained by completing custom layout of a plurality of sub-pictures on a configuration page through a control; and synthesizing at least one path of video stream according to the configuration parameters and the terminal performance, and transmitting the video stream. The unified multi-picture method and the system thereof realize that a user can easily and custom layout of multiple pictures directly in a conference control without manually editing codes, thereby greatly improving user experience and working efficiency; and the server synthesizes video streams with the least number of supported video streams for all terminals according to the terminal performance, and selects a proper code rate for each terminal, so that the terminal can flexibly adjust the picture layout, and the display capability of the terminal is fully utilized to realize the self-adaptive code rate, thereby optimizing the viewing experience.

Description

Unified multi-picture method and system thereof

Technical Field

The invention relates to the technical field of video communication, in particular to a unified multi-picture method and a system thereof.

Background

Internet-based video communication technology is widely used in video conference scenes for work and life. The multi-picture layout plays an important role in the video conference, and videos of a plurality of participants are allowed to be displayed on the same screen at the same time, so that each person can see facial expressions and limb languages of other participants, the visual angles do not need to be frequently switched, misunderstanding and interruption of communication are reduced, communication and communication effects are enhanced, team cooperation efficiency is improved, all members can better understand each other's views and feedback, joint work and decision are promoted, and the conference is more interactive and vivid. And because the video pictures are simultaneously displayed on the screen, each participant can feel the importance of the participant, thereby being beneficial to improving the participation and investment of the participant.

In addition, in an educational, training, or presentation scene, a multi-screen layout can simultaneously display a plurality of contents, such as a presenter's screen, slide, video, etc., and the display effect can be enhanced.

In a word, the multi-picture layout has an important function, and provides a multi-azimuth information display and communication mode for users. Not only can better communication and cooperation experience be provided, but also conference efficiency and participation degree can be improved, and team cooperation is promoted, and remote work and remote education are supported.

However, the multi-screen layout in the prior art is only a layout template stored by the server, and the selectable number is limited. The requirement of adding a layout template requires a technician to manually edit the settings through codes, and a visual configuration page is not set, so that a controllable person is not supported to perform custom layout through the visual configuration page, and the difficulty of custom layout is certainly increased.

For example, the video conference software APP used by the king, although having some preset layout templates for selection, in order to better meet the requirements in the video conference, the king wants to add a specific custom layout template, which needs to be manually edited by codes. Firstly, finding the file position for storing the configuration of the layout template in the APP; then, how the configuration file of the layout template works needs to be known, and the format of the configuration file, the parameters and the attributes of the layout elements and the like are related; secondly, a text editor or a code editor is used for opening the found layout template configuration file, and the preset layout template setting is found; thirdly, manually adding a new custom layout code according to the format of a preset layout template, wherein the new custom layout code relates to the name of a designated layout, the number and the position of layout elements, the size of a picture and other attributes; finally, after the editing is completed, the configuration file is saved, and the APP is restarted.

In summary, manually adding layout templates through code editing settings often requires some programming skill and in-depth knowledge of video conferencing software, is complex and difficult for non-technicians, and is prone to error accidents leading to software problems, such as incorrect settings that can lead to software crashes or dysfunctions; moreover, different versions of video conference software may have different configuration file structures and setting modes, and manual editing may cause compatibility problems, especially incompatibility after software update; adding layout templates by manual editing settings requires periodic maintenance and updating, especially when software updates or configuration files change, requiring re-editing to accommodate new situations; manual editing of settings is difficult to propagate and share among multiple devices or users, resulting in different layout templates being used among different users, and inconvenient for unified management and use.

Therefore, the manual addition of the layout template through codes has complex process, has technical requirements on editors, is easy to edit errors to cause software abnormality, has compatibility problems, is not easy to maintain and update, and is not easy to propagate and share.

In addition to the above problems, on the other hand, in the prior art, the server only synthesizes one path of video stream with a fixed code rate according to the layout template selected by the conference control and respectively sends the video stream to each terminal, and the own performances of different terminals are not considered, for example, the terminal can support several paths of video streams, screen sizes and network environments.

For example, the server uniformly synthesizes and transmits a video stream with a bit rate of more than 10Mbps to the terminals A-D. And the terminal A is a smart phone supporting 2 paths of video streams, and the terminals B-D are video conference terminals supporting 4 paths of video streams, namely, the terminals A-D can all support 2 paths of video streams.

Terminal a is a smart phone with a small screen size, e.g., 5 inches. Higher code rates are not suitable for playback on smart phones with small-sized screens with poor processing power or poor network conditions: firstly, a higher code rate means that the video data amount is larger, and if the processing capability of the terminal A is poorer or the network condition is not good, the large amount of data cannot be effectively processed, so that the phenomenon of blocking and buffering occurs in the playing of the video, and the watching experience is poor; second, higher code rates require more computing resources and memory to decode and display video content, which can lead to overheating or excessive power consumption on smaller size devices; third, high resolution and high rate video typically contains more detail and picture information, but when viewed on a smaller screen, the detail is not obvious, rather bandwidth and resources are wasted; fourth, high rate video is typically designed for large screens to maintain a high quality viewing experience, but is viewed on smaller screens, which appear too sharp and instead do not fit the screen size, resulting in viewing discomfort.

In addition, the terminals A-D can support 2 paths of video streams, and the server only transmits one path of synthesized video stream to four terminals, so that the terminals A-D can only see synthesized pictures and can not respectively display 2 independent video pictures. And because only one path of synthesized video stream is provided, the position and the size of the picture cannot be arranged and adjusted according to personal requirements to carry out custom layout.

Therefore, in the prior art, on one hand, the operation of adding a layout template is complex, the software crash risk and the compatibility problem exist, and on the other hand, the service end does not comprehensively consider different terminal performances, only synthesizes a path of video stream with a fixed code rate and respectively sends the video stream to each terminal, and the video stream cannot be flexibly configured aiming at the terminal performances, so that the watching experience and the watching effect are poor.

Disclosure of Invention

The invention provides a unified multi-picture method and a system thereof, wherein a controllable person directly completes the self-defined layout of a plurality of sub-pictures by dragging a control on a controllable visual configuration page to obtain configuration parameters, the configuration parameters are sent to a server, and the server synthesizes at least one path of video stream according to the configuration parameters and the terminal performance to be sent to the terminal, so that the user can easily self-define the multi-picture layout directly in the controllable manner, and the technical threshold of the self-defined multi-picture layout is reduced without manual editing codes, friendly visual interface and direct mouse dragging operation mode, more users can conveniently customize layout templates to meet the diversified layout requirements of the users, and the user experience and the working efficiency are greatly improved; in addition, the server synthesizes the video streams with the least number of supported video streams for all terminals according to the terminal performance, and selects a proper code rate for each terminal, so that the terminal can flexibly adjust the picture layout, and fully utilizes the display capability of the terminal to realize the self-adaptive code rate, thereby optimizing the viewing experience, such as sending high-code rate video streams for terminals with larger screen sizes and stronger network environments. And for the terminal with smaller screen size and weaker network environment, the video stream with lower code rate can be sent, so that smooth playing and watching effects of the video stream are ensured.

In a first aspect, the present invention provides a unified multi-picture method, which is characterized in that the method includes:

receiving configuration parameters obtained by completing custom layout of a plurality of sub-pictures on a configuration page through a control;

and synthesizing at least one path of video stream according to the configuration parameters and the terminal performance, and transmitting the video stream.

The invention also provides a unified multi-picture system, which is characterized by comprising a server and a conference control, wherein the server comprises a receiving unit, a synthesizing unit and a sending unit, and the conference control comprises a control; wherein the method comprises the steps of

The conference controls the configuration parameters obtained by completing the custom layout of a plurality of sub-pictures on the configuration page through the control;

the receiving unit is used for receiving the configuration parameters;

the synthesizing unit is used for synthesizing at least one path of video stream according to the configuration parameters and the terminal performance;

the sending unit is used for sending the at least one path of video stream.

The invention provides a unified multi-picture method and a system thereof: firstly, the user-defined layout of a plurality of sub-pictures is completed by dragging the control on the controllable visual configuration page, so that the user can easily and custom layout of the sub-pictures directly in the controllable visual configuration page without manually editing codes, the friendly visual interface and the operation mode of direct mouse dragging reduce the technical threshold of the custom layout of the sub-pictures, more users can conveniently customize the layout template to meet the diversified layout requirements of the user, and the user experience and the working efficiency are greatly improved; secondly, the server synthesizes and synthesizes video streams with the least number of paths of supported video streams of all terminals according to configuration parameters and the capability of the terminals to support the video streams, and sends the video streams to the terminals, and a terminal user can arrange and adjust picture positions and sizes corresponding to a plurality of paths of video streams according to personal requirements to carry out custom layout; thirdly, the server side considers the screen size and the network environment of the terminal, selects a video stream with a proper code rate to transmit, for example, according to the network bandwidth and the quality of the terminal, the server side can dynamically adjust the code rate so as to ensure the stability and the fluency of video transmission; fourth, under the condition of bad network condition, weaker terminal performance or smaller screen size, the video stream of the large-picture combination can be guaranteed to be normally transmitted and played preferentially, and other sub-picture combinations can be not transmitted or displayed according to the situation or transmitted with low code rate so as to adapt to the specific situations of various terminals.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a unified multi-screen method provided by an embodiment of the present invention;

FIG. 2 is a block diagram of a unified multi-screen system provided by an embodiment of the present invention;

FIG. 3 is a configuration page interface provided by an embodiment of the present invention;

FIG. 4 is a configuration sprite filling mode interface provided by an embodiment of the present invention;

FIG. 5 is a custom 10 sprite layout interface provided by an embodiment of the present invention;

fig. 6 is a layout name setting popup interface according to an embodiment of the present invention.

Detailed Description

The technical scheme of the invention is further described in detail through the drawings and the embodiments.

Summary of The Invention

As described above, the present invention provides a unified multi-picture method and system with convenient operation, in which a controllable person directly completes the custom layout of multiple sub-pictures by dragging a control on a controllable visual configuration page, obtains configuration parameters, sends the configuration parameters to a server, and then the server synthesizes at least one path of video stream according to the configuration parameters and terminal performance to send to a terminal, so that the working efficiency and viewing experience can be significantly improved.

Exemplary method

Fig. 1 is a flowchart of a unified multi-screen method according to an embodiment of the present invention, where the embodiment includes the following steps:

s101: and receiving configuration parameters obtained by completing the custom layout of the multiple sub-pictures on the configuration page through the control.

Specifically, the configuration area and the control are rectangles with the aspect ratio of 16:9, the configuration parameters include at least two groups of sub-picture vertex coordinates and the corresponding relation between each sub-picture and the video stream content, and the configuration parameters can further include filling parameters of at least one sub-picture. The custom layout not only comprises the position and the size of each sub-picture, but also comprises the corresponding relation between each sub-picture and the content of the video stream. The correspondence can be adaptively adjusted according to the variation of the speaker during the conference. For example, at least one large screen, which is larger in size than the other sub-screens, is included in the plurality of sub-screens for showing a close-up of a speaker or a product or PPT being shown. When the speaker changes, the video stream displayed on the large screen is correspondingly switched. The switching process can be automatically switched based on preset rules, priorities, the volume of a speaker, action amplitude and appointed special action identification, or can be switched after meeting control agreeing by a participant to apply for speaking.

For example, the custom layout shown in fig. 5 is 10 sub-frames, the top left large frame content is the detail of the product presentation, the top right large frame content is the face close-up of the presenter, and the remaining 8 small frame contents are the video of the remote participant.

The conference control personnel, namely conference control personnel, complete the custom layout of a plurality of sub-pictures in the conference control. The conference control refers to the part of functions and operations for managing and controlling the conference in the video conference or the remote conference, and relates to various settings of the conference, management of participants, adjustment of video layout, permission control and the like. Conference control is a very important part of video conferencing, which ensures smooth progress of the conference and the order of participants, while providing better user experience and conference management. The meeting control personnel plays a key role in the meeting and is responsible for adjusting and controlling various parameters and settings so as to ensure the effect and quality of the meeting.

The step of completing the configuration parameters obtained by the custom layout of the multiple sub-frames on the configuration page through the control may specifically include: arranging the control in a configuration area of the configuration page, completing the size and position adjustment of a corresponding sub-picture, and if dragging the control to the configuration area, inputting the vertex coordinates of the upper left corner and the size of the control, or dragging the control to the configuration area, directly pulling four corners or four sides of the control to adjust the size; generating corresponding sub-picture vertex coordinates according to the size and the position of the control; and sequentially obtaining the vertex coordinates of each sub-picture to complete the layout of a plurality of sub-pictures.

When the size or position of the control is changed, the adjusted vertex coordinates of the corresponding sub-picture are updated in real time. Specifically, the conference may acquire the latest vertex coordinates by polling. Polling is a common computer programming and communication technique for periodically querying or checking for changes in a certain state or resource. In polling, a system or program periodically sends requests to specific resources, devices, or servers to obtain up-to-date information or status updates. Although polling is a simple and common method, it has some drawbacks, for example, polling may result in waste of system resources because requests are sent frequently even without new information. To avoid this problem, other more efficient methods may be used to obtain the latest vertex coordinates, such as using event driven approaches or long polling techniques.

When the configuration area is blank up and down or left and right in the layout of the multiple sub-pictures, the whole layout is centered, and the vertex coordinates of all the sub-pictures are updated.

And when at least one sub-picture which is not 16:9 exists in the layout of the sub-pictures, configuring the filling mode of the sub-pictures to obtain filling parameters.

Including but not limited to equal scale filling, width-preserving filling, height-preserving filling, stretch filling, and tiling filling. In practical applications, the selection of a suitable small-screen filling method depends on the specific requirements and visual effects. The equal scale filling approach to maintaining the original aspect ratio of the picture is often a more common approach to maintaining the integrity and aesthetics of the picture. Different filling modes are suitable for different video layouts and viewer viewing requirements.

After the custom layout of the multiple sub-pictures is completed, the configuration parameters can be clicked and stored to the server side on the configuration page, and the configuration parameters are used as template parameters for later selection. In addition, the controllable personnel can obtain a default setting instruction on the configuration page by setting the layout of the multiple sub-frames to be the default layout corresponding to the number of the sub-frames, and send the default setting instruction to the server, where the server stores the default setting instruction, for example, the layout shown in fig. 5 is set to be the default layout of 10 frames, and when the number of sub-frames input next time is 10, the controllable personnel can control the default to select the layout shown in fig. 5.

As an alternative embodiment, fig. 3 is a configuration page in a conference, where a rectangle in blue at the upper left is the control, a grid area below the control is a configuration area, the configuration page provides a 16:9 basic unit, and a custom layout design can be performed by dragging the control to the configuration area, and the configuration parameters include at least two groups of sprite vertex coordinates.

The specific configuration operation is as follows: and dragging the control to the configuration area, inputting the top left corner vertex coordinates and the control size, such as the width and height size, to finish the size and position adjustment of the corresponding sub-picture, generating the corresponding sub-picture vertex coordinates according to the size and position of the control, and displaying 4 vertex coordinates of each picture after the size adjustment of the control is finished or the position of the control is changed. And sequentially adjusting the positions and the sizes of all the sub-pictures to obtain the vertex coordinates of each sub-picture, and completing the layout of a plurality of sub-pictures. Wherein the coordinate points and sizes support only numeric and english decimal point inputs, supporting 2-bit decimal. And the control supports dragging to the configuration area for multiple times, and the configuration area maximally supports 8 x 8 layout. And supporting the control of the configuration area, for example, a large picture A comprises a small picture B, edge intersection is not supported, and if the edges are intersected, namely, at least one vertex of one picture but not all vertex coordinates fall within the range of the other picture, an error is prompted and a controllable person is required to reset the vertex coordinates and the size of the control. When two control edges overlap, the overlapping line is highlighted and the coordinate point where the coordinates overlap may only display one coordinate.

When the layout of all the sub-pictures is completed and blanks are arranged on the upper side, the lower side or the left side and the right side, the whole layout can be centered by configuring the whole left-right centering or the whole up-down centering of the bottom of the page, and then the coordinates are automatically refreshed. When a small picture in the layout has a picture which is not 16:9, the 'non-standard 16:9 size' at the bottom of the configuration page can be selected, and a configuration small picture filling mode is shown as shown in fig. 4, if equal-scale cutting is selected, the small picture is cut into a 16:9 area according to the equal-scale cutting mode, and the original aspect ratio of the picture is maintained; if not, filling the small picture of non-16:9 according to a default mode, if the stretching filling is kept by default, and directly stretching the small picture to the same width and height as those of the large picture area.

After the layout of all the sub-pictures is completed, the default layout of the number of the pictures can be set as the current layout, and when the number of the participants is insufficient in the automatic multi-picture scene, the pictures are displayed locally according to the default configuration of the user when being degraded. For example, the layout shown in fig. 5 is set to be a default layout of 10 frames, the number of sub-frames of the video conference is 10, and the default layout shown in fig. 5 is controlled to be selected, but during the conference, the participant a exits the conference, the frames are automatically degraded, and the frames are automatically displayed locally according to the default configuration of the user, such as automatically blacking a small frame in the lower right corner of fig. 5.

After the layout design is completed, clicking a storage button, popping up a name setting popup window shown in fig. 6, supporting Chinese and English, numbers and visible special characters by the name, and storing the name within 20 characters in a coordinate point set mode for service use after the layout is successfully stored, wherein the name can be exported and shared to other controllable personnel.

In addition, the conference control front end multi-picture layout pane and the conference template multi-picture layout pane calculate the layout according to the coordinate points for the conference control personnel to select.

The conference control front-end multi-picture layout pane is a plurality of pictures displayed on a front-end interface of a conference in a video conference, such as a host, a presenter, each participant and the like of the conference. These pictures can be displayed simultaneously at the front-end interface of the conference so that the conference personnel can clearly see the video pictures of each participant. Conference template multi-screen layout panes are common layouts including split screens, tiling, picture-in-picture, sidebars, and the like.

Because the controllable personnel can complete the custom layout of the multiple sub-pictures on the visual configuration page through easily dragging the control, the controllable personnel can control the selectable layout templates more and more than the prior art.

The visual custom layout of the invention has at least the following advantages:

firstly, the visual configuration page provides an visual user interface, and a user can complete the configuration of the layout through simple drag and click operations without writing complex codes or configuration files, so that the use threshold is reduced, and non-technical staff can easily use the visual configuration page;

second, in the visualized configuration page, the user can preview the effect of the layout in real time. When the control is dragged, the layout of the picture can immediately respond to the change, so that a user is helped to quickly adjust and optimize the layout, the debugging time is saved, the picture is intuitively adjusted, the layout effect is checked in real time, and the operation is more intuitive, natural and smooth;

thirdly, the layout setting completed by the visual configuration page can be easily stored, exported and shared to other users, so that the same layout template is used among different users, and unified management and use are convenient.

In a word, the visualized configuration page completes the custom layout of a plurality of sub-pictures through a simple drag control, so that the layout configuration becomes simple, flexible and visual, the working efficiency of a controllable person is improved, and better user experience is provided.

S102: and synthesizing at least one path of video stream according to the configuration parameters and the terminal performance, and transmitting the video stream.

The terminal capabilities include the terminal supporting video streaming, screen size, and network environment.

The video streams supported by the terminal refer to the number of video streams that can be simultaneously processed at the terminal. It generally depends on factors such as the hardware performance of the terminal, the software capabilities, and the network bandwidth. Different types of terminals may support different numbers of video streams, such as smartphones and tablet computers typically support 1 to 2 video streams; typical personal computers (e.g., notebook and desktop) typically support 2 to 4 video streams, depending on their processor capabilities and graphics processing capabilities, terminals dedicated to video conferencing typically support more video streams, and advanced video conferencing terminals can support 4 to 16 video streams. It should be noted that the number of video streams supported by the terminal is also closely related to the network bandwidth. Even if the terminal itself has the capability to handle multiple video streams, if the network bandwidth is insufficient, the video streams actually supported by it will be correspondingly reduced.

Screen size refers to the physical size of a terminal display device (e.g., television, computer display, smart phone, tablet, etc.), typically expressed in diagonal length in inches (inch). The screen size determines the size of the viewable area of the display device, i.e., the size of the effective image area that can be displayed on the screen. Larger screen sizes may provide a wider field of view and better viewing experience, particularly when viewing high rate video, multitasking, or presenting multiple video conferences. The smaller screen size accommodates low-rate video.

By reasonably matching the code rate of the video stream and the screen size of the terminal, smooth playing of the video on the terminal can be ensured, the image is clear, the conditions of blocking and picture distortion can not occur, the viewing experience can be optimized, and the user satisfaction can be improved; for terminals with smaller screens, selecting a proper code rate can save bandwidth, a smaller screen size means lower resolution requirements, and properly reducing the code rate can reduce data transmission amount and reduce network load; terminals with smaller screens generally have lower hardware performance, and selecting an appropriate code rate can reduce resource consumption in the decoding process and ensure normal operation of the terminals.

In addition, the server side can dynamically adjust the code rate according to the real-time network environment so as to ensure the stability and fluency of video transmission.

Specifically, the server judges the number of channels of the least supported video stream of all terminals according to the video stream supported by all the terminals, and divides the video stream into the video streams of the number of channels of the supported video stream according to the custom layout, for example, a plurality of sub-pictures forming a rectangle in the custom layout are divided into one channel of video stream, then code rates of the video stream are respectively selected for each terminal according to the screen size and the network environment of each terminal, and the code rates are sent to each terminal.

For example, terminal a is a 14-inch notebook computer supporting 4 video streams, terminal B is a 21.5-inch video conference terminal supporting 5 video streams, and terminals C-F are 32-inch video conference terminals supporting 8 video streams, i.e., terminals a-F support a minimum number of 4 video streams, i.e., terminals a-F can all support 4 video streams. The network environments of terminals a-F are all good. The number of sub-frames of the conference is 10, and the configuration layout of the conference control personnel is the multi-frame layout shown in fig. 5. The video stream contents of the sub-pictures in the upper left corner and the upper right corner are respectively the upper body of a teacher and the PPT of the teacher, and the video stream contents of the rest 8 sub-pictures are the head portraits of 8 students. The server divides the layout shown in fig. 5 into 4 paths of video streams according to the received vertex coordinates of 10 groups of sub-pictures, the corresponding relation between each sub-picture and the video stream content, and the minimum number of paths, screen sizes and network environments of supported video streams of terminals a-F, wherein the 4 paths of video streams are specifically an upper left sub-picture, an upper right sub-picture, a lower left 4 sub-picture and a lower right 4 sub-picture, namely, the 4 paths of video streams of the lower left 4 sub-pictures are combined into 1 path, and the 4 paths of video streams of the lower right 4 sub-pictures are combined into 1 path. And the server selects 4 paths of video streams with medium and low code rates to be sent to the terminal A, selects 4 paths of video streams with medium and high code rates to be sent to the terminal B, and selects 4 paths of video streams with high code rates to be sent to the terminals C-F according to the screen sizes of the terminals.

In a word, the server synthesizes the video streams with the least number of supported video streams of all terminals according to the configuration parameters and the video stream supporting capability of the terminals, and sends the video streams to the terminals, a terminal user can arrange and adjust the positions and the sizes of pictures corresponding to the video streams according to personal requirements to carry out custom layout, and can comprehensively consider the screen size and the network environment of the terminals, select video streams with proper code rates to send, dynamically adjust the code rates according to the network environment, and ensure the stability and the fluency of video transmission.

In the case of dividing a sub-picture into a plurality of video streams, each sub-picture combination may be encoded with a different code rate. And under the conditions of poor network condition, weaker terminal performance or smaller screen size, the video stream of the large-picture combination can be preferentially ensured to be normally transmitted and played, and other sub-picture combinations can be not transmitted or displayed according to the situation or transmitted with a low code rate so as to adapt to the specific situations of various terminals.

Exemplary System

Correspondingly, the embodiment of the invention also provides a unified multi-picture system. Fig. 2 is a block diagram of a unified multi-screen system according to an embodiment of the present invention, and as shown in fig. 2, a system 100 according to the present embodiment includes: the system comprises a server 101 and a conference 102, wherein the server 101 comprises a receiving unit 103, a synthesizing unit 104 and a sending unit 105, and the conference 102 comprises a control 106; wherein the method comprises the steps of

The conference control 102 is configured to complete configuration parameters obtained by the custom layout of a plurality of sub-frames on a configuration page through the control 106;

the receiving unit 103 is configured to receive the configuration parameter;

the synthesizing unit 104 is configured to synthesize at least one path of video stream according to the configuration parameters and the terminal performance;

the sending unit 105 is configured to send the at least one video stream.

The control 106 is rectangular;

the configuration parameters include at least two sets of sprite vertex coordinates.

The conference 102 further comprises an arrangement unit 107 and a generation coordinate unit 108; wherein the method comprises the steps of

The arrangement unit 107 is configured to arrange the control 106 in a configuration area of the configuration page, so as to complete size and position adjustment of the corresponding sub-frame;

the generating coordinate unit 108 is configured to generate vertex coordinates of the corresponding sub-frames according to the size and the position of the control 106, and sequentially obtain the vertex coordinates of each sub-frame, so that the conference control 102 completes the layout of the multiple sub-frames.

The arrangement unit 107 comprises a drag module 109 and an input module 110; wherein the method comprises the steps of

The drag module 109 is configured to drag the control 106 to the configuration area, and the input module 110 is configured to input the coordinates of the top left corner vertex and the size of the control 106, so that the arrangement unit 107 completes the size and position adjustment of the corresponding sub-frame.

The control 106 supports multiple drags to the configuration area;

inclusion is supported between at least two of the controls 106 of the configuration region, without edge intersection being supported.

When the size or position of the control 106 changes, the generating coordinate unit 108 is further configured to update the adjusted vertex coordinates of the corresponding sprite in real time.

When the configuration area is blank up and down or left and right in the layout of the multiple sub-frames, the layout unit is further configured to center the whole layout, and the generating coordinate unit 108 is further configured to update vertex coordinates of all the sub-frames.

The configuration area and the control 106 are rectangles with the aspect ratio of 16:9;

the configuration parameters also comprise filling parameters of at least one sub-picture;

the conference control 102 further includes a filling unit 111, where when at least one sub-picture other than 16:9 exists in the layout of the sub-pictures, the filling unit 111 is configured to configure a filling manner of the sub-picture, so as to obtain a filling parameter.

The server 101 further comprises a storage unit 112 for storing the configuration parameters as template parameters.

The conference control 102 further includes a default setting unit 113, configured to obtain a default setting instruction by setting the layouts of the multiple sub-frames to a default layout corresponding to the number of sub-frames;

The receiving unit 103 and the storage unit 112 are respectively configured to receive and store the default setting instruction.

The terminal capabilities include the terminal supporting video streaming, screen size, and network environment.

The configuration parameters comprise the corresponding relation between each sub-picture and the video stream content.

The synthesizing unit 104 comprises a road number judging module 114, a dividing module 115 and a code rate selecting module 116; wherein the method comprises the steps of

The judging path number module 114 is configured to judge the path number of the least supported video stream of all terminals according to the video stream supported by all the terminals;

the splitting module 115 is configured to split the video stream into the number of video streams supported according to the custom layout;

the code rate selecting module 116 is configured to select a code rate of the video stream for each terminal according to the screen size and the network environment of each terminal;

the sending unit 105 is configured to send the video stream after the code rate is selected.

The splitting module 115 is configured to split a plurality of sub-frames forming a rectangle in the custom layout into a video stream.

It should be noted that although the operations of the unified multi-picture method of the present invention are depicted in the drawings in a particular order, this does not require or imply that the operations must be performed in that particular order or that all of the illustrated operations be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform.

Furthermore, although several devices, units, or modules of a unified multi-picture system are mentioned in the above detailed description, such partitioning is merely exemplary and not mandatory. Indeed, the features and functions of two or more modules described above may be embodied in one module in accordance with embodiments of the present invention. Conversely, the features and functions of one module described above may be further divided into a plurality of modules to be embodied.

While the spirit and principles of the present invention have been described with reference to several particular embodiments, it is to be understood that the invention is not limited to the disclosed embodiments nor does it imply that features of the various aspects are not useful in combination, nor are they useful in any combination, such as for convenience of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

The invention provides:

1. a unified multi-picture method, the method comprising:

receiving configuration parameters obtained by completing custom layout of a plurality of sub-pictures on a configuration page through a control;

and synthesizing at least one path of video stream according to the configuration parameters and the terminal performance, and transmitting the video stream.

2. The unified multi-screen method according to claim 1, wherein the control is rectangular;

the configuration parameters include at least two sets of sprite vertex coordinates.

3. The unified multi-screen method according to claim 1 or 2, wherein the step of completing configuration parameters obtained by custom layout of a plurality of sub-screens on a configuration page through a control specifically comprises:

arranging the control in a configuration area of the configuration page to finish the size and position adjustment of the corresponding sub-picture;

generating corresponding sub-picture vertex coordinates according to the size and the position of the control;

and sequentially obtaining the vertex coordinates of each sub-picture to complete the layout of a plurality of sub-pictures.

4. The unified multi-screen method according to claim 3, wherein the step of arranging the control in the configuration area of the configuration page to complete the adjustment of the size and position of the corresponding sub-screen comprises: and dragging the control to the configuration area, inputting the vertex coordinates of the upper left corner and the size of the control, and finishing the size and position adjustment of the corresponding sub-picture.

5. The unified multi-screen method of any of clauses 3 or 4, wherein the control supports multiple drags to the configuration region;

And supporting inclusion and non-supporting edge intersection between at least two controls of the configuration area.

6. The unified multi-screen method as recited in any one of claims 3 to 5 wherein the step of completing configuration parameters obtained for a custom layout of a plurality of sub-screens on a configuration page through a control further comprises: and when the size or the position of the control is changed, updating the adjusted vertex coordinates of the corresponding sub-picture in real time.

7. The unified multi-screen method as claimed in any one of claims 3 to 6 wherein the step of completing configuration parameters obtained by custom layout of a plurality of sub-screens on a configuration page through a control further comprises: when the configuration area is blank up and down or left and right in the layout of the multiple sub-pictures, the whole layout is centered, and the vertex coordinates of all the sub-pictures are updated.

8. The unified multi-screen method of any one of claims 3-7, wherein the configuration area and the controls are rectangles with an aspect ratio of 16:9;

the configuration parameters also comprise filling parameters of at least one sub-picture;

the step of completing the configuration parameters obtained by the custom layout of the multiple sub-pictures on the configuration page through the control further comprises the following steps: and when at least one sub-picture which is not 16:9 exists in the layout of the sub-pictures, configuring the filling mode of the sub-pictures to obtain filling parameters.

9. The unified multi-picture method as claimed in any one of claims 1 to 8, further comprising:

and storing the configuration parameters as template parameters.

10. The unified multipicture method of claim 9, wherein the method further comprises:

and receiving and storing default setting instructions obtained by setting the layouts of the multiple sub-pictures to default layouts corresponding to the number of the sub-pictures.

11. The unified multi-picture method as claimed in any one of claims 1 to 10 wherein the terminal capabilities include the terminal supporting video streaming, screen size and network environment.

12. The unified multi-picture method as claimed in any one of claims 1 to 11 wherein the configuration parameters comprise correspondence of each sub-picture to video stream content.

13. The unified multi-picture method according to claim 11 or 12, wherein the step of synthesizing at least one video stream according to the configuration parameters and the terminal performance and transmitting the same specifically comprises: judging the minimum number of paths of the supported video streams of all terminals according to the video stream supported by all the terminals, dividing the video stream into the video streams of the supported video streams according to the custom layout, selecting the code rate of the video stream for each terminal according to the screen size and the network environment of each terminal, and transmitting the code rate.

14. The unified multipicture method of claim 13, wherein the step of dividing the video stream into the number of supported video streams according to the custom layout comprises: and dividing a plurality of sub-pictures forming a rectangle in the custom layout into a video stream.

15. The unified multi-picture system is characterized by comprising a server and a conference control, wherein the server comprises a receiving unit, a synthesizing unit and a sending unit, and the conference control comprises a control; wherein the method comprises the steps of

The conference controls the configuration parameters obtained by completing the custom layout of a plurality of sub-pictures on the configuration page through the control;

the receiving unit is used for receiving the configuration parameters;

the synthesizing unit is used for synthesizing at least one path of video stream according to the configuration parameters and the terminal performance;

the sending unit is used for sending the at least one path of video stream.

16. The unified multi-screen system according to claim 15 wherein the control is rectangular;

the configuration parameters include at least two sets of sprite vertex coordinates.

17. The unified multi-screen system according to claim 15 or 16, wherein the conference further comprises an arrangement unit and a generation coordinate unit; wherein the method comprises the steps of

The arrangement unit is used for arranging the control in a configuration area of the configuration page to finish the size and position adjustment of the corresponding sub-picture;

the coordinate generating unit is used for generating vertex coordinates of the corresponding sub-pictures according to the size and the position of the control, and sequentially obtaining the vertex coordinates of each sub-picture, so that the control can complete the layout of a plurality of sub-pictures.

18. The unified multi-screen system according to claim 17 wherein the arrangement unit comprises a drag module and an input module; wherein the method comprises the steps of

The drag module is used for dragging the control to the configuration area, and the input module is used for inputting the vertex coordinates of the upper left corner and the size of the control, so that the arrangement unit completes the size and position adjustment of the corresponding sub-picture.

19. The unified multi-screen system of any of clauses 17 or 18, wherein the control supports multiple drags to the configuration area;

and supporting inclusion and non-supporting edge intersection between at least two controls of the configuration area.

20. The unified multi-screen system of any one of claims 17-19, wherein the generated coordinates unit is further configured to update the adjusted corresponding sprite vertex coordinates in real time when the control size or position changes.

21. The unified multi-screen system according to any one of claims 17 to 20 wherein when the configuration area is blank up and down or left and right in the layout of the plurality of sub-screens, the layout unit is further configured to center the layout as a whole, and the generation coordinate unit is further configured to update the vertex coordinates of all sub-screens.

22. The unified multi-screen system according to any one of claims 17-21, wherein the configuration area and the controls are rectangles with an aspect ratio of 16:9;

the configuration parameters also comprise filling parameters of at least one sub-picture;

the conference control further comprises a filling unit, and when at least one sub-picture which is not 16:9 exists in the layout of the sub-pictures, the filling unit is used for configuring the filling mode of the sub-pictures to obtain filling parameters.

23. The unified multi-screen system according to any one of claims 15 to 22 wherein the server further comprises a storage unit for storing the configuration parameters as template parameters.

24. The unified multi-screen system according to claim 23 wherein the conference control further comprises a default setting unit for obtaining a default setting instruction by setting the layout of the plurality of sub-screens to a default layout corresponding to the number of sub-screens;

The receiving unit and the storage unit are respectively used for receiving and storing the default setting instruction.

25. The unified multi-picture system according to any one of claims 15-24, wherein the terminal capabilities include the terminal supporting video streaming, screen size, and network environment.

26. The unified multi-picture system according to any one of claims 15 to 25 wherein the configuration parameters comprise correspondence of each sub-picture to video stream content.

27. The unified multi-screen system according to claim 25 or 26 wherein the synthesizing unit comprises a judgment path number module, a dividing module, and a selection code rate module; wherein the method comprises the steps of

The judging path number module is used for judging the path number of the least supported video stream of all terminals according to the video stream supported by all the terminals;

the segmentation module is used for segmenting the video streams into the number of the supported video streams according to the custom layout;

the code rate selecting module is used for selecting the code rate of the video stream for each terminal according to the screen size and the network environment of each terminal;

the sending unit is used for sending the video stream after the code rate is selected.

28. The unified multi-picture system according to claim 27 wherein the splitting module is configured to split a plurality of sub-pictures comprising rectangles in the custom layout into a video stream.

Claims (10)

1. A unified multi-picture method, the method comprising:

receiving configuration parameters obtained by completing custom layout of a plurality of sub-pictures on a configuration page through a control;

and synthesizing at least one path of video stream according to the configuration parameters and the terminal performance, and transmitting the video stream.

2. The unified multi-screen method of claim 1, wherein the control is rectangular;

the configuration parameters include at least two sets of sprite vertex coordinates.

3. The unified multi-screen method according to claim 1 or 2, wherein the step of completing configuration parameters obtained by custom layout of a plurality of sub-screens on a configuration page through a control comprises:

arranging the control in a configuration area of the configuration page to finish the size and position adjustment of the corresponding sub-picture;

generating corresponding sub-picture vertex coordinates according to the size and the position of the control;

and sequentially obtaining the vertex coordinates of each sub-picture to complete the layout of a plurality of sub-pictures.

4. The unified multi-screen method as claimed in claim 3 wherein the step of arranging the controls in the configuration area of the configuration page to complete the size and position adjustment of the corresponding sub-screen is specifically: and dragging the control to the configuration area, inputting the vertex coordinates of the upper left corner and the size of the control, and finishing the size and position adjustment of the corresponding sub-picture.

5. The unified multi-screen method of claim 3 or 4, wherein the control supports multiple drags to the configuration area;

and supporting inclusion and non-supporting edge intersection between at least two controls of the configuration area.

6. The unified multi-screen method of any one of claims 3 to 5, wherein the step of completing configuration parameters obtained by custom layout of a plurality of sub-screens on a configuration page through a control further comprises: and when the size or the position of the control is changed, updating the adjusted vertex coordinates of the corresponding sub-picture in real time.

7. The unified multi-screen method of any one of claims 3 to 6, wherein the step of completing configuration parameters obtained by custom layout of a plurality of sub-screens on a configuration page through a control further comprises: when the configuration area is blank up and down or left and right in the layout of the multiple sub-pictures, the whole layout is centered, and the vertex coordinates of all the sub-pictures are updated.

8. The unified multi-screen method of any one of claims 3-7, wherein the configuration area and the controls are rectangles with an aspect ratio of 16:9;

the configuration parameters also comprise filling parameters of at least one sub-picture;

the step of completing the configuration parameters obtained by the custom layout of the multiple sub-pictures on the configuration page through the control further comprises the following steps: and when at least one sub-picture which is not 16:9 exists in the layout of the sub-pictures, configuring the filling mode of the sub-pictures to obtain filling parameters.

9. The unified multi-picture method as claimed in any one of claims 1 to 8, further comprising:

and storing the configuration parameters as template parameters.

10. The unified multi-picture system is characterized by comprising a server and a conference control, wherein the server comprises a receiving unit, a synthesizing unit and a sending unit, and the conference control comprises a control; wherein the method comprises the steps of

The conference controls the configuration parameters obtained by completing the custom layout of a plurality of sub-pictures on the configuration page through the control;

the receiving unit is used for receiving the configuration parameters;

the synthesizing unit is used for synthesizing at least one path of video stream according to the configuration parameters and the terminal performance;

The sending unit is used for sending the at least one path of video stream.