WO2011140812A1

WO2011140812A1 - Multi-picture synthesis method and system, and media processing device

Info

Publication number: WO2011140812A1
Application number: PCT/CN2010/080320
Authority: WO
Inventors: 孙波; 吴衍平; 田智平; 黄书平
Original assignee: 中兴通讯股份有限公司
Priority date: 2010-05-14
Filing date: 2010-12-27
Publication date: 2011-11-17
Also published as: CN101860715A

Abstract

The present invention discloses a multi-picture synthesis method and system, and a media processing device, which are applied to video conference systems. The multi-picture synthesis method includes the following steps: acquiring multiple paths of media code streams, wherein each path of media code streams is used for displaying one picture of a scene; synthesizing multiple paths of media code streams into a multi-picture group according to a layout, wherein the size of at least one picture in the layout is equal to the actual size of the displayed scene. According to the technical scheme provided by the invention, the actual size of the picture-displaying scene can be kept while multiple conference place information is synthesized as much as possible, which effectively improves the presence experiences of users.

Description

TECHNICAL FIELD The present invention relates to the field of communications, and in particular to a multi-picture synthesis method, system, and media processing apparatus. BACKGROUND OF THE INVENTION The emergence and development of conference television technology has gradually changed the way human beings socialize, and video communication has become an indispensable part of human social and economic life. Traditional communication tools, such as telephones and fax machines, cannot achieve the communication effect of face-to-face or group of people. For enterprises and institutions, business trips have become an annoying and prohibitive business. The use of conference television has not only achieved the purpose of convening a conference, but also avoided traveling to the field. The video conference system mainly includes a multipoint control unit (MCU) and a terminal system. The MCU is a key device of the multipoint video conference system, and extracts audio from the information flow from each conference site. Video, data and other information and signaling, and then send the information and signaling of each conference site to the multi-point control module and the media processing module respectively, to complete the corresponding audio mixing or switching, frequency mixing or switching, data broadcasting and routing. Processes such as selection, timing, and conference control, and finally reassembling the various information required for each conference site and sending it to each terminal system device. The terminal system is divided into two types: desktop conference terminal and conference room conference terminal. The desktop conference terminal is low in cost and easy to use, and is suitable for personal office use and small-scale conferences. The conference room type terminal is equipped with high-quality zoom lens, high-fidelity audio, large-screen color TV or projection and other external auxiliary equipment, plus video pre/post processor, which makes the picture quality clearer and achieve better conference results. Suitable for holding large-scale conference room terminals suitable for conference rooms, several to dozens of participants' environment. A broadcast terminal is a one-way receiving terminal that can receive images and sounds of a conference, but cannot transmit images and sounds. The broadcast terminal can be used in situations where only one-way information needs to be transmitted, such as a policy in which the superior level conveys a policy to the lower level. The mobile terminal is based on the desktop terminal, and is equipped with a wireless access card and a wireless transmitting device to move into the conference within a certain area. The conference terminal is configured in the local conference site and each venue and conference site in the video conference. In the traditional conference TV, through the close cooperation of the MCU and the terminal system, the simultaneous output of multiple pictures is realized in a split manner. Symmetrical split from 4, 9, 16 to 1 + 5, 1 + 7, 3 + 4, 2 + 8, 1 + 12 and other asymmetric exhibitions, from 1 + 1, 1 + 2 and so on to 6, 2 + 4, etc. 16: 9 widescreen display, a variety of different split screen combinations, to meet the user's arbitrary Display requirements. The multi-screen layout of a conventional conference television display will be described below with reference to FIG. The layout is 9 screens, and each screen has the same size, which is 1/9 of the entire screen size. Traditional venues do not require real size, and they are expected to see more venue information. 9 screens Each row of 3 small screens can display a multi-stream venue, and 9 screens can display 3 multi-stream venues. It can be seen that although the user can see as many sites as possible, the user can not maintain the true size of the displayed scene, and it is difficult to effectively communicate with the eyes, thus reducing the user's presence experience. SUMMARY OF THE INVENTION In view of the problem in the related art that a plurality of scenes cannot be synthesized as much as possible, and the true size of the scene is maintained, the main object of the present invention is to provide a multi-picture synthesis method, system, and media processing apparatus to solve At least one of the above issues. According to an aspect of the present invention, a multi-picture synthesis method is provided. According to the multi-picture synthesis method of the present invention, the method is applied to a video conference system, the method comprising: acquiring a multi-channel media code stream, wherein each media code stream is used to display one picture of the scene; and the multi-channel media stream is arranged according to the layout. The composition is a set of multiple pictures, wherein the size of at least one of the pictures is equal to the true size of the displayed scene. According to another aspect of the present invention, a multi-picture composition system is provided. The multi-picture composition system according to the present invention includes: an access module, configured to acquire a multi-channel media code stream, wherein each media code stream is used to display one picture of the scene; and a media processing module is configured to multi-channel according to the layout The media stream is synthesized into a set of multiple pictures, wherein the size of at least one of the frames is equal to the true size of the displayed scene. According to still another aspect of the present invention, a media processing apparatus is provided. The media processing device according to the present invention includes: a receiving module, configured to receive a multi-channel media code stream, wherein each media code stream is used to display one picture of the scene; and a processing module, configured to receive the received multi-channel media code according to the layout Streaming into a set of multiple pictures, where at least one of the frames is in the layout The size is equal to the true size of the displayed scene. An output module for outputting a combined set of multi-pictures. According to the present invention, the multiplexed media streams are combined into a plurality of frames according to a predetermined layout, wherein the size of at least one of the layouts is equal to the real size of the displayed scene, and the related art cannot be solved as much as possible. When a plurality of scenes are synthesized, the problem of the true size of the scene is also maintained, and in addition, a plurality of venue information can be synthesized as much as possible, and the real size of the scene can be maintained, thereby effectively improving the user's presence and danger. BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are set to illustrate,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, In the drawings: FIG. 1 is a multi-screen layout of a conventional conference television display; FIG. 2 is a schematic structural diagram of a video conference system; FIG. 3 is a flowchart of a multi-screen synthesis method according to an embodiment of the present invention; BRIEF DESCRIPTION OF THE DRAWINGS FIG. 5 is a schematic diagram of a multi-screen layout according to a preferred embodiment 2 of the present invention; FIG. 6 is a multi-group multi-screen according to a preferred embodiment 3 of the present invention; Figure 7 is a flow chart of a multi-picture synthesis method according to a preferred embodiment of the present invention; Figure 8 is a block diagram showing the structure of a multi-picture synthesis system according to an embodiment of the present invention; A block diagram of a multi-picture synthesis system of a preferred embodiment of the invention; FIG. 10 is a block diagram showing the structure of a media processing device in accordance with a preferred embodiment of the present invention. BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. It should be noted that the embodiments in the present application and the features in the embodiments may be combined with each other without conflict. 2 is a schematic structural diagram of a video conference system. As shown in FIG. 2, the video conference system includes: a single and multiple video stream conference site and a multipoint control unit (MCU for short) 13 . Among them, 111, 112, and 113 are multi-stream venues, and 121, 122, and 123 are single-stream venues. Among them, 111 venues are broadcast source venues, other venues are watching 111 venues; 111 are viewing venues 113 venues. The single-stream sites 121, 122, and 123 can only select one stream of multiple streams in the 111 site, and there is a defect that the information is incomplete. Therefore, the media processing module needs to synthesize the multiple streams into one stream. Although the 111 venue is a multi-stream venue, it can only see one multi-stream venue. It also needs a media processing module to combine other site code streams into one stream. In the above video conferencing system, if the user needs to see multiple site information as much as possible, and can also see the real size of the site scene displayed on the screen, it is necessary to provide a new multi-screen synthesis method, which is described below in conjunction with FIG. description. 3 is a flow chart of a multi-screen synthesis method in accordance with an embodiment of the present invention. As shown in FIG. 3, the multi-screen synthesis method includes the following processes: Step S302: Acquire a multi-media media code stream, where each media code stream is used to display one picture of the scene; Step S304: multiplex media code stream according to the layout The composition is a set of multiple pictures, wherein the size of at least one of the pictures is equal to the true size of the displayed scene. According to the above method, since the size of at least one of the above-mentioned layouts is equal to the real size of the displayed scene, the screen viewed by the user can maintain the true size of the displayed scene, and can also effectively perform eye contact, thereby improving the user's presence. Experience. Preferably, there may be at least one small picture in the multi-picture, wherein the size of each small picture is smaller than the real size of the displayed picture. Through the above processing, in one set of multiple pictures, one or more scenes of a real body size may be included, or one or more pictures having a size smaller than the real size of the displayed scene may be included, so that as many synthesized as possible can be achieved. Scenes, while maintaining the true size of the scene. The above preferred implementation process will be described below in conjunction with FIGS. 4 and 5. 4 is a schematic diagram of a multi-screen layout according to a preferred embodiment 1 of the present invention; as shown in FIG. 4, the layout is composed of 4 screens, wherein 414 corresponds to a large screen, and 411 is superimposed in the middle of the bottom thereof. 412, 413 a total of 3 aspect ratio 16:9 small picture, the size of the small picture, 1/36 in the large picture Between 1/16. The 411, 412, and 413 media sources are from the same multi-stream site or multiple single-stream sites. It can be seen that the 414 maintains the true size and does not affect the eye contact, and the 411, 412, and 413 can see the image information of other venues. FIG. 5 is a schematic diagram of a multi-screen layout according to a preferred embodiment 2 of the present invention; as shown in FIG. 5, the layout differs from FIG. 4 mainly in that a small screen is superimposed in the middle of the top. As in Fig. 4, the large screen 52 maintains the true size, does not affect the eye contact, and 511, 512, and 513 can see the image information of other venues. Preferably, according to the state of the conference, the number of multi-screen groups that need to be synthesized can be dynamically determined, and each group of multi-screens is displayed by one screen. In a preferred implementation process, a conventional multi-screen layout may also be used in the multiple sets of multiple pictures. For details, refer to FIG. 1. The above preferred implementation process will be described below in conjunction with FIG. FIG. 6 is a schematic diagram of a multi-group multi-screen layout according to a preferred embodiment 3 of the present invention; in this embodiment, FIG. 2 is taken as an example to show image information after multi-screen synthesis in different meeting places. 611 is a schematic diagram of the code stream viewed from the left screen of 111, and the images are from the left left seat of 113 and the venues of 121, 122, and 123 respectively. 612 is a schematic diagram of the code stream seen in the 111 exhibition, and the images are from the 113 seats and the 112 site images respectively. 613 is a schematic diagram of the code stream viewed from the right screen of 111, and the image is from the 113 right-seat image. It can be seen that the image information of the large screen of 111 is from 113, its size is kept true, and 112, 121, 122, 123 venue images are superimposed. 621, 622, and 623 are the image information of 113, and the large-screen media source is the 111 site, and the multi-screen superimposed 112 site information is synthesized on 622, so 113 can simultaneously view the 111 and 112 site videos. 63 is the nine screen images of the 121, 122, and 123 single-flow meeting places, and the media code streams are from the three conference sites of 111, 112, and 113 respectively. Preferably, the method may further include: processing, in the conference, determining to change the current layout according to the conference state; and synthesizing the multi-media media streams into a group of multiple images according to the changed layout. Preferably, the method may further include the following processing: determining a more according to the state of the conference in the conference? The code stream source; acquires a new multi-media stream from the new stream source; synthesizes the new multi-media code 3⁄43⁄4 into a set of multi-pictures according to the layout. According to the state of the conference, flexible changes to the current layout or code stream source can effectively improve the user's viewing experience. The above preferred implementation process will be described below in conjunction with FIG. 7 is a flow chart of a multi-picture synthesis method in accordance with a preferred embodiment of the present invention. As shown in FIG. 7, the multi-screen synthesis method includes the following processing: Step S702: During the conference holding process, according to the MCU image synthesis capability and the characteristics of the conference terminal, it is determined to synthesize several groups of multiple screens, and what layout is used for each group of multiple screens . Among them, the MCU image synthesis capability refers to how many multi-pictures can be synthesized, and also includes transcoding capabilities between different protocols at different rates. When the ability is limited, first ensure the multi-screen synthesis of the chairman's venue. The layout of multi-screen use can be selected according to the actual situation, the layout shown in Figure 1, Figure 4, Figure 5. Ordinary venues generally use the layout of Figure 1, and the telepresence venues generally use the layout of Figure 4 or Figure 5. In addition, the layout and number of frames can be dynamically adjusted during the meeting. Step S704: In the conference, according to the state of the conference, change which terminal stream is synthesized by the multi-screen at any time.

(Equivalent to the above in the meeting according to the state of the meeting, to determine the source of the code stream). The broadcast source change may occur in the conference, and the media stream synthesized by each group of multiple pictures will change accordingly. In the layout of Figure 4 and Figure 5, select the broadcast source site stream or the site stream viewed by the broadcast source. The small screen selects other site streams and fills them in order. The media code stream in the layout of Figure 1 can be specified by the user, or multiple sites that have recently spoken. Step S706: Select different media code streams to be transmitted to the site according to the characteristics of each site. The site can directly view the scenes of other sites through multiple streams, or multiple screens synthesized by other sites. Specifically, it can be determined according to actual conditions. In a preferred implementation, at least one of the following may be adjusted in real time: multi-picture group number, multi-picture layout, media source. Specifically, the system may dynamically adjust according to a predetermined configuration, or may dynamically adjust in response to a user's adjustment instruction. Fig. 8 is a block diagram showing the structure of a multi-picture synthesizing system according to an embodiment of the present invention. As shown in FIG. 8, the multi-screen synthesis system is applied to a video conference system, and includes: an access module 80 and a media processing module 82. The access module 80 is configured to acquire a multi-media media stream, where each media code stream is used to display one picture of the scene, and the media processing module 82 is configured to synthesize the multiple media code streams into a set of multiple pictures according to the layout. The size of at least one of the screens in the layout is equal to the true size of the displayed scene. According to the above system, since the size of at least one of the above-mentioned layouts is equal to the real size of the displayed scene, the screen viewed by the user can maintain the true size of the displayed scene, and can also effectively perform eye contact, thereby improving the user's presence. Experience. Preferably, the access module 80 is further configured to output a set of multiple pictures processed by the media processing module to the terminal. Preferably, as shown in FIG. 9, if there are multiple access modules and media processing modules, the system may further include: a media switching module 84, configured to forward the media code stream acquired by the access module to the access module The corresponding media processing module outputs the multi-screen processed by the media processing module to the access module corresponding to the media processing module. It should be noted that the preferred embodiments of the foregoing modules in the multi-screen synthesis system may be referred to the description in FIG. 3 to FIG. 7, and details are not described herein again. Figure 10 is a block diagram showing the structure of a media processing device in accordance with a preferred embodiment of the present invention. As shown in FIG. 10, the media processing device mainly includes: a receiving module 10, a processing module 12, and an output module 14. The receiving module 10 is configured to receive a multi-media media stream, where each media code stream is used to display a picture of the scene. In a preferred implementation, the receiving module 10 receives the terminal code stream, where the terminal code stream passes through the network and The MCU access module enters the MCU. The processing module 12 is configured to synthesize the received multi-media media code streams into a set of multiple pictures according to a layout, wherein a size of at least one of the pictures is equal to a real size of the displayed scene. In a preferred implementation, the processing module 12 synthesizes the multi-picture according to the picture layout and the multi-picture stuffing stream information. The processing module 12 is further configured to perform code stream protocol processing and rate conversion. The output module 14 is configured to output the combined set of multi-pictures. The media processing device may synthesize the received multi-media media code streams into a plurality of multi-pictures according to a predetermined layout, wherein at least one of the above-mentioned layouts has a size equal to a real size of the displayed scene, and thus is processed by the media processing device. The latter image can maintain the true size, does not affect the eye contact, and effectively improves the user experience. Preferably, there may be at least one small picture in the multi-picture, wherein the size of each small picture is smaller than the real size of the displayed picture. The layout of the multi-picture can be specifically seen in FIG. 4 and FIG. 5. Through the above processing, the user can see the screen of multiple scenes as much as possible, and effectively improve the viewing experience of the user. In summary, with the above embodiments provided by the present invention, it is possible to synthesize a plurality of site information as much as possible, and maintain the true size of the scene displayed on the screen, thereby facilitating eye contact between users and effectively improving the user's presence. Body-risk. Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general-purpose computing device, which can be concentrated on a single computing device or distributed over a network composed of multiple computing devices. Alternatively, they may be implemented by program code executable by the computing device, such that they may be stored in the storage device by the computing device and, in some cases, may be different from the order herein. The steps shown or described are performed, or they are separately fabricated into individual integrated circuit modules, or a plurality of modules or steps are fabricated as a single integrated circuit module. Thus, the invention is not limited to any specific combination of hardware and software. The above is only the preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes can be made to the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the scope of the present invention are intended to be included within the scope of the present invention.

Claims

Claim

A multi-screen synthesis method, which is applied to a video conference system, and includes: acquiring a multi-channel media code stream, wherein each media code stream is used to display a picture of the scene;

The multi-media media stream is synthesized into a set of multi-pictures according to a layout, wherein a size of at least one of the layouts is equal to a real size of the displayed scene.

The method according to claim 2, wherein the multi-picture has at least one small picture, wherein a size of each small picture is smaller than a real size of the displayed picture.

The method according to claim 1 or 2, further comprising:

According to the state of the conference, the number of multi-screen groups that need to be synthesized is determined, and each of the plurality of screens is displayed on one screen.

4. The method according to claim 1, further comprising:

In the meeting, according to the state of the meeting, it is determined to change the layout;

The multi-media media streams are combined into a set of multi-pictures according to the changed layout.

5. The method according to claim 1, further comprising:

Determine the source of the change code stream according to the status of the conference during the conference;

Obtaining a new multi-media media code stream from the changed code stream source;

The new multi-media stream is synthesized into a set of multi-pictures according to a layout.

A multi-screen synthesis system, which is applied to a video conference system, and includes: an access module, configured to acquire a multi-channel media code stream, where each media code stream is used to display a picture of the scene;

And a media processing module, configured to synthesize the multiple media code streams into a set of multiple pictures according to a layout, wherein a size of at least one of the pictures is equal to a real size of the displayed scene.

7. The system of claim 6 wherein:

The access module is further configured to output the set of multiple pictures processed by the media processing module to the terminal.

The system of claim 7, wherein the access module and the media processing module are both, and the system further includes:

a media switching module, configured to forward the media code stream obtained by the access module to a media processing module corresponding to the access module, and output the multi-screen processed by the media processing module to a corresponding one of the media processing module Access module.

A media processing device, applied to a video conferencing system, comprising: a receiving module, configured to receive a multi-media media code stream, where each media code stream is used to display a picture of a scene;

a processing module, configured to synthesize the received multi-media media stream into a plurality of multi-screens according to a layout, wherein a size of at least one of the layouts is equal to a real size of the displayed scene;

And an output module, configured to output the synthesized set of multiple pictures.

10. The apparatus according to claim 9, wherein the multi-picture has at least one small picture, wherein a size of each small picture is smaller than a real size of the displayed picture.