WO2013015816A1

WO2013015816A1 - Positioning video streams from video conferencing sites

Info

Publication number: WO2013015816A1
Application number: PCT/US2011/045711
Authority: WO
Inventors: Mark E. Gorzynski; Michael D. Derocher
Original assignee: Hewlett-Packard Development Company, L.P.
Priority date: 2011-07-28
Filing date: 2011-07-28
Publication date: 2013-01-31

Abstract

Implementations disclosed herein relate to positioning video streams from video conferencing sites. Video conference sites may be assigned a position within a display area, and a position for the video stream windows may be determined within the assigned display area for the associated video conference site. A processor may cause the video stream windows to be displayed according to the determined positions.

Description

POSITIONING VIDEO STREAMS FROM VIDEO CONFERENCING SITES

BACKGROUND

[0001 ] Video conferencing provides a manner for communicating with individuals in remote locations. A video conferencing site may include a camera for capturing the participants at the site and a display for displaying participants at other video conferencing sites.

BRIEF DESCRIPTION OF THE DRAWINGS

[0002] The drawings describe example implementations. The drawings show methods performed in an example order, but the methods may also be performed in other orders. The following detailed description references the drawings, wherein:

[0003] Figure 1 is a block diagram illustrating one example of a computing system.

[0004] Figure 2 is a flow chart illustrating one example of a method to position video streams from multiple video conferencing sites.

[0005] Figures 3A-G are diagrams illustrating one example of positioning video streams from multiple video conferencing sites.

[0008] Figures 4A-F are diagrams illustrating examples of applying spacing rules to position video stream windows for display.

DETAILED DESCRIPTION

[0007] Video conferencing allows people at remote locations to conduct a meeting. For example, people at three separate video conferencing sites may communicate where each site includes a camera for capturing video at the site and transmits the video to the other two sites for display. A video conferencing system may in some cases lack a similar feel to in person communications. To address this issue, a video conferencing system may organize video stream windows for display in a manner intended to preserve non-verbal communication, such as gestures and eye contact.

[0008] In one implementation, video conferencing stream windows are organized for display based on the video conferencing site associated with the video conferencing streams and further based on an order of participants within each of the video conferencing sites. For example, each site may receive video stream windows grouped by video conference site and position them according to an assigned order where video conferencing site groups next to each other in the order are positioned at least one of in the same column or in different rows. The same relative video conferencing site order may be provided to each o the sites participating in the conference to mimic a mutually agreed upon seating arrangement of the sites relative to one another. In one implementation, the display area is divided into a grid structure, and the video conference sites are assigned to grid locations in order such that grid positions in a first column are filled before grid positions in a second column in a "serpentine" type manner. For example, for video conferencing sites in order of 1 , 2, 3, 4, 5 may be displayed at a video conference site 6 with a display area including a three row by two column grid. Video streams 1 , 2, and 3 may be placed in the first column and video conference sites 4 and 5 may be placed in the second column. This may provide a sense of video conference sties 1 , 2, and 3 being to the left of a participant at video conference site 6 and video conference sites 4 and 5 being to the right of a participant at video conference site 6.

[0009] The video stream windows may be positioned in the associated video conferencing site groups where video stream windows of participants next to one another at a video conference site are positioned next to one another. The video stream windows may then be displayed at the particular site according to the determined positions. In one implementation, video stream windows associated with a video conference site may be placed within the grid location associated with the video conference site, in some cases, a video conference site may transmit multiple video conference streams where the video conference streams are each associated with different participants at the particular location. Video stream windows from a particular site may be positioned within the grid location for the video conference site in a particular order that reflects the physical location of the participants at the site. For example, a participant A gesturing to a participant B on participant A^:s right side may be displayed with a video stream window of participant B to the left of a video stream window of participant A such that the gestures at the physical location may be more understandable by participants at remote locations.

[0010] In some cases, associating video stream windows with a grid location where the grid locations have a different size of video streams, such as due to different numbers of video streams in different locations or different aspect ratios of video streams, there may be gaps within the grid of the display area, in one implementation, the spacing between video stream windows is adjusted based on a spacing rule to improve a user's experience with the display setup. The spacing rule may involve, for example, aligning various video streams. In some implementations, the spacing rule may be applied at an individual location, and other video conferencing sites may apply different spacing rules. For example, different sites may have different display sizes or a different number of displays that make a spacing rule tailored to the particular site desirable.

[001 1 ] Grouping the video conferencing windows by site may provide for better non-verbal communication between participants at different sites and may provide viewers with context as to where another participant is located. The organization may also preserve non-verbal communication occurring within a particular site where there are multiple participants at the same site each captured by a different video stream.

[0012] Figure 1 is a block diagram illustrating one example of a video conferencing system 100. The video conferencing system 100 includes n video conferencing sites, a first site 01 , a second site 102, a third site 103, to an nth site 104. The sites may communicate with one another directly via a network 105, such as the Internet, or may communicate with each other via one or more intermediate devices (e.g., a Multipoint Control Unit (MCU). Each video conferencing site may include equipment for capturing video of participants at the video conferencing site and may transmit the video to the other video conferencing sites to be displayed for participants at the other video conferencing sites. For example, participants at the first video conferencing site 101 may view participants at the second video conferencing site 102, the third video conferencing site 103, to the nth video conferencing site 104.

[0013] The first video conferencing site 101 illustrates an example video conferencing site. The first video conferencing site 101 includes a display 106, a camera 107, and an electronic device 108. The display 106 may display video streams received from the network 105 from other video conferencing sites. The display 106 may be, for example, a laptop screen, monitor, or wall display device. In some cases, the first video conferencing site 101 includes multiple displays such that some video streams are displayed on a first display and other video streams are displayed a second display.

[0014] The camera 107 may capture video streams o participants at the first video conferencing site 101 and transmit the video streams to the other video conferencing sites via the network 105. In one implementation, the first video conferencing site 101 may include multiple cameras. The cameras may, for example, capture video of the same participants from different angles or each camera may capture video of a different participant or set of participants.

[0015] The electronic device 108 may be, for example, a server, desktop, laptop, or mobile computing device. The electronic device 100 may include capabilities for organizing video streams for display on the display 106, The electronic device 108 may position received video streams from the other video conference sites to displa them in a manner that enhances non-verbal communication, such as gestures and eye contact.

[0016] The electronic device 108 may include a processor 109 and a machine- readable storage medium 1 10. The processor 109 may be any suitable processor, such as a centra! processing unit (CPU), a semiconductor-based microprocessor, or any other device suitable for retrieval and execution of instructions. In one implementation, the electronic device 108 includes logic instead of or in addition to the processor 109. As an alternative or in addition to fetching, decoding, and executing instructions, the processor 109 may include one or more integrated circuits (!Cs) (e.g., an application specific integrated circuit (ASIC)) or other electronic circuits that comprise a plurality of electronic components for performing the functionality described below. In one implementation, the electronic device 109 includes multiple processors. For example, one processor may perform some functionality and another processor ma perform other functionality described below.

[0017] The machine-readable storage medium 1 10 may be any suitable machine readable medium, such as an electronic, magnetic, optical, or other physical storage device that stores executable instructions or other data (e.g., a hard disk drive, random access memory, flash memory, etc.). The machine-readable storage medium 1 10 may be, for example, a computer readable non-transitory medium. The machine-readable storage medium 1 10 may include instructions executable by the processor 109.

[0018] The machine-readable storage medium 1 10 may include video stream accessing instructions 1 1 1 , video stream group location associating instructions 2, video stream window positioning instructions 1 13, and video stream displaying instructions 1 14. The video stream accessing instructions 1 1 1 may include instructions for accessing groups of video streams where each group of video streams is associated with a video conference site. The number of participants and video streams may vary for each site. For example, the second video conferencing site 102 may transmit two video streams with one participant in each video stream, and the third video conferencing site 103 may transmit three video streams with two participants in each video stream.

[0019] The video stream location groups may include an order. For example, the same relative order may be provided to each of the video conference sites. The order may indicate a meeting order for creating a more personal feel. For example, for a meeting with six sites, the sites may be ordered 1 though 6 where the meeting is organized to provide an illusion of a round fable with participants at site 1 sifting between participants at sites 2 and 6 and participants at site 2 sitting between participants at sites 1 and 3. Information about video conference location order may be received via a network or retrieved from a storage device.

[0020] The received video streams within the site groups may be further ordered based on the physical location of the participants at each site. For example, a site may capture video stream 1 of person A, video stream 2 of person B, and video stream 3 of person C where person B is sitting in the middle of person A and C with person A to B's left and person C to B's right as the three face a camera at the video conferencing site. To reflect the physical/spatial ordering of participants, the corresponding video streams may indicate an order of A, B, C, or C, B, A.

[0021 ] The video stream group location associating instructions 1 12 may include instructions for associating a display location with a video stream group. A display area may be divided into a substantially grid pattern for placing video streams. The grid may^¬ or may not be visible to the user. Each video stream location group may be assigned to a position in the grid. The video stream locations groups may be assigned based on the order of the sites, in one implementation, the video stream sites are assigned such that the video stream sites are assigned in order to a first column in the grid before assigning the video stream sites to a second column in the grid and so on for all of the columns in the grid.

[0022] The video stream positioning instructions 1 13 may include instructions for organizing the accessed video streams for display. For example, the video streams may be organized in a way that preserves the order of participants at each video conference site. Preserving the order of participants in the video conference site may make nonverbal communication between participants at the same site more understandable to participants at other sites. For example, a site may include video stream 1 of person A, video stream 2 of person B, and video stream 3 of person C where person B is sitting in the middle of person A and C with person A to B's left and person C to B's right as the three face a camera at the video conferencing site. The video streams may be placed within the grid location for the site in order left to right of C, B, and A, such as in a row within the grid location, in a manner that preserves non-verbal communication between the three participants. For example, if person A gestures to his right to person B, the gesture will appear in the direction of person B when shown to other video conference sites.

[0023] The video stream displaying instructions 1 14 may include instructions for causing the video streams to be displayed according to the determined organization. For example, the video streams may be displayed on a display area of the display 106 according to the determined positioning information, in one implementation, the processor 109 transmits the determined organization via a network to a display.

[0024] Figure 2 is a flow chart 200 illustrating one example of a method to position video streams from multiple video conferencing sites. Video stream windows may be positioned for display in a manner that enhances non-verbal communication between video conferencing participants. The video streams may be grouped by video conferencing site and organized in order based on the video conferencing site and then based on an order of video streams within the video conferencing site. Non-verbal communication between video conferencing sites may be enhanced, and non-verbal communication among participants at a particular video conferencing site may be more easily interpreted by participants at other locations. The method may be implemented, for example, by the electronic device 108 of Figure 1. For example, one video conferencing site may receive video streams from other video conferencing sites and organize the video streams for display for participants at the particular video conferencing site.

[0025] Beginning at 201 , a processor accesses an ordered list of video conference sites each including an ordered list of video streams. The processor may be associated with a video conferencing site receiving information about video streams from other sites participating in the video conference. The processor may access information about video conference sites, for example, by receiving them via a network or retrieving them from a storage device, information about the order of the video conference sites may be received from a network or retrieved from a storage device.

[0028] In one implementation, each video conference site receives the video conference sites list in the same relative order. An order of video conference sites may be determined prior to a video conferencing meeting, in some cases, the order of the video conference sites may be updated after the start of a meeting, and the position of video streams may be updated accordingly at the video conferencing sites. The order of the video conferencing sites may create an illusion of video conferencing sites around a table, such as by giving an impression of which sites are to a participant's right or left, and the same sense of relative position may be created at other sites where the same relative order is used. The order may be set by a particular site, agreed upon by multiple sites, or determined automatically. For example, one site may select an order that positions a particular participant or site in the middle of the display and send the order to the other sites. [0027] Each of the video conference sites in the ordered list may be associated with one or more video streams. The video streams may images of one or multiple participants and/or specific regions of their corresponding video conferencing sites. For example, a video conference site may include multiple cameras where two people are positioned sitting in front of each camera. In some cases, a video conferencing site may include multiple cameras that capture images of the same participants from different angles such that a single video stream may be created from the different images. Each video conference location may include an ordered list of video streams, and the order ma be based on the relative physical position of participants captured in the video stream.

[0028] Figures 3A-G illustrate examples of positioning video streams from multiple video conferencing sites. The method of Figure 2 is discussed in relation to Figure 3. Figure 3A shows a video conference layout 300. The layout 300 includes six video conferencing sites, sites A, B, C, D, E, and F. The sites are placed in an order to mimic a seating arrangement. For example, video conferencing site A may receive an ordered list of sites B, C, D, E, and F. The layout 300 shows that video streams may be positioned such that participants at site A may perceive participants at site B to their left and participants at site F to their right.

[0029] Figure 3B shows an example video conferencing site E 301 which may be similar to the other video conferencing sites. Video conferencing site E includes four participants, participant W 306, participant X 307, participant Y 308, and participant Z 309. Each of the participants may view on the display area 302 video streams of participants received via a network from other sites. The display area may be made up of one or more displays. Video conferencing site E includes three cameras where each camera captures a video stream of a set of participants. For example, video stream 3E camera 303 captures video of participants W and X, video stream 2E camera 304 captures video of participant Y, and video stream camera 305 captures video of participant Z. The video streams may be transmitted to the other sites for display.

[0030] Figure 3C shows an example 310 of an ordered list of sites that each includes an ordered list of video streams. For example, site A includes video stream 1 A. Site E includes the three video streams in order of video stream 1 E, video stream 2E, and video stream 3E. The video streams are in a "reverse" order to mimic the perspective of participants at other video conferencing sites looking at the participants from the display 302.

[0031 ] Referring back to Figure 2 and continuing to 202, a processor assigns the video conference sites to a display area location in order. The display area may include a portion of a display or a portion of multiple displays. For example, a user may identify a portion of his monitor to be used for displaying video conferencing streams such that the rest of the display is available for other uses. The display area may be organized in a substantially grid-like structure. The grid for organizing the video conferencing sites may be invisible to a user. The grid may be a set of columns or rows, or both columns and rows. The grid may include columns or rows of the same size, or the sizes may be adjusted based on the number or sizes of the video streams such that one grid location is smaller than another grid location.

[0032] The processor may determine a number of rows or columns automatically or based on user input, in some cases, a user may want the video streams to appear in a certain section or percentage of display, such as where a user may simultaneously look at other material on the display. A user may specify a particular number of rows or columns, or the processor may determine a number of rows or columns based on the size of the desired display area. A number of video streams to be displayed and the aspect ratio of the video streams may also be taken into account when determining the grid dimensions.

[0033] The video conferencing sites may be assigned to the display area in order of the video sites. For example, the video conferencing sites may be assigned to the grid where each grid location corresponds to a video conferencing site. A first column in the grid may be filled before video conferencing sites are assigned to a second column in the grid. For example, in a video conferencing meeting with 6 sites in order 1-6, video conferencing sites 1 , 2, and 3 may be assigned to a first column, and video conferencing sites 4, 5, and 6 may be assigned to a second column. In one implementation, the display area is divided into rows such that video conferencing sites are assigned in order to different rows until the last row is filled. At the point where each row has been assigned one video conferencing site, the next video conferencing site is assigned to the first row. The video conferencing site grid location assignment may be maintained through the video conference and in some cases may be preserved for a future video conference with the same sites, in one implementation, multiple video conferencing sites use the same dimensions for a grid for displaying video streams from particular sites, in some cases, other sites may use a different grid pattern but assign the video streams in the same relative order, such as in a column based fashion in a site order used for the conference. Using the same relative order at each site may lead to better non-verbal communication, such as gestures and eye contact, between participants at different sites. [0034] Figure 3D shows an example of a display area grid 31 1 for assigning the video conference sites B-F for display to site A. Display area grid 311 includes a display area of three columns and two rows. The video conference sites are assigned in order such that sites B and C are assigned to column 1 , sites D and E are assigned to column 2, and site F is assigned to column 3. The video conference sites are assigned to a bottom row proceeding to a top row. in an alternative implementation, the video conference sites are assigned to a top row proceeding to a bottom row. The video conference sites may be assigned to the grid in a way that preserves a sense of physical position. For example, layout 300 shows video conferencing site A with video conferencing sites B and C to A^:s left, sites D and E across from site A, and sites F to A^:s right. Similarly, the displa area grid 31 1 when displayed to video conferencing site A shows sites B and C to the left, D and E in the center, and site F to the right.

[0035] Figure 3F shows an example of a display area grid 313 for display at video conference site A with two columns and three rows. The size of the grid may differ, for example, based on user input. The video conference sites are assigned to the display area grid 313 starting at the top row and continuing towards the bottom row where the first column is filled before assigning the video conferencing sites to the second column. For example, video conferencing sites B, C, and D are assigned to the first column, and video conferencing sites E and F are assigned to the second column. Participants at video conferencing site A may perceive video conferencing sites B, C, and D to A's left and video conferencing sites E and F to A^'s right, similar to the layout 300.

[0038] Referring back to Figure 2 and proceeding to 203, a processor assigns video stream windows for displaying the video streams to the display area location of the associated video conference site in order of the video streams for the video conference site. The processor may sequentially assign the video streams the grid location where a video stream of participants physically next to one another at a video conference location site are shown in video stream windows adjacent to one another in the grid location. For example, the video streams from a site may be placed one after another in a row in the grid location. Keeping the video streams in rows according to position at a video conference site may make the gestures and communications between participants at the same site appear more natural and interpretable.

[0037] Figure 3E shows an example of positioning video stream windows within the display area grid 312 after the assignment of the video conferencing sites in the display area grid 31 1 for view from video conferencing site A. Each video stream window is positioned in order in the corresponding grid location for the particular video conferencing site. For example, video stream windows 1 E, 2E, and 3E are assigned to the second column first row in order. Figure 3B shows that if participant Z 308 turns to participant Z's left to gesture to participant Y 308 video stream 1 E will show participant Z looking towards video stream 2E to participant Y. The organization may allow participants at video conferencing site A to interpret non-verbal communication between participants together at a site.

[0038] Figure 3G shows an example of positioning video stream windows within the display area grid 314 where the video conferencing sites are organized according to Figure 3F. The video stream windows are assigned the same position relative to other video stream windows at the same site, but the video stream windows are positioned within different grid positions due to the different grid dimensions compared to Figure 3E.

[0039] Referring back to Figure 2 and continuing to 205, a processor causes the video stream windows associated with the video streams to be displayed according to the assignment. For example, the video stream windows may be displayed on a display device associated with the electronic device, or the electronic device may send the video streams and position information to another electronic device for display. The sites may be labeled on the display such that a viewer may associate a group of video streams with a particular site.

[0040] In some cases, a participant video stream may be added to a site. The added video stream may be displayed in the appropriate grid location according to its order for the site. In some cases, the site order or video stream order may be updated, and the positioning of the video stream windows may also be updated.

[0041 ] The spacing of video stream windows within an assigned grid section may be adjusted based on a spacing rule. The spacing of video stream windows may appear awkward because of the varying number of video streams in each group. The spacing rules may be applied to compensate for spaces created by placing video streams on a grid based on a video conference location grouping. The aspect ratios of the video streams may also vary and make the video stream area of different groups different. The spacing rule may include any rule for updating the position of the video streams within the assigned order and row. The spacing may be adjusted to make it easier for a user to focus on the group of video streams and to make a more cohesive appearance.

[0042] The spacing rule may be any suitable rule for adjusting the spacing between video stream windows. The spacing rule may be a rule for aligning video streams horizontally or vertically, such as where the left or right of video stream windows is aligned. For example, the video stream windows may be left or right justified or centered. In some cases, multiple spacing rules may be applied. For example, video stream windows may be aligned so that they appear in the center of a display area and aligned with one another to the left.

[0043] The spacing rule may be selected automatically, determined based on a set of user preferences or previous behaviors, or selected by a user. The types of spacing rules may be stored in a storage and may be presented to a user for selection, in one implementation, the spacing rules available at one video conferencing site may be different than those at another video conferencing site. In one implementation, some video conferencing sites ma offer spacing adjustments and others may not. The spacing rule may be applied according a local site preference such that different sites position the video streams differently.

[0044] Figures 4A-F are diagrams illustrating examples of applying spacing rules to position video stream windows for display. Other spacing rules are also contemplated. Figure 4A shows a display area 400 that includes three video streams from a first video conference site X on a first grid row and two video streams from a second video conference site Y on a second video row. Figure 4B shows the video streams from Figure 4A in a display area 401 with a spacing rule applied to center the video streams within the display area. Figure 4C illustrates a display area 402 where the video streams of Figure 4A in each row are left justified. For example, the left side of video stream 1 and 4 are aligned. Figure 4D illustrates the video streams of Figure 4A in a display area 404 where a spacing rule for right justifying the rows is applied. For example, the right side of video stream 3 and 5 are aligned. Figure 4E illustrates the video streams of Figure 4A in a display area 405 where a spacing rule is applied to make the rows the same width. For example, there is more spacing between video streams 4 and 5 so that together they are the same width as the row above. Figure 4F illustrates the video streams of Figure 5A in a display area 406 with a spacing rule where a first video stream should occur at a particular position relative to a second video stream. For example, Figure 4F shows a rule where video stream 5 should not occur before video stream 2.

[0045] Video conferencing systems provide useful methods for meeting with people in remote locations. Organizing video stream windows for display based on a site order may lead to better non-verbal communication between participants at different sites. Organizing video streams grouped by each site based on a relative physical location of the participants in the video streams may make it easier for a viewer to interpret communications occurring between participants at a different site. Better non- verbai communication between participants in a video conference may create a video conference with a more natural fee! and may improve the experience of participants.

Claims

CLAIMS 1 , A method, comprising:

determining, by a processor, positions for video stream windows, including:

grouping the video stream windows by video conferencing site;

positioning the video conferencing site groups according to an assigned order wherein video conferencing site groups next to each other in the order are positioned at least one of: in the same column or in different rows;

positioning the video stream windows in the associated video conferencing site groups where video stream windows of adjacent areas of a video conference site are positioned next to one another; and

causing the video stream windows to be displayed according to the determined positions.

2. The method of claim 1 , wherein the assigned order of the video conference site is an assigned meeting order received via a network.

3. The method of claim 1 , further comprising determining an amount of spacing between the video stream windows based on a spacing rule.

4. The method of claim 3, wherein the spacing rule comprises a rule for aligning the video stream windows.

5. An apparatus comprising:

a processor to:

access an ordered list of video conference sites where each site is associated with an ordered list of video streams;

assign the video conference sites to a display area location wherein video conference sites are assigned in order of the ordered list to a first column within the display area before assigning video conference sites to a second column within the display area;

assign video stream windows for displaying the video streams to the associated video conference site display area location in order of the ordered list of the video streams for the video conference site, wherein a second ordered video stream is assigned a position to a first ordered video stream; and cause the video stream windows associated with the video streams to be displayed according to the assignment.

6. The apparatus of claim 5, wherein the order of the video conference sites is based on an order for the video conference.

7. The apparatus o claim 5, wherein the order of the video streams associated with a video conference site is based on the relative physical position of participants in the video streams.

8. The apparatus of claim 5, wherein the processor further adjusts spacing between the video stream windows based on a spacing rule.

9. The apparatus of claim 8, wherein the spacing rule is based on at least one of: a number of video streams in a row or the aspect ratio of video streams in a row.

10. The apparatus of claim 5, wherein a first one of the video conference sites is associated with a single video stream and a second one of the video conference sites is associated with multiple video streams.

1 1. A machine-readable non-transitory storage medium comprising instructions executable by a processor to:

access an ordered list of video conferencing sites associated with an ordered list of video streams wherein the video streams include video of at least one participant at the video conference site;

associate the video conferencing sites with locations in a display area in order of the ordered list of video conferencing sites such that two adjacent groups in the order are in at least one of the same column or different rows;

determine a position of video stream windows of the video streams in order of the ordered list of video streams in the video conferencing site display area location wherein two adjacent video stream windows in the order of the ordered list of video streams are adjacent to one another within the display area location; and

cause the video stream windows to be displayed in the display area based on the determined position.

12. The machine-readable non-transitory storage medium of claim 1 1 , wherein display area location associated with a video conferencing site comprises a grid location.

13. The machine-readable non-transitory storage medium of claim 1 further comprising instructions to: adjust spacing between the video stream windows based on a spacing rule.

14. The machine-readable non-transitory storage medium of claim 13, wherein the spacing rule comprises a rule to space the video stream windows to account for differences in the number of video streams and the aspect ratios of video streams between the groups.

15. The machine-readable non-transitory storage medium of claim 13, wherein the spacing rule comprises at least one of: aligning a video stream window vertically relative to another video stream window or aligning a video stream window horizontally relative to another video stream window.