JP2014050018A - Information processing device and video conference system and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing device and a video conference system and program for realizing smoother communication.SOLUTION: The present invention is an information processing device connected via a network to video conference devices installed in a plurality of places, the information processing device comprising: a receiving unit for receiving an image captured by an image pickup unit of the video conference device; an analysis unit for analyzing the gaze direction of a person in the image; a calculation unit for calculating, on the basis of the gaze direction, an attention degree indicating the degree of attention paid per unit time in each screen region of the screen layout of a display unit of the video conference unit watched by the person; a change unit for changing the screen layout on the basis of the attention degree; and a distribution unit for distributing information indicating the screen layout and a plurality of video data shown in a plurality of screen regions included in the screen layout to the video conference device in which the person was image-captured.

Description

  The present invention relates to an information processing apparatus, a video conference system, and a program.

  When communicating with a remote party, a video conference system using a dedicated line, a web conference system using the Internet, a telepresence system, and the like are used in addition to the conventional telephone and fax. The telepresence system outputs a life-size image with a more realistic feeling on a large display or the like. With the spread of the Internet and large displays, more direct communication is possible compared to conventional telephones and fax machines.

  In a general video conference system, the state of the other party during the conference is displayed in accordance with the size of the display screen. For example, Patent Document 1 is known as a technique for simultaneously displaying the appearances of multiple sites on one screen. Patent Document 1 discloses an image display device that displays an image with high visibility only in a screen region that is noticed by a user. Thereby, low power consumption of the image display apparatus is realized.

  In a general video conference system, when the state of multiple locations is displayed simultaneously on one screen, the video size per location is reduced. For this reason, a function for enlarging and displaying a screen on which a partner in a meeting is displayed has been known.

  Examples of functions for enlarging and displaying the display screen include a manual layout switching function and a display enlarging function linked with a speaker emphasizing function. The manual layout switching function is a function in which the user manually switches the screen layout according to the manual procedure of the video conference system. The enlarged display function linked with the speaker emphasizing function is a function for detecting a speaker who is talking in the video conference system and expanding a screen area on which the speaker is displayed.

  However, the conventional manual layout switching function requires an operation according to the manual procedure every time the user of the video conference system wants to enlarge the screen area. This hinders smooth communication between users of the video conference system. Further, the screen area that the user wants to enlarge is not necessarily the screen area on which the speaker who is currently talking is displayed. Therefore, even if the video conference system has an enlarged display function that is linked with the speaker emphasis function, the user may have to manually enlarge the screen area.

  Moreover, the image display apparatus of patent document 1 aims at the reduction in power consumption of an image display apparatus. Patent Document 1 does not mention improvement of the user interface related to a change in the screen layout of the video conference apparatus, a change in the size of the video display area, or the like.

  The present invention has been made in view of the above, and an object thereof is to provide an information processing apparatus, a video conference system, and a program that realize smoother communication.

  In order to solve the above-described problems and achieve the object, the present invention is an information processing apparatus connected to video conferencing apparatuses installed in a plurality of places via a network, and an imaging unit of the video conferencing apparatus Included in the screen layout of the video conferencing apparatus in which the person is imaged based on the line-of-sight direction, a receiving unit that receives the image captured by the image processing unit, an analysis unit that analyzes the line-of-sight direction of the person in the image For each of the plurality of screen areas, a calculation unit that calculates a degree of attention indicating a degree of attention per unit time, a change unit that changes the screen layout based on the degree of attention, and information that indicates the screen layout A distribution unit that distributes a plurality of video data displayed in the plurality of screen areas included in the screen layout to the video conference device in which the person is imaged. Obtain.

  The present invention is also a video conference system in which video conference devices installed in a plurality of places and information processing devices are connected via a network, wherein the information processing devices are captured by an imaging unit of the video conference device. A reception unit that receives the captured image, an analysis unit that analyzes the line-of-sight direction of the person in the image, and a plurality of images included in the screen layout of the video conference device in which the person is captured based on the line-of-sight direction A calculation unit that calculates a degree of attention indicating the degree of attention per unit time, and a change unit that changes a screen layout of the video conference device in which the person is captured based on the degree of attention The person is imaged with information indicating the screen layout and a plurality of video data displayed in the plurality of screen areas included in the screen layout. And a delivery unit for delivering the video conferencing device.

  Further, the program of the present invention is based on the line-of-sight direction, a reception unit that receives an image captured by the imaging unit of the video conference apparatus, an analysis unit that analyzes the line-of-sight direction of a person in the image, and A calculation unit that calculates a degree of attention indicating a degree of attention per unit time for each of a plurality of screen regions included in the screen layout of the video conference device in which the person is captured, and based on the degree of attention, A change unit that changes a screen layout of the video conference device in which the person is imaged, information indicating the screen layout, and a plurality of video data displayed in the plurality of screen areas included in the screen layout, And a distribution unit that distributes the image to the video conference device in which a person is imaged.

  According to the present invention, it is possible to provide an information processing apparatus, a video conference system, and a program that realize smoother communication.

FIG. 1 is a diagram illustrating an example of a connection relationship between the information processing apparatus and the video conference apparatus according to the present embodiment. FIG. 2 is a block diagram illustrating an example of a control unit of the information processing apparatus according to the present embodiment. FIG. 3 is a diagram illustrating an example of an imaging unit and a display unit of the video conference apparatus according to the present embodiment. FIG. 4 is a sequence diagram for explaining an example of a method for changing the screen layout of the information processing apparatus according to the present embodiment. FIG. 5 is a diagram illustrating an example of a screen layout at the start of the video conference of the video conference apparatus according to the present embodiment. FIG. 6 is a diagram illustrating an example of the degree of attention calculated by the calculation unit of the information processing apparatus according to the present embodiment. FIG. 7 is a diagram showing a screen layout pattern A according to the present embodiment. FIG. 8 is a diagram showing a screen layout pattern B according to the present embodiment. FIG. 9 is a diagram showing a screen layout pattern C of the present embodiment. FIG. 10 is a flowchart for explaining an example of a screen layout pattern selection method of the information processing apparatus according to the present embodiment. FIG. 11 is a diagram illustrating an example of a screen layout based on the example of the degree of attention in FIG. FIG. 12 is a diagram illustrating an example of a hardware configuration of the information processing apparatus according to the present embodiment.

  Hereinafter, embodiments of an information processing apparatus, a video conference system, and a program will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a diagram illustrating an example of a connection relationship between the information processing apparatus 1 and the video conference apparatuses 2a to 2i according to the present embodiment. As shown in FIG. 1, the information processing apparatus 1 and the video conference apparatuses 2 a to 2 i installed at the bases A to I are connected via a network 3. Hereinafter, when it is not necessary to distinguish the video conference apparatuses 2a to 2i, they are simply referred to as the video conference apparatus 2. The number of video conference apparatuses 2 connected to the network 3 may be an arbitrary number. The network 3 is the Internet or the like. The communication means between the information processing device 1 and the video conference device 2 may be wired or wireless.

  The information processing apparatus 1 implements a video conference system in cooperation with the video conference apparatus 2 installed at each of the bases A to I. The information processing apparatus 1 according to the present embodiment includes a control unit 11 and a distribution unit 12 in order to perform a screen layout change process that realizes smoother communication. The control unit 11 controls the screen layout change based on the line of sight of the person in the image transmitted from the video conference apparatus 2. Details of the control unit 11 will be described later. The distribution unit 12 distributes information indicating the screen layout and each video data displayed in any screen area of the screen layout to the video conference apparatus 2 in which the person is captured.

  FIG. 2 is a block diagram illustrating an example of the control unit 11 of the information processing apparatus 1 according to the present embodiment. The control unit 11 of the information processing apparatus 1 includes a reception unit 111, an analysis unit 112, a calculation unit 113, and a change unit 114.

  The reception unit 111 receives a user image captured by the imaging unit 21 of the video conference apparatus 2. Here, the video conference apparatus 2 will be described. The configuration of the video conference apparatus 2 according to the present embodiment is the same as that of a general video conference apparatus 2. That is, the video conference apparatus 2 includes a communication I / F for connecting to the network 3, a codec that compresses and combines video data and audio data, and the like. The video conferencing apparatus 2 may be a video conferencing apparatus, or may be general-purpose as long as it has hardware (camera, monitor, speaker, microphone, etc.) necessary for conducting a video conference and video conferencing software. A personal computer or the like may be used.

  FIG. 3 is a diagram illustrating an example of the imaging unit 21 and the display unit 22 of the video conference apparatus 2 according to the present embodiment. The display unit 22 displays the video conference video captured by the video conference devices 2 a to 2 i based on the screen layout determined by the control unit 11 of the information processing device 1. The imaging unit 21 captures an image of a user who views the display unit 22.

  Returning to FIG. 2, the analysis unit 112 analyzes the user image received by the reception unit 111 and detects the gaze direction of the person in the image. For example, the analysis unit 112 analyzes the position of the pupil of the user's eyes from the user's image, and detects the line-of-sight direction based on the direction of the pupil. The method of detecting the line-of-sight direction from the image is not limited to this, and any conventionally used method can be applied. Note that the analysis unit 112 acquires in advance calibration data determined based on the positional relationship between the imaging unit 21, the display unit 22, and the user in order to improve the detection accuracy of the line-of-sight direction. May be. Alternatively, the calibration data may be acquired from the video conference apparatus 2 before the video conference is started.

  Based on the line-of-sight direction analyzed by the analysis unit 112, the calculation unit 113 indicates the degree of attention indicating the degree of attention per unit time for each screen area of the screen layout of the video conference apparatus 2 in which the person is captured. Is calculated. Details of the attention level will be described later.

  The changing unit 114 changes the screen layout of the display unit 22 of the video conference apparatus 2 in which the person is captured based on the attention level of each screen area calculated by the calculating unit 113. For example, the changing unit 114 changes the screen layout so that a screen area with a higher degree of attention is easier to see. Details of the method of changing the screen layout will be described later.

  Note that the control unit 11 (the receiving unit 111, the analyzing unit 112, the calculating unit 113, and the changing unit 114) and the distributing unit 12 of the information processing apparatus 1 according to the present embodiment may be realized by software. Further, it may be realized by hardware such as an IC. Moreover, you may implement | achieve combining software and hardware.

  Further, the control unit 11 and the distribution unit 12 of the information processing apparatus 1 may be realized by distributing to two or more devices instead of one device in order to achieve the purpose such as load distribution. The same applies to the control unit 11 (the receiving unit 111, the analyzing unit 112, the calculating unit 113, and the changing unit 114).

  FIG. 4 is a sequence diagram for explaining an example of a method for changing the screen layout of the information processing apparatus 1 according to the present embodiment.

  When receiving an input operation indicating the start of the video conference from the user, the video conference apparatus 2 starts the video conference (step S11).

  The imaging unit 21 of the video conference apparatus 2 starts capturing a user image of the user who views the display unit 22. The video conference device 2 transmits the user image to the information processing device 1 (step S12).

  The distribution unit 12 of the information processing apparatus 1 distributes the information indicating the initial screen layout and the video data of each of the bases A to I displayed in any screen area of the screen layout to the video conference apparatus 2. (Step S13). FIG. 5 is a diagram illustrating an example of a screen layout at the start of the video conference of the video conference apparatus 2 according to the present embodiment. In the example of FIG. 5, the images of the bases A to I are displayed in a uniform size.

  Returning to FIG. 4, the receiving unit 111 of the information processing apparatus 1 receives a user image from the video conference apparatus 2 (step S14).

  The analysis unit 112 of the information processing apparatus 1 analyzes the user's line-of-sight direction based on the user image (step S15). The analysis unit 112 calculates the intersection point with the screen by extending the user's line-of-sight direction, and detects which screen region the intersection point is included in.

  Based on the line-of-sight direction, the calculation unit 113 of the information processing device 1 calculates the degree of attention indicating the degree of attention per unit time by the user for each screen area of the screen layout of the display unit 22 of the video conference device 2. (Step S16). The calculation unit 113 calculates the attention level of the base X by (time when attention is paid to the screen area where the video of the base X is displayed) / (unit time). For example, when the user pays attention to the screen area in which the video of the site A is displayed per unit time (10 seconds) for 1 second, the calculation level of the site A is 0.1. (10%) is calculated. FIG. 6 is a diagram illustrating an example of the degree of attention calculated by the calculation unit 113 of the information processing apparatus 1 according to the present embodiment. In FIG. 6, the attention degree of the base A is 10%, the attention degree of the base B is 70%, the attention degree of the base C is 15%, the attention degree of the base D is 5%, and the base E It shows that the attention degree of ˜I is 0%.

  Returning to FIG. 4, the changing unit 114 of the information processing device 1 changes the screen layout of the video displayed on the display unit 22 of the video conference device 2 based on the degree of attention (step S <b> 17). The changing unit 114 may change the screen layout by selecting one screen layout pattern from a predetermined screen layout pattern. FIG. 7 is a diagram showing a screen layout pattern A according to the present embodiment. FIG. 8 is a diagram showing a screen layout pattern B according to the present embodiment. FIG. 9 is a diagram showing a screen layout pattern C of the present embodiment. A method for selecting a screen layout pattern will be described later with reference to FIG.

  Returning to FIG. 4, the distribution unit 12 of the information processing device 1 receives the information indicating the screen layout changed in step S <b> 17 and the video data of each of the bases A to I displayed in any screen area of the screen layout. Then, it is distributed to the video conference apparatus 2 (step S18). The distribution unit 12 changes the distribution data amount of the video data displayed in each screen area to a distribution data amount corresponding to the degree of attention (data amount corresponding to the size of the screen area) and distributes the distribution data. Also good.

  The distribution unit 12 requests the control unit 11 to perform a screen layout change control process after 10 seconds have elapsed since the video having the changed screen layout is distributed to the video conference apparatus 2 (step S19). The interval for requesting the screen layout change control process is not limited to 10 seconds. Further, this interval may be changed as appropriate according to the setting of the information processing apparatus 1.

  Next, a method for selecting a screen layout pattern in changing the screen layout in step S17 will be described. FIG. 10 is a flowchart for explaining an example of a screen layout pattern selection method of the information processing apparatus 1 according to the present embodiment.

  The changing unit 114 of the information processing apparatus 1 changes the screen layout using a plurality of determination conditions that are conditions of interest. For example, the changing unit 114 changes the screen layout to be determined according to the determination condition that the calculated degree of attention matches. The changing unit 114 determines whether there is a screen area with an attention degree of 50% or more among the attention degrees of the screen areas (video display areas of the respective bases A to I) (step S21). The change unit 114 selects the screen layout pattern A in FIG. 7 when there is a screen area with a degree of attention of 50% or more (step S22). If there is no screen area with a degree of attention of 50% or more, the process proceeds to step S23.

  The changing unit 114 determines whether there are two or more screen areas having a degree of attention of 20% or more among the degrees of attention of each screen area (video display areas of the respective bases A to I) (step S23). When there are two or more screen areas having a degree of attention of 20% or more, the changing unit 114 selects the screen layout pattern B in FIG. 8 (step S24). The change unit 114 selects the screen layout pattern C in FIG. 9 when there are no two or more screen regions with an attention degree of 20% or more (step S25).

  FIG. 11 shows a case where the screen layout pattern selection method of the information processing apparatus 1 according to the present embodiment is applied to an example of the degree of attention shown in FIG. FIG. 11 is a diagram illustrating an example of a screen layout based on the example of the degree of attention in FIG. FIG. 11 shows an example in which the screen layout pattern A is applied to the screen layout of the display unit 22 of the video conference apparatus 2 by the processes in steps S21 and S22. The image of the base B with the highest degree of attention is displayed in the largest screen area. Further, the video of the base C with the second attention level, the video of the base A with the third level of attention, and the video of the base D with the fourth level of attention are displayed as screen areas of the same size. In addition, the images of the bases E to I having the fifth attention degree are not displayed.

  Note that the method of changing the screen layout based on the attention level by the changing unit 114 is not limited to the method described above. The changing unit 114 may change the arrangement of the image of the screen area with a high degree of attention to a position that is easy for the user to view, such as the center of the display unit 22 of the video conference apparatus 2 without changing the size. Further, the changing unit 114 may dynamically change the size of the screen area according to the degree of attention of the screen area. For example, if the degree of attention is 100%, the entire area of the display unit 22 is occupied, and if the degree of attention is 50%, 50% of the area of the display unit 22 is occupied.

  Further, the changing unit 114 may change the screen layout so as not to display the video data of the screen area whose attention level is smaller than a predetermined value. For example, a screen area with a degree of attention of 0% may be used as a screen area for displaying other video data. In this case, the screen layout may be changed so that a screen area larger than usual is displayed preferentially from a screen area with a high degree of attention.

  According to the information processing apparatus 1 of the present embodiment, it is possible to provide a video conference system in which the user of the video conference apparatus 2 does not need to manually change the screen layout according to a manual procedure. Thereby, smoother communication between the users of the video conference apparatus 2 can be realized.

  FIG. 12 is a diagram illustrating an example of a hardware configuration of the information processing apparatus 1 according to the present embodiment. The information processing apparatus 1 according to the present embodiment includes a control device such as a CPU 31, a main storage device such as a ROM (Read Only Memory) 32 and a RAM (Random Access Memory) 33, and a communication I / F 34 for connection to a network. And an external storage device such as an HDD (Hard Disk Drive) 35 and an optical drive 36. The CPU 31, ROM 32, RAM 33, communication I / F 34, HDD 35, and optical drive 36 are connected via a bus 41.

  A program executed by the information processing apparatus 1 according to the present embodiment is a file in an installable format or an executable format, such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), or the like. It is recorded on a computer-readable recording medium and provided as a computer program product.

  The program executed by the information processing apparatus 1 according to the present embodiment may be configured to be stored by being stored on a computer connected to a network such as the Internet and downloaded via the network. Further, the program executed by the information processing apparatus 1 of the present embodiment may be configured to be provided or distributed via a network such as the Internet.

  Further, the program of the information processing apparatus 1 of the present embodiment may be configured to be provided by being incorporated in advance in a ROM or the like.

  The program executed by the information processing apparatus 1 according to the present embodiment includes a module including the above-described units (the control unit 11 (the reception unit 111, the analysis unit 112, the calculation unit 113, and the change unit 114), and the distribution unit 12). As actual hardware, a CPU (processor) reads out and executes a program from the storage medium, and the respective units are loaded onto the main storage device. The control unit 11 (reception unit 111, analysis unit) 112, a calculation unit 113, a change unit 114), and a distribution unit 12 are generated on the main storage device.

DESCRIPTION OF SYMBOLS 1 Information processing apparatus 2, 2a-2i Video conference apparatus 3 Network 11 Control part 12 Distribution part 21 Imaging part 22 Display part 31 CPU
32 ROM
33 RAM
34 Communication I / F
35 HDD
36 optical drive 41 bus 111 receiving unit 112 analyzing unit 113 calculating unit 114 changing unit

JP 2010-211017 A

Claims (6)

An information processing apparatus connected to video conferencing apparatuses installed in a plurality of places via a network,
A receiving unit that receives an image captured by the imaging unit of the video conference device;
An analysis unit for analyzing a gaze direction of a person in the image;
A calculation unit that calculates a degree of attention indicating a degree of attention per unit time for each of a plurality of screen regions included in the screen layout of the video conference device in which the person is captured based on the line-of-sight direction;
A changing unit for changing the screen layout based on the degree of attention;
An information processing apparatus comprising: information indicating the screen layout; and a distribution unit that distributes a plurality of pieces of video data displayed in the plurality of screen areas included in the screen layout to the video conference device in which the person is captured. . The information according to claim 1, wherein the changing unit changes the screen layout to be determined according to the determination condition that matches the calculated attention level among a plurality of determination conditions that are conditions of the attention level. Processing equipment. The changing unit does not display the video data of the screen area whose attention level is smaller than a predetermined value, but displays the other video data in the screen area whose attention level is smaller than a predetermined value. The information processing apparatus according to claim 1, wherein the information processing apparatus is used as the screen area. The distribution unit distributes after changing the distribution data amount of the video data displayed in the screen area having the attention level to a distribution data amount corresponding to the attention level. The information processing apparatus according to item 1. A video conference system in which video conference devices installed in a plurality of places and information processing devices are connected via a network,
The information processing apparatus includes:
A receiving unit that receives an image captured by the imaging unit of the video conference device;
An analysis unit for analyzing a gaze direction of a person in the image;
A calculation unit that calculates a degree of attention indicating a degree of attention per unit time for each of a plurality of screen regions included in the screen layout of the video conference device in which the person is captured based on the line-of-sight direction;
A changing unit that changes a screen layout of the video conference device on which the person is imaged based on the attention degree;
A video conference system comprising: information indicating the screen layout; and a distribution unit that distributes a plurality of pieces of video data displayed in the plurality of screen areas included in the screen layout to the video conference device in which the person is captured. . A receiving unit that receives an image captured by the imaging unit of the video conference device;
An analysis unit for analyzing a gaze direction of a person in the image;
A calculation unit that calculates a degree of attention indicating a degree of attention per unit time for each of a plurality of screen regions included in the screen layout of the video conference device in which the person is captured based on the line-of-sight direction;
A changing unit that changes a screen layout of the video conference device on which the person is imaged based on the attention degree;
A computer realizes information indicating the screen layout and a distribution unit that distributes a plurality of video data displayed in the plurality of screen areas included in the screen layout to the video conference device in which the person is captured. Program for.

Images