JP2010239327A - Conference device, conference support method, and conference support program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a conference device having a simple structure and capable of performing reliable majority voting without being influenced by opinions in other strongholds, a conference support method, and a conference support program. <P>SOLUTION: In a teleconference system 1, a terminal 3 functioning as a chairman terminal transmits operation information of majority voting to terminals 4-6 when performing majority voting for a certain agendum. In the operation information, types of operations in clarifying intentions with majority voting are predetermined on an opinion basis. The types of the operations to the opinions are varied on a terminal basis. Thereby, even when the appearance of a user in another stronghold is displayed on a display 28, the user acts in a way different from that in user's own stronghold, and thereby does not influence decision in the own stronghold. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a conference apparatus, a conference support method, and a conference support program that can make a majority decision via an image.

  2. Description of the Related Art Conventionally, there has been known a video conference system in which a plurality of terminal devices are connected via a network and images and sound are bidirectionally transmitted and received to realize a conference between persons in remote locations. In video conferences, the chair may pick up the opinions of the participants and make a resolution on the agenda. When discussions are not possible, decisions are often made by majority vote. However, in the case of a video conference, the opinions of other bases may be known by video or audio, which may affect the judgment of the local base.

  Therefore, for example, an operation button device that asserts affirmation or denial is provided on the video conference terminal side, and a screen composition device that configures a vote screen from the output of this operation button device is provided on the system control side, and by pressing the operation button device A multipoint video conference system that displays positive opinions and negative opinions on a vote screen by color is known (see, for example, Patent Document 1).

JP-A-5-145915

  However, in the multipoint video conference system described in Patent Document 1, when there are a plurality of conference participants at each base, it is necessary to prepare operation buttons corresponding to the number of conference participants, and the configuration is complicated. There was a problem. In addition, in order to view and record who has what opinions, the operation buttons must be associated with the user, and must be updated each time the user changes.

  The present invention has been made in order to solve the above-described problem, and provides a conference device, a conference support method, and a conference support program that have a simple configuration and can make a reliable majority decision that is not affected by the opinions of other bases. The purpose is to provide.

  In order to achieve the above object, the conference device according to the first aspect of the present invention is a conference device that can be connected to a plurality of terminals via a communication line, and is a user output from the photographing means of the plurality of terminals. In a conference held between a display unit that combines the captured images of each terminal and displays a multi-image combining the captured images of each terminal on a display screen, and a plurality of users of the terminals, the conference device includes the conference When a majority vote is instructed to the terminal when the majority vote is instructed to the terminal by the majority vote instruction means, the intention display is made for each type of opinion of the user when making a majority vote. An allocation unit that allocates an operation for performing each operation to each terminal, and a result that is information on a total result of a user's opinion expressed intentionally by the operation allocated by the allocation unit for each terminal Result information acquisition means for acquiring information, result information storage means for storing the result information acquired by the result information acquisition means, and result information output means for outputting the result information stored in the result information storage means And.

  According to a second aspect of the present invention, in the conference apparatus according to the second aspect of the invention, in addition to the configuration of the first aspect of the invention, the allocating unit has the operation information in which the type information that is the type of operation is different for each terminal. Operation information storage means for storing the operation information, and operation information transmission means for transmitting the operation information stored in the operation information storage means to each of the plurality of terminals.

  According to a third aspect of the present invention, there is provided a conference device that receives the operation information transmitted from a chairman terminal that is a terminal functioning as the chairman of the plurality of terminals, in addition to the configuration of the second invention. Operation information receiving means, and operation display control means for displaying the operation information received by the operation information receiving means on the display screen.

  According to a fourth aspect of the present invention, there is provided a conference device according to the second or third aspect of the invention, in addition to the user recognition means for recognizing a user from a photographed image photographed by the photographing means; An action image storage means for storing an action image when the user recognized by the person recognition means operates according to the action information displayed on the display screen; and the action image stored in the action image storage means. Based on the action information specified by the user recognition means, the action type specifying means for specifying the type of action performed by the user according to the action information for each user, and the action type specified by the action type specifying means The opinion type specified by the opinion type specifying means for specifying the type of opinion indicated by the action information received by the action information receiving means, and the opinion type specifying means Opinion totaling means for totaling the number of users for each type of written opinion, and result information transmitting means for transmitting information on the results totaled by the opinion totaling means to the terminal as the result information .

  According to a fifth aspect of the present invention, there is provided a conference apparatus according to any one of the second to fourth aspects, an image size analyzing means for analyzing a user image size from the photographed image, and the image. First image size determining means for determining whether or not the image size analyzed by the size analyzing means is greater than or equal to a first threshold, and the action for displaying the intention is a gesture that is a user's physical action And a line-of-sight motion that directs the user to a predetermined area on the display screen, wherein the motion information assigns the gesture for each type of opinion of the user, and the gesture Assigning a region for directing the line of sight on the display screen for each type of user's opinion and the first operation information with different types for each terminal, and the position of the region Second action information that varies from end to end, and the action type specifying means includes a gesture specifying means for specifying the user's gesture, and an area to which the user's line of sight is directed on the display screen. Eye movement area specifying means for specifying, and when the image display size is determined to be less than a first threshold, the action display control means displays the first action information on the display screen, and Is determined to be greater than or equal to the first threshold, the second operation information is displayed on the display screen.

  According to a sixth aspect of the present invention, there is provided a conference apparatus according to the fifth aspect of the invention, in addition to the configuration of the fifth aspect of the invention, a gaze direction detecting means for detecting a gaze direction of the user from the captured image, a user, and the display screen. A positional relationship information storage unit that stores positional relationship information that is information on the positional relationship of the image, and a line-of-sight direction detected by the line-of-sight direction detection unit based on the positional relationship information stored in the positional relationship information storage unit A line-of-sight position specifying unit for specifying a line-of-sight position on the display screen, wherein the line-of-sight region specifying unit is configured to display the type of opinion on the display screen based on the line-of-sight position specified by the line-of-sight position specifying unit A region to which the line of sight is directed is specified from among regions assigned for each.

  According to a seventh aspect of the present invention, there is provided the conference device according to the fifth aspect, wherein the image size analyzed by the image size analyzing means is smaller than the first threshold value. A second image size determining unit configured to determine whether or not the image size is less than a threshold; and the operation display control unit is configured to determine whether the image size is equal to or larger than the first threshold by the first image size determining unit. Only the second motion information is displayed on the display screen, and when the second image size determination means determines that the image size is less than the second threshold, only the first motion information is displayed on the display screen. And the first image size determination means determines that the image size is less than the first threshold value, and the second image size determination means determines the image size to be less than or equal to the second threshold value. If he is determined that, and wherein the displaying the first operation information and second operation information on the display screen.

  According to an eighth aspect of the present invention, in addition to the configuration of the first aspect of the present invention, the result information includes the action image.

  In addition to the configuration of the invention according to any one of claims 1 to 8, the conference information device according to claim 9 includes, in the result information storage means, agenda information as information on the agenda, and the agenda The result information related to information is stored in association with information.

  According to a tenth aspect of the present invention, there is provided a conference support method for a conference apparatus that can be connected to a plurality of terminals via a communication line, and the user's image output from the imaging means of the plurality of terminals. In a display step of combining images and displaying a multi-image combining the captured images of each terminal on a display screen, and a conference held between users of the plurality of terminals, the conference device is the chairman of the conference In order to display a will for each type of opinion of the user when making a majority decision when a majority decision is instructed to the terminal in the majority instruction step. Assignment step for assigning the operation of each terminal to each terminal, and the result of counting the opinions of the users who indicated their intention by the operation assigned by the assigning means from the terminal A result information acquisition step for acquiring result information as information, a result information storage step for storing the result information acquired in the result information acquisition step in a result information storage means, and the result information storage step in the result information storage step A result information output step for outputting the result information stored in the means.

  According to an eleventh aspect of the present invention, a conference support program causes a computer to function as various processing means of the conference apparatus according to any one of the first to ninth aspects.

  In the conference device according to the first aspect of the present invention, the display means combines the photographed images of the users output from the photographing means of the plurality of terminals, and displays a multi-image combining the captured images of the terminals on the display screen. To do. The majority instruction means instructs a terminal to vote when the conference device functions as the chairperson of the conference in a conference held between a plurality of users of the terminals. The allocating unit allocates, for each terminal, an operation for displaying a will for each type of opinion of the user when performing the majority voting when the majority voting is instructed by the majority deciding unit. Since the operation for displaying the intention can be assigned, for example, by changing the type of operation for each terminal, users at other sites appearing on the multi-screen displayed on the display screen can operate differently from their own sites. Because it shows an opinion, the opinion of your site is not affected. The result information acquisition means acquires result information which is information on the total result of the opinions of the user who intentionally displayed by the operation assigned by the assignment means from the terminal. The result information storage means stores the result information acquired by the result information acquisition means. The result information output means outputs the result information stored in the result information storage means. Thereby, the totalization result of the majority decision of each base can be browsed.

  In the conference apparatus according to the second aspect of the present invention, in addition to the effect of the first aspect of the invention, the assigning means includes an operation information storage means and an operation information transmission means. The operation information storage means stores the type information, which is information about the type of operation, as operation information that is different for each terminal. The operation information transmitting unit transmits the operation information stored in the operation information storage unit to each of the plurality of terminals. As a result, a majority decision is made at each site based on the operation information transmitted to each terminal, and therefore an operation for displaying a will for each type of opinion of the user when making a majority decision can be assigned to each terminal. it can.

  Further, in the conference device according to the third aspect of the present invention, in addition to the effect of the second aspect of the invention, the operation information receiving means is an operation transmitted from a chairman terminal which is a terminal functioning as the chairman among a plurality of terminals. Receive information. The operation display control unit displays the operation information received by the operation information receiving unit on the display screen. Thereby, the user of each base can perform an operation corresponding to his / her opinion based on the operation information displayed on the display screen.

  In the conference apparatus according to the fourth aspect of the invention, in addition to the effect of the invention according to the second or third aspect, the user recognition means recognizes the user from the photographed image photographed by the photographing means. The action image storage means stores an action image when the user recognized by the user recognition means operates according to the action information displayed on the display screen. The action type specifying means specifies, for each user, the kind of action performed by the user recognized by the user recognition means according to the action information, based on the action image stored in the action image storage means. The opinion type specifying means specifies the type of opinion indicated by the type of action specified by the action type specifying means from the action information received by the action information receiving means. The opinion counting means counts the number of users for each type of opinion specified by the opinion type specifying means. The result information transmitting unit transmits the information of the results totaled by the opinion totaling unit to the terminal as result information. As a result, it is possible to transmit the result information indicating the opinion by the operation based on the operation information to the chairman terminal that transmitted the operation information.

  In the conference apparatus according to the fifth aspect of the invention, in addition to the effect of the invention according to any one of the second to fourth aspects, the image size analyzing means analyzes the image size of the user from the photographed image. The first image size determining means determines whether or not the image size analyzed by the image size analyzing means is greater than or equal to a first threshold value. The action for intention display is a gesture that is a user's body action and a line-of-sight action that causes the user to turn the line of sight into a predetermined area on the display screen. The motion information includes first motion information and second motion information. The first operation information is obtained by assigning a gesture for each type of opinion of the user and making the type of gesture different for each terminal. The second operation information is obtained by assigning an area in which a line of sight is directed on the display screen for each type of opinion of the user, and changing the position of the area for each terminal. The action type specifying unit includes a gesture specifying unit and a line-of-sight area specifying unit. The gesture specifying means specifies a user's gesture. The line-of-sight area specifying means specifies an area where the line of sight on the display screen of the user is directed. When it is determined that the image size is less than the first threshold, the motion display control means is based on a gesture to facilitate the determination of the user's motion because the user's image size displayed on the display screen is small. The first operation information is displayed on the display screen. On the other hand, when it is determined that the image size is equal to or larger than the first threshold, the size of the user displayed on the display screen is large, and therefore the second motion information based on the line of sight is displayed on the display screen. As described above, since the first motion information or the second motion information can be properly used based on the image size of the user displayed on the display screen, the user's motion can be accurately and easily detected from the captured image.

  In the conference apparatus according to the sixth aspect of the invention, in addition to the effect of the fifth aspect of the invention, the gaze direction detecting means detects the gaze direction of the user from the photographed image. The positional relationship information storage means stores positional relationship information that is information on the positional relationship between the user and the display screen. The line-of-sight position specifying unit specifies the line-of-sight position on the display screen ahead of the line-of-sight direction detected by the line-of-sight direction detection unit based on the positional relationship information stored in the positional relationship information storage unit. The line-of-sight area specifying means specifies an area to which the line of sight is directed among the areas allocated for each opinion type on the display screen based on the line-of-sight position specified by the line-of-sight position specifying means. Thereby, the user can show his / her opinion at the line-of-sight position.

  In the conference device according to the seventh aspect of the invention, in addition to the effect of the invention according to the fifth or sixth aspect, the second image size determining means is configured such that the image size analyzed by the image size analyzing means is the first threshold value. It is determined whether it is less than a second threshold value smaller than the second threshold value. The action display control means displays only the second action information on the display screen when the first image size determining means determines that the image size is equal to or larger than the first threshold value. When the second image size determining means determines that the image size is less than the second threshold, only the first operation information is displayed on the display screen. When the first image size determining means determines that the image size is less than the first threshold and the second image size determining means determines that the image size is equal to or greater than the second threshold, the first action information and the second Display both operation information on the display screen. That is, when the image size of the user displayed on the display screen is equal to or larger than the second threshold value and smaller than the first threshold value, both the first motion information and the second motion information are displayed on the display screen. It becomes more difficult to guess the opinion from the person's actions. This makes it possible to make a reliable majority vote that is not affected by the opinions of other bases.

  Further, in the conference device according to the eighth aspect of the invention, in addition to the effect of the invention according to any one of the first to seventh aspects, the opinion for each user can be grasped from the operation image included in the result information.

  In the conference device of the invention according to claim 9, in addition to the effects of the invention according to any one of claims 1 to 8, the result information storage means is associated with the agenda information which is agenda information and the agenda information. And associated result information. Thereby, it can memorize | store as a minutes which has an agenda and its result.

  In the conference support method of the invention according to claim 10, in the display step, the user's captured images output from the imaging means of the plurality of terminals are combined and a multi-image combining the captured images of each terminal is displayed on the display screen. To display. In the majority decision instruction step, when a conference device functions as the chairperson of a conference in a conference held between users of a plurality of terminals, the majority command is instructed to the terminal. In the operation assigning step, when a majority vote is instructed to the terminal in the majority vote instructing step, an action for displaying a will for each type of opinion of the user when making the majority vote is assigned to each terminal. Since the operation for displaying the intention can be assigned, for example, by changing the type of operation for each terminal, users at other sites appearing on the multi-screen displayed on the display screen can operate differently from their own sites. Because it shows an opinion, the opinion of your site is not affected. In the result information acquisition step, the result information, which is information on the total result of the opinions of the user who intentionally displayed the operation assigned in the operation assignment step, is acquired from the terminal. In the result information storage step, the result information acquired in the result information acquisition step is stored in the result information storage means. In the result information output step, the result information stored in the result information storage means in the result information storage step is output. Thereby, the totalization result of the majority decision of each base can be browsed.

  Further, in the conference support program of the invention according to claim 11, since the computer functions as various processing means of the conference apparatus according to claim 1, the program according to claim 1. An effect can be obtained.

1 is a block diagram (transmission of gesture information and line-of-sight area information) showing a configuration of a video conference system 1. FIG. It is a block diagram (reception number data, result image data reception) which shows the structure of the video conference system. It is a figure which shows the multi screen displayed on the display 28 of the terminal device 4. FIG. 3 is a block diagram showing an electrical configuration of a terminal device 3. FIG. 3 is a conceptual diagram showing various storage areas of an HDD 31. FIG. It is a conceptual diagram of the gesture table 3121. 3 is a conceptual diagram of a line-of-sight area table 3131. FIG. It is the figure (terminal device 4) which showed arrangement of line-of-sight fields 95 and 96. It is the figure (terminal device 5) which showed arrangement of line-of-sight fields 95 and 96. It is the figure (terminal device 6) which showed arrangement of line-of-sight fields 95 and 96. It is a conceptual diagram of the total table 3151. 3 is a conceptual diagram showing various storage areas of a RAM 22. FIG. It is a conceptual diagram of gesture information 2231. It is a conceptual diagram of the line-of-sight area information 2241. It is the figure which showed the mutual positional relationship of the eyeball 72 and the display 28. FIG. It is a flowchart of chairperson terminal processing. It is a flowchart of a base terminal process. 5 is an explanatory diagram of a method for analyzing a person image region F. FIG. It is a figure in which the user is gesturing according to the gesture instruction image displayed on the display. FIG. 3 is a diagram in which a user directs his / her line of sight to the line-of-sight areas 95 and 96 displayed on the display 28. FIG. 3 is a diagram in which a user directs his / her line of sight to the line-of-sight areas 95 and 96 while gesturing according to a gesture instruction image displayed on the display 28. It is a figure which shows the result image. It is a figure which shows the result image 121. FIG. It is a figure which shows all the terminal total images 130. FIG. It is a figure which shows the all terminal result image 131. FIG. It is a figure which shows the all terminal result image 132. FIG.

  Hereinafter, a terminal device 3 according to an embodiment of the present invention will be described with reference to the drawings. First, the configuration of the video conference system 1 including the terminal device 3 will be described with reference to FIGS. The video conference system 1 includes a plurality of terminal devices 3 to 6 that are connected to each other via a network 2 and provided at each site. In the video conference system 1, a remote conference is performed by transmitting and receiving images and sounds between the terminal devices 3 to 6 via the network 2.

  On the displays 28 of the terminal devices 3 to 6, a multi-screen that displays the captured images of the respective bases is displayed. For example, on the display 28 of the terminal device 4 illustrated in FIG. 3, the divided screen 281 on which the captured image of the terminal device 3 is displayed, the divided screen 282 on which the captured image of the terminal device 4 is displayed, and the imaging of the terminal device 5. A split screen 283 on which an image is displayed is displayed. The user can have a conversation while looking at the face of the user at the other base displayed on the multi-screen. The number of divided screens is set according to the number of terminal devices participating in the conference. The display 28 shown in FIG. 3 corresponds to the “display unit” of the present invention.

  In the present embodiment, in such a video conference system 1, when the terminal device 3 functioning as the chairperson terminal makes a resolution by majority vote for a certain agenda item, the operation information of the majority vote at each base of the terminal devices 4, 5, 6 (Gesture information, line-of-sight area information) is transmitted (see FIG. 1). Aggregated information (majority data, result image data, etc.) of majority decisions made according to the operation information can be collected from each base and recorded (see FIG. 2). The recorded result can be displayed on the display 28 as required. The terminal devices 3 to 6 shown in FIG. 1 correspond to the “conference device” of the present invention.

  Next, the electrical configuration of the terminal device 3 will be described with reference to FIG. Since the terminal devices 3 to 6 all have the same configuration, only the configuration of the terminal device 3 will be described, and the description of the terminal devices 4 to 6 will be omitted.

  The terminal device 3 is provided with a CPU 20 as a controller that controls the terminal device 3. Connected to the CPU 20 are a ROM 21 that stores BIOS, a RAM 22 that temporarily stores various data, and an I / O interface 30 that mediates data transfer. The I / O interface 30 is connected to a hard disk drive 31 (hereinafter referred to as HDD 31) having various storage areas.

  The I / O interface 30 includes a communication device 25 for communicating with the network 2, a mouse 27, a video controller 23, a key controller 24, a camera 34 for photographing a user, and a user's voice. Are connected to each other, a microphone 35 for capturing infrared rays, an infrared light 37 for irradiating the user's eyeball with infrared rays, a CCD camera 38 for imaging the user's eyes, and a CD-ROM drive 26. A display 28 is connected to the video controller 23. A keyboard 29 is connected to the key controller 24. The infrared light 37 is provided at the same position as the CCD camera 38. The CCD camera 38 is fixed to the upper end portion of the display 28. The CCD camera 38 and the infrared light 37 are prepared according to the number of users at each base.

  The CD-ROM 114 inserted into the CD-ROM drive 26 stores the main program of the terminal device 3 and the conference support program of the present invention. When the CD-ROM 114 is introduced, these various programs are set up from the CD-ROM 114 to the HDD 31 and stored in a program storage area 319 (see FIG. 5) described later.

  Next, various storage areas of the HDD 31 will be described with reference to FIG. The HDD 31 includes a participating terminal storage area 311, a gesture table storage area 312, a line-of-sight area table storage area 313, a total table storage area 314, an all terminal result image storage area 315, and an all terminal total image storage area 316. A gesture instruction image storage area 317, a person image operation pattern storage area 318, a program storage area 319, another information storage area 320, and the like are provided.

  The participation terminal storage area 311 stores an identification ID of a terminal device currently participating in the conference. When the terminal device 3 participates in the conference, a login signal for notifying participation is transmitted to each of the other terminal devices. In the terminal device 3 that has received the login signal, the identification ID of the terminal device that has transmitted the login signal is stored in the participating terminal storage area 311 of the HDD 31. As a result, the terminal devices currently participating in the conference can be managed. For example, a Mac address or the like can be used as the identification ID. Also, an identification ID corresponding to the Mac address may be stored in advance.

  The gesture table storage area 312 stores a gesture table 3121 (see FIG. 6). The line-of-sight area table storage area 313 stores a line-of-sight area table 3131 (see FIG. 7). In the total table storage area 314, a total table 3151 (see FIG. 11) is stored. All terminal result image storage area 315 stores all terminal result image 131 (see FIG. 25) obtained by combining the agenda with the result image of each base. In the total terminal total image storage area 316, a total terminal total image 130 (see FIG. 24) in which the agenda is added to a table summarizing the total number of each base registered in the total table 3151 is stored. The gesture instruction image storage area 317 stores a gesture instruction image (not shown) to be displayed on the display 28 based on the gesture information transmitted from the chairperson terminal.

  In the person image operation pattern storage area 318, an area F (see FIG. 18) of a person image when a gesture is performed in an image captured by the camera image of the camera 34 is stored for each gesture pattern. For example, a person image area F when the right hand is raised, a person image area F when the left hand is raised, a person image area F when sitting, a person image area F when standing, and the like are stored in advance. The person image area F for each gesture pattern performs pattern matching of the shape by comparison with the person image area F acquired in the person image analysis. Thereby, it is possible to determine what kind of gesture the person has performed. In addition, the method of detecting the motion of the human motion is not limited to this method, and various methods are possible.

  In the program storage area 319, the main program of the terminal device 3, the communication control program for executing a remote conference with the other terminal devices 4 to 6, and the conference support of the present invention executed when making a majority decision Programs and so on are stored. In the other information storage area 320, other information used in the terminal device 3 is stored. When the terminal device 3 is a dedicated machine that does not include the HDD 31, various programs are stored in the ROM 21.

  Next, the gesture table 3121 will be described with reference to FIG. The gesture table 3121 stores, for each base, gesture information indicating a gesture for displaying a will for a user when making a majority decision. In the gesture table 3121, a terminal ID column 61 and a gesture column 62 are provided in association with each other. In the terminal ID column 61, terminal IDs of the terminal devices 4 to 6 other than the own base are stored. The gesture column 62 is divided into a YES column 63 and a NO column 64.

  The YES column 63 stores the type of gesture when the user makes a will display of “YES”. The NO column 64 stores the type of gesture when the user makes a will display of “NO”. The types of gestures are stored differently for each terminal device.

  For example, in the terminal device 4 (identification ID = 002) in the first row, the gesture corresponding to “YES” is “right hand raised” and the gesture corresponding to “NO” is “left hand raised”. On the other hand, in the terminal device 5 (identification ID = 003) in the second row, the gesture corresponding to “YES” is “left hand raised” and the gesture corresponding to “NO” is “right hand raised”. Further, in the terminal device 4 (identification ID = 004) in the third row, the gesture corresponding to “YES” is “standing” and the gesture corresponding to “NO” is “both hands raised”.

  Next, the line-of-sight area table 3131 will be described with reference to FIG. The line-of-sight area table 3131 stores data of the coordinate range of the line-of-sight area for directing the user's line-of-sight position to a specific area of the display 28 when making a majority decision. In the line-of-sight area table 3131, a terminal ID column 71 and a line-of-sight area range column 73 are provided in association with each other. In the terminal ID column 71, terminal IDs of the terminal devices 4 to 6 other than the own base are stored. The line-of-sight area range column 73 is divided into a YES column 77 and a NO column 78.

  The YES column 77 stores a line-of-sight area range in which a line of sight is directed when the user makes a “YES” intention display. The NO column 78 stores a line-of-sight area range in which a line of sight is directed when the user makes a “NO” intention display. The line-of-sight area range is stored differently for each terminal device.

  For example, in the terminal device 4 (identification ID = 002) in the first row, the line-of-sight area range corresponding to “YES” is (0, 0) to (30, 30), and the line-of-sight area range corresponding to “NO” Are (60, 30) to (80, 60). On the other hand, in the terminal device 5 (identification ID = 003) in the second row, the line-of-sight area range corresponding to “YES” is (30, 0) to (80, 30), and the line-of-sight area range corresponding to “NO”. Are (0, 30) to (25, 60). Further, in the terminal device 6 (identification ID = 003) in the third row, the line-of-sight area range corresponding to “YES” is (0, 0) to (30, 60), and the line-of-sight area range corresponding to “NO”. Are (50, 0) to (80, 60).

  Here, the “YES” and “NO” line-of-sight areas of the respective terminals stored in the line-of-sight area table 3131 are on the following positions on the display 28. As shown in FIG. 8, the line-of-sight area designated by the terminal device 4 includes a line-of-sight area 95 of “YES” at the upper left and a line-of-sight area 96 of “NO” at the lower right. As shown in FIG. 9, a line-of-sight area 95 of “YES” is arranged in the upper right, and a line-of-sight area 96 of “NO” is arranged in the lower left, respectively. As shown in FIG. 10, the line-of-sight area designated by the terminal device 6 is arranged with a line-of-sight area 95 of “YES” on the left side and a line-of-sight area 96 of “NO” on the right side. Thus, the line-of-sight areas of “YES” and “NO” are different for each terminal device.

  As will be described later, the line-of-sight areas 95 and 96 of “YES” and “NO” are displayed according to the line-of-sight area stored in the line-of-sight area table 3131. In these line-of-sight regions 95 and 96, there are provided contour lines 99 (see FIGS. 8 to 10) that further narrow the respective ranges. By dividing the boundary between the line-of-sight areas 95 and 96 with the outline 99, the line-of-sight can be surely directed into the line-of-sight areas 95 and 96.

  Next, the summary table 3151 will be described with reference to FIG. The tabulation table 3151 stores the tabulation data transmitted from each site collectively for each site. The aggregation table 3151 is provided with a terminal ID column 81, a YES column 82, and a NO column 83 in association with each other. A terminal ID is stored in the terminal ID column 81. The YES column 82 stores the number of people who have indicated “YES”. In the NO column 83, the number of persons who show the will of “NO” is stored. Note that the total of “YES” and “NO” for all bases is stored in the last row of the aggregation table 3151, respectively.

  Next, various storage areas of the RAM 22 will be described with reference to FIG. The RAM 22 includes a CCD camera image storage area 221, a line-of-sight detection information storage area 222, a gesture information storage area 223, a line-of-sight area information storage area 224, a voted image storage area 225, an opinion count storage area 226, and results. At least an image storage area 227 and a camera image storage area 228 are provided.

  The CCD camera image storage area 221 stores a captured image of the CCD camera. The line-of-sight detection information storage area 222 stores line-of-sight detection information, which is information such as various parameters and setting values, which will be described later, calculated when performing line-of-sight detection.

  In the gesture information storage area 223, gesture information 2231 (see FIG. 13) transmitted from the chairperson terminal is stored. The gesture information 2231 includes the type of gesture for displaying the will of “YES” and the type of gesture for displaying the will of “NO”. The line-of-sight area information storage area 224 stores line-of-sight area information 2241 (see FIG. 14) transmitted from the chairperson terminal. The line-of-sight area information 2241 includes a line-of-sight area when displaying a will of “YES” and a line-of-sight area when displaying a will of “NO”.

  In the voting image storage area 225, data of voting images when voting is performed by majority vote is stored. That is, when voting is performed by gestures, it is image data when each user is gesturing. When voting is performed at the line-of-sight position, it is image data when each user points the line-of-sight position toward a predetermined line-of-sight area. In the opinion number storage area 226, data on the number of opinions of “YES” and data on the number of opinions of “NO” for the users of the respective bases are stored.

  In the result image storage area 227, a result image 120 (see FIG. 22) in which the characters “YES” and “NO” are displayed on the vote image stored in the vote image storage area 225 is stored.

  The result image 120 is obtained by adding “YES” or “NO” to the side of each person's head shown in the vote image. For example, a result image 121 (see FIG. 23) obtained by cutting out individual images for each opinion of “YES” and “NO” and arranging them for each opinion may be stored in the result image storage area 227. . In the camera image storage area 228, a captured image of the camera 34 is stored.

  Next, a method for detecting the user's line-of-sight position will be described. In the method of detecting the line-of-sight position, the face portion centered on the user's eyes is captured by the CCD camera 38 (see FIG. 4), and the resulting image is analyzed to detect the line-of-sight direction. The line-of-sight position on the display 28 is detected from the detected line-of-sight direction. Various methods can be applied as a method of detecting the line-of-sight direction.

  For example, Japanese Patent Laid-Open No. 2000-138872 detects the contours of both eyes from image data obtained by photographing a user, and determines the center position and radius of the eyeball from the geometrical positional relationship between the ends of the eyes and the nostrils. A method for estimating and detecting the gaze direction is shown.

  For example, Yoshio Matsumoto, Junichi Tsujido, Kentaro Takemura, Tsukasa Ogasawara, Development of Real-time Face / Gaze Measurement System and Application to Intelligent Interface, Information Processing Society of Japan Journal of Computer Vision and Image Media, vol. 47, no. SIG15 (CVIM16), pp. In the non-patent literatures of 10-21 and 2006/10, as a line-of-sight measurement method using image analysis, a face region is cut out from a CCD camera image and model fitting with a face discovery template is performed. A method of obtaining a position / posture, estimating an eyeball center position, and detecting a gaze direction in real time is shown.

  For example, Takehiko Ohno, Gaze-based interface, information processing, Vol. 44, no. 7, pp. In non-patent documents of Nos. 726-732 and 2003, as a line-of-sight measurement method using near infrared rays, a true pupil is obtained from a three-dimensional model relating to the shape of a human eyeball, an observed pupil image and a Purkinje image using an infrared camera. A method of estimating the position and the position of the Purkinje image and calculating the line-of-sight direction is shown.

  In the present embodiment, the line of sight is detected using a line-of-sight measurement method using near infrared rays. Therefore, a gaze detection method using near infrared rays will be described with reference to FIG.

  When the infrared light 37 irradiates the user's eyeball 72 with infrared light, the irradiated infrared light is reflected on the cornea reflection surface of the user's eyeball 72, and a virtual image (Purkinje image) 74 of the infrared light 37 is generated. When the infrared light 37 is positioned ahead of the user's line of sight, the generation position of the Purkinje image 74 coincides with the center position of the pupil 75. Then, as the tip of the user's line of sight moves away from the infrared light 37, the interval between the Purkinje image 74 and the center of the pupil 75 increases.

  This will be specifically described. A rotation angle of the eyeball 72 from a state where the Purkinje image 74 and the center of the pupil 75 coincide is θ. The angle between the display 28 and the infrared light 37 (CCD camera 38) is α. Assuming that the interval between the Purkinje image 74 and the center of the pupil 75 is S, the interval S increases almost in proportion to the sine of the rotation angle θ. By calculating the position of the Purkinje image 74, the center position of the pupil 75, and the interval S in the photographed image photographed by the CCD camera 38, the rotation angle θ of the eyeball 72 and further the direction of the user's line of sight can be detected. . The detection of the Purkinje image 74 and the pupil 75 in the captured image is performed by a known method for detecting an edge value indicating a density change in each pixel.

  Next, a method for detecting the user's line-of-sight position T on the display 28 ahead of the detected line-of-sight direction will be described. The user stands at a position marked on the floor surface in front of the terminal device 3 and adjusts the height using a platform or the like, whereby the center of the user's eyeball 72 and the display 28 are separated by a predetermined distance. L.

  Here, the marking position is adjusted so that the distance between the center of the user's eyeball 72 standing at the position and the display 28 is a predetermined distance L. Since the height of the user differs depending on the user, the position of the eyeball 72 is adjusted using a platform or the like so as to be a predetermined height. In order to fix the position of the eyeball 72, the position of the user's head may be fixed. Since it is assumed that there are a plurality of users at each site, a plurality of markings are set in front of the display 28. Since the marking positions are different, the predetermined distance L is also different for each position.

An intersection point that intersects the display 28 at a right angle from the center of the eyeball 72 is defined as a center point O. The distance K between the line-of-sight position T1 on the display 28 ahead of the user's line-of-sight direction and the center point O is expressed by the following equation.
・ K = L ・ tanβ
.Beta. =. Theta .- (90-.alpha.)
* K = L * tan ([theta]-(90- [alpha]))
From the above formula, the user's line-of-sight position T on the display 28 can be specified. It should be noted that parameters relating to eye gaze detection, such as the rotation angle θ, the interval S, the predetermined distance L, and the angle α, and values relating to the gaze position calculation are stored in the eye gaze detection information storage area 222 (see FIG. 12) of the RAM 22. . The predetermined distance L is different for each of a plurality of marking positions in front of the display 28. Accordingly, the predetermined distance L for each marking position is stored in the line-of-sight detection information storage area 222, respectively. The predetermined distance L and the angle α are stored in the line-of-sight detection information storage area 222 of the RAM 22 as input from the keyboard 29 of the terminal device 3.

  Next, the majority process executed in the video conference system 1 will be described. The majority process is performed based on a conference support program stored in the HDD 31 of the terminal devices 3 to 6. The conference support program is divided into “chairman terminal processing” executed by the chairman terminal and “base terminal processing” executed by a terminal other than the chairman terminal.

  Here, for convenience of explanation, the terminal device 3 is described as a chairman terminal, and the other terminal devices 4 to 6 are described as base terminals that make a majority decision in response to an instruction from the chairman terminal. In the conference support program, two or more terminal IDs including its own terminal are registered in the participating terminal storage area 311 (see FIG. 5) stored in the HDD 31, and a conference via audio and images can be held with other bases. Runs when started.

  First, the chairman terminal process by the CPU 20 will be described with reference to the flowchart of FIG. First, it is determined whether or not to make a majority decision (S11). Here, in the case of voting by majority vote, the determination is made based on whether or not the “execute majority vote” button (not shown) displayed on the display 28 is selected. While the majority decision is not made (S11: NO), the process returns to S11 and the process is repeated.

  When a majority decision is made (S11: YES), this terminal becomes the “chairman terminal”. Next, the gesture information 2231 (see FIG. 13) of each terminal is acquired from the gesture table 3121 (see FIG. 6) stored in the HDD 31, and the line-of-sight area information 2241 (see FIG. 7) of each terminal is obtained from the line-of-sight region table 3131 (see FIG. 7). 14) is obtained. Then, the acquired gesture information 2231 and line-of-sight area information 2241 are acquired according to the terminal participating in the conference, and transmitted to each of the terminal devices 4 to 6 (S12).

  As will be described later, at each base of the terminal devices 4 to 6 that has received the gesture information 2231 and the line-of-sight area information 2241, an image based on the received gesture information 2231 or line-of-sight area information 2241 is displayed on the display 28 (see FIGS. 19 to 21). Is displayed. In each base, a majority decision is made by each user performing a predetermined operation based on the image displayed on the display 28. The result of the majority decision is processed at each base, and the total information composed of the opinion number data and the data of the result image 120 (see FIG. 22) (hereinafter referred to as result image data) is directed to the terminal device 3 which is the chairperson terminal. Is transmitted (see FIG. 2).

  Next, as shown in FIG. 16, it is determined whether or not the total information has been received (S13). When the total information is received (S13: YES), the opinion number data is stored in the total table 3151 (see FIG. 11) stored in the HDD 31 (S14). Further, the result image data is associated with the agenda and stored in the all terminal result image storage area 315 (see FIG. 5) of the HDD 31 (S15). Then, returning to S13, when the total information is received from another base (S13: YES), the number-of-opinions data is added to the total table 3151 (see FIG. 11) in the same manner as described above (S14). Further, the result image data is added to the all terminal result image storage area 315 (see FIG. 5) (S15).

  As described above, by repeating S13 to S15, the data of the result image 120 of each site is collected in the all-terminal result image storage area 315, and the all-terminal result image 131 (in which the result images of each site are arranged together with the agenda). 25) is synthesized and stored. In the all terminal result image 131, for example, the agenda “Do you agree with this plan?” Is displayed at the top, and for example, the result image 120 of the base B (terminal device 4) and the base D (terminal device 6) are displayed below it. Result image 120 and the like are displayed side by side. In the result image 120 of each base, characters “YES” and “NO” are displayed beside the heads of the persons participating in the majority decision. Thereby, it is possible to recognize at a glance who has an opinion of “YES” or “NO”.

  When the data of the result image 121 (see FIG. 23) is received from each base, all terminal result image storage areas 315 have “YES” and “NO” opinions from the result image data of each base. Individual images are cut out, and all terminal result images 132 (see FIG. 26) arranged for each of “YES” and “NO” are combined and stored. Even in this all terminal result image 132, it is possible to recognize at a glance who has an opinion of “YES” or “NO”.

  On the other hand, as shown in FIG. 16, when the total information is not received (S13: NO), it is determined whether or not the total information is collected from the terminals of all the bases (S16). Here, it is determined whether or not the total information of all terminals registered in the participating terminal storage area 311 (see FIG. 5) of the HDD 31 is stored in the total table 3151 (see FIG. 11). If there is a terminal that has not yet received the total information (S16: NO), the process returns to S13, and the reception of the total information is continuously monitored.

  When it is determined that the total information of all terminals has been received with reference to the total table 3151 (S16: YES), “YES” and “NO” are calculated from the total table 3151 (see FIG. 11) stored in the HDD 31. The total terminal total image (see FIG. 24) in which the number of opinions is displayed together with the agenda is synthesized and stored in the total terminal total image storage area 316 (see FIG. 5) of the HDD 31 (S17).

  Next, the all terminal result image 131 (see FIG. 25) stored in the all terminal result image storage area 315 (see FIG. 5) and the all terminal total image stored in the all terminal total image storage area 316 (see FIG. 5). 130 (see FIG. 24) is displayed on the display 28 as a vote result (S18). Further, these voting results are transmitted to each terminal device (S19), and the process ends.

  In the terminal devices 4 to 6 that have received the data of the total terminal total image 130 and the data of the total terminal result image 131, these image data are stored in the HDD 31. By displaying the total terminal total image 130 or the total terminal result image 131 stored in the HDD 31 at each site on the display 28, the result of the majority vote can be known at each site.

  Next, the base terminal processing by the CPU 20 will be described with reference to FIG. For convenience of explanation, a base terminal process executed by the CPU 20 of the terminal device 4 functioning as a base terminal will be described. First, it is determined whether or not the gesture information 2231 (see FIG. 13) and the line-of-sight area information 2241 (see FIG. 14) transmitted from the chairman terminal are received (S21). Until the gesture information 2231 and the line-of-sight area information 2241 are obtained (S21: NO), the process returns to S21 and enters a standby state.

  When it is determined that the gesture information 2231 and the line-of-sight area information 2241 have been received (S21: YES), a camera image captured by the camera 34 is acquired (S22). The camera image is stored in a camera image storage area 228 of the RAM 22 (see FIG. 12). Further, based on the camera image stored in the camera image storage area 228, a human image size analysis process is executed (S23).

  The person image size analysis processing will be described with reference to FIG. As a method for detecting the person U from the camera image stored in the camera image storage area 228 of the RAM 22, the “background difference method” is applied. In the background subtraction method, first, a frame image 200 as a background P is determined in advance. Based on this, a difference is taken for each frame. An area where the difference value is equal to or greater than a specified value is detected as an image area F of the person F. The area of the detected person image area F is analyzed as the size of the person image.

  Next, as shown in FIG. 17, it is determined whether or not the size of the image of the person projected on the camera image is all equal to or greater than the first predetermined value (S24). When a plurality of persons are displayed in the camera image, it is determined whether or not all the image sizes are equal to or larger than the first predetermined value. When all the image sizes are equal to or larger than the first predetermined value (S24: YES), the size is sufficient to detect the line of sight for any user. In this case, based on the received line-of-sight area information 2241 (see FIG. 14), the display 28 displays a line-of-sight area 95 of “YES” and a line-of-sight area 96 of “NO”.

  For example, when the line-of-sight area information 2241 shown in FIG. 14 is received, as shown in FIG. 8, the line-of-sight area 95 of “YES” is arranged on the upper right side and the line-of-sight area 96 of “NO” is placed on the right side. Located in the lower left. In the line-of-sight area 95 of “YES”, a contour line 99 that curves between (30, 0) and (0, 30) is displayed, and “YES” is displayed inside. The area inside the contour line 99 is displayed in red. In the line-of-sight area 96 of “NO”, a curved outline 99 is also displayed between (60, 60) and (80, 30), and a character “NO” is displayed inside thereof. The inside of the contour line 99 is displayed in blue. Thus, by further narrowing the line-of-sight areas 95 and 96 by the contour line 99, the user's line-of-sight can be surely directed into the line-of-sight areas 95 and 96.

  Then, as shown in FIG. 20, line-of-sight areas 95 and 96 are displayed on the display 28. By displaying the line-of-sight areas 95 and 96 displayed on the display 28, the users A and B perform a will display of “YES” and “NO” by their own lines of sight. The position of the line-of-sight area displayed on the display 28 is different for each terminal device. Therefore, even if the state of the user at the other site is shown on the multi-screen displayed on the display 28, the judgment at the own site is not affected.

  On the other hand, returning to FIG. 17, when there is a person whose person image size is not equal to or larger than the first predetermined value (S24: NO), it is determined whether or not all the person image sizes are smaller than the second predetermined value (S26). The second predetermined value is a value smaller than the first predetermined value. When all the person image sizes are less than the second predetermined value (S26: YES), since all the person images are too small, it is difficult to detect the line-of-sight position on the display 28. In this case, a gesture instruction image corresponding to the received gesture information 2231 (see FIG. 13) is displayed on the display 28 (S27). Specifically, a gesture instruction image corresponding to the received gesture information 2231 is acquired from a plurality of gesture instruction images stored in the gesture instruction image storage area 317.

  For example, when the gesture information 2231 shown in FIG. 13 is received, the gesture instruction image shown in FIG. 19 is displayed on the display 28. That is, the divided screens 285 and 286 are displayed on the left and right sides of the display 28, respectively. On the left split screen 285, a person character 91 with a right hand is displayed, and a message “YES: Please raise your right hand” is displayed at the top. On the right split screen 286, a human character 92 with a left hand is displayed, and a message “NO: Please raise your left hand” is displayed above it. By displaying the operation instruction image displayed on the display 28, the users A and B perform a will display of “YES” and “NO” by their own gestures. The gesture instruction image displayed on the display 28 is different for each terminal device. Therefore, even if the state of the user at the other site is shown on the multi-screen displayed on the display 28, the judgment at the own site is not affected.

  When the person image size varies depending on the person (S24: NO, S26: NO), as shown in FIG. 21, the line-of-sight areas 95 and 96 and the gesture instruction image are displayed on the display 28 so as to overlap each other. (S28). By displaying the line-of-sight areas 95 and 96 displayed on the display 28 and the gesture instruction image, the users A and B are “YES” and “NO” depending on their line of sight and the gesture. Will display will. Each position of the line-of-sight areas 95 and 96 displayed on the display 28 and the gesture instruction image are different for each terminal device. Therefore, since the operation for displaying the will at each terminal becomes more complicated, even if the state of the user at the other site is shown on the multi-screen displayed on the display 28, the judgment at the own site is affected. There is no.

  Next, the user's gesture or line-of-sight position is detected (S29). As shown in FIG. 20, when the line-of-sight areas 95 and 96 are displayed on the display 28, Mr. A and Mr. B align their line-of-sight positions with the line-of-sight areas 95 or 96 corresponding to their opinions. On the other hand, as shown in FIG. 21, when a gesture instruction image is displayed on the display 28, according to the instruction, Mr. A and Mr. B make a gesture corresponding to their own opinion.

  The line-of-sight position of each person is detected by the method described above. Each person's gesture is performed by person image analysis processing. Specifically, shape pattern matching is performed by comparing the person image area F acquired in the person image analysis described above with the person image area F for each gesture stored in the person image operation pattern storage area 318 of the HDD 31. Thus, it is determined what kind of gesture the person has performed.

  Next, the type of opinion is specified (S30). If the intention display is based on the line-of-sight position and the line-of-sight position detected on the display 28 is within the line-of-sight area 95 of “YES”, the opinion is “YES”. When the line-of-sight position detected on the display 28 is within the line-of-sight area 96 of “NO”, the opinion is “NO”. When the intention display is based on a gesture, the meaning of the gesture whose type is detected is specified with reference to the gesture information 2231 stored in the gesture information storage area 223.

  When the opinions of each person displayed in the captured image are specified, the number of opinions of “YES” and the number of opinions of “NO” at the base are counted (S30). The count results are stored as opinion number data in the opinion number storage area 226 (see FIG. 12) of the RAM 22.

  Next, the voting image at the time of voting is stored in the voting image storage area 225 (see FIG. 12) of the RAM 22. Further, a result image 120 (see FIG. 22) in which the opinion result of each person is displayed is combined with the voted image stored in the voted image storage area 225 and stored in the result image storage area 227 (see FIG. 12). (S31). Individual images may be cut out from the voted images, and the result image 121 (see FIG. 23) arranged in “YES” and “NO” may be stored.

  Then, the opinion count data stored in the opinion count storage area 226 of the RAM 22 and the result image data which is the data of the result image 120 stored in the result image storage area 227 are transmitted to the chairperson terminal (S33), and processed. Ends.

  In this way, the majority decision is made based on the gesture information 2231 and the line-of-sight area information 2241 transmitted from the chairman terminal, and the result of aggregation at each base is transmitted to the chairman terminal, so that the summation result of each base is obtained at the chairman terminal. Can be put together. Further, the gesture information 2231 and the line-of-sight area information 2241 transmitted to each terminal device are different for each terminal device. That is, the gesture or the line-of-sight position for displaying will by majority decision is made different for each terminal. Thereby, even if the state of the user at the other site is shown on the multi-screen displayed on the display 28, the judgment at the own site is not affected. Further, as a result of the majority decision at each site, the result image data obtained by editing the voted image and the opinion number data are transmitted to the chairman terminal, so there is no need to analyze the opinions of each site at the chairman terminal.

  As described above, in the video conference system 1, when the terminal device 3 functioning as the chairman's terminal makes a resolution by majority vote on a certain agenda, the operation information (gestures) of the majority vote at each base of the terminal devices 4, 5, 6. Information, line-of-sight area information). In the operation information, the type of operation to be performed when displaying a will by majority decision is defined for each opinion. The type of action for opinions is different for each terminal. As a result, even if the state of the user at the other site is shown on the multi-screen displayed on the display 28, the operation is different from the own site, so that the judgment at the own site is not affected. Therefore, it is possible to make a reliable majority vote without being influenced by the opinions of other bases without increasing the number of parts. Also, the total number of majority information (opinion number data, result image data, etc.) made according to the operation information is transmitted from each base. By receiving and storing them, they can be displayed on the display 28. Thereby, it is possible to quickly grasp the voting result of a certain agenda item.

  In the above description, the gesture table 3121 and the line-of-sight area table 3131 stored in the HDD 31 correspond to the “motion information storage unit” of the present invention. The total table storage area 314 of the HDD 31 and the all terminal result image storage area 315 correspond to the “result information storage unit” of the present invention. The CPU 20 that executes the operation of S12 shown in FIG. 16 corresponds to the “operation information transmitting unit” of the present invention. The CPU 20 that executes the operation of S13 corresponds to the “result information receiving means” of the present invention. The CPU 20 that executes the operation of S18 corresponds to the “result information output unit” of the present invention.

  The CPU 20 that executes the operation of S30 shown in FIG. 17 corresponds to the “opinion counting means” of the present invention. The CPU 20 that executes the operation of S33 corresponds to the “result information transmitting unit” of the present invention. The CPU 20 that executes the operation of S23 corresponds to the “image size analyzing means” of the present invention. The CPU 20 that executes the operation of S24 corresponds to the “first image size determining means” of the present invention. The CPU 20 that executes the operation of S26 corresponds to the “second image size determining means” of the present invention. The gesture information 2231 shown in FIG. 13 corresponds to “first motion information” of the present invention. The line-of-sight area information 2241 shown in FIG. 14 corresponds to “second motion information” of the present invention. The CPU 20 that executes the operation of S29 in FIG. 17 corresponds to “gesture specifying means” and “line-of-sight area specifying means” of the present invention. The line-of-sight detection information storage area 222 of the RAM 22 corresponds to the “positional relationship information storage unit” of the present invention. The CPU 20 executing the operation of S29 in FIG. 17 corresponds to the “line-of-sight position specifying unit” of the present invention.

  The present invention is not limited to the above-described embodiment, and various modifications can be made. For example, in the above embodiment, the user's opinions are “YES” and “NO”, but for example, “YES”, “NO”, “Neither” can be added, and more opinions can be added. Good. Also. The type of opinion is not limited to “YES” and “NO”.

Further, in the above embodiment, the operation information (gesture information line-of-sight area information) is transmitted from the chairman terminal to each terminal, and the type of operation of each terminal is determined. You may make it select the operation | movement for transmitting to a terminal and displaying intention intention at each terminal at random.

  In the above-described embodiment, the distance between the user's eyeball 72 and the display 28 is approximated to be constant and is stored in advance as the predetermined distance L, but is not limited thereto. For example, the distance between the user's eyeball 72 and the display 28 is calculated at the start of a teleconference or when the line-of-sight direction is detected, and the user's line-of-sight position T on the display 28 is specified with the calculated value as the predetermined distance L. You may do it. As a method for calculating the predetermined distance L, a known method can be applied. For example, the following method can be applied.

  In this method, the user is photographed in a state where the CCD camera 38 and the user (the user's eyeball 72) are set to a predetermined distance in advance. The position of the pupil 75 is extracted for each eyeball 72 of the user from the photographed user image, and the extracted interval between the pupils 75 is stored in the HDD 31 in advance as a reference pupil interval. In general, the pupil interval in the captured image is inversely proportional to the distance between the pupil 75 and the CCD camera 38. Therefore, the distance between the CCD camera 38 and the eyeball 72 can be calculated from the actual pupil interval photographed by the CCD camera 38 and the reference pupil interval stored in advance. By storing the positional relationship between the CCD camera 38 and the display 28 in the HDD 31 in advance, the distance between the display 28 and the eyeball 72 can be calculated, and the calculated value can be used as the predetermined distance L.

DESCRIPTION OF SYMBOLS 1 Video conference system 2 Network 3 Terminal device 4 Terminal device 5 Terminal device 6 Terminal device 28 Display 31 Hard disk drive 34 Camera 37 Infrared light 38 CCD camera 2231 Gesture information 2241 Gaze area information 312 Gesture table storage area 3121 Gesture table 313 Gaze area table Storage area 3131 Line-of-sight area table 314 Total table storage area 3151 Total table

Claims (11)

A conference device that can be connected to a plurality of terminals via a communication line,
Display means for combining the captured images of the users output from the imaging means of the plurality of terminals and displaying a multi-image combining the captured images of the terminals on a display screen;
In a conference held between a plurality of users of the terminals, when the conference device functions as the chairman of the conference, a majority instruction means for instructing the terminal to perform a majority vote;
An assigning means for assigning, for each terminal, an operation for displaying a will for each type of opinion of a user when making a majority vote when the majority is instructed to the terminal by the majority instruction means;
Result information acquiring means for acquiring result information that is information on the total results of the opinions of the users who intentionally displayed by the operation assigned by the assigning means for each terminal;
Result information storage means for storing the result information acquired by the result information acquisition means;
A conference apparatus comprising: result information output means for outputting the result information stored in the result information storage means. The assigning means includes
Operation information storage means for storing type information, which is information about the type of operation, as operation information made different for each terminal;
The conference apparatus according to claim 1, further comprising operation information transmission means for transmitting the operation information stored in the operation information storage means to each of the plurality of terminals. Action information receiving means for receiving the action information transmitted from the chairman terminal which is a terminal functioning as the chairman among the plurality of terminals;
The conference apparatus according to claim 2, further comprising an operation display control unit that displays the operation information received by the operation information receiving unit on the display screen. User recognition means for recognizing a user from a photographed image photographed by the photographing means;
Action image storage means for storing an action image when the user recognized by the user recognition means operates according to the action information displayed on the display screen;
Based on the action image stored in the action image storage means, the action type specifying means for specifying for each user the type of action performed by the user recognized by the user recognition means according to the action information;
Opinion type specifying means for specifying the type of opinion indicated by the type of action specified by the action type specifying means from the action information received by the action information receiving means;
Opinion counting means for counting the number of users for each type of opinion specified by the opinion type specifying means;
4. The conference apparatus according to claim 2, further comprising a result information transmitting unit that transmits information of the results totaled by the opinion totaling unit to the terminal as the result information. Image size analysis means for analyzing the image size of the user from the captured image;
First image size determination means for determining whether or not the image size analyzed by the image size analysis means is greater than or equal to a first threshold;
The action for displaying the will is:
Gestures that are the user's physical movements,
A line-of-sight operation that directs the user to line of sight within a predetermined area on the display screen,
The operation information is
Assigning the gesture for each type of opinion of the user, and the first action information that makes the type of gesture different for each terminal;
For each type of user's opinion, the second operation information that assigns an area for directing the line of sight on the display screen, and that makes the position of the area different for each terminal,
The operation type specifying means includes:
A gesture specifying means for specifying the gesture of the user;
A line-of-sight area specifying means for specifying an area to which the line of sight is directed on the display screen of the user,
The operation display control means includes
When it is determined that the image size is less than the first threshold, the first operation information is displayed on the display screen,
The conference apparatus according to claim 2, wherein when it is determined that the image size is equal to or larger than the first threshold, the second operation information is displayed on the display screen. Gaze direction detecting means for detecting the gaze direction of the user from the captured image;
Positional relationship information storage means for storing positional relationship information that is information on the positional relationship between the user and the display screen;
Line-of-sight position specifying means for specifying a line-of-sight position on the display screen ahead of the line-of-sight direction detected by the line-of-sight direction based on the position relation information stored in the position relation information storage means,
The line-of-sight area specifying means includes
6. The region to which the line of sight is directed is specified from among regions allocated to the display screen for each type of opinion based on the line-of-sight position specified by the line-of-sight position specifying unit. The conference device described in 1. Second image size determination means for determining whether or not the image size analyzed by the image size analysis means is less than a second threshold smaller than the first threshold;
The operation display control means includes
When the first image size determining means determines that the image size is greater than or equal to the first threshold, only the second operation information is displayed on the display screen,
When the second image size determination means determines that the image size is less than the second threshold, only the first operation information is displayed on the display screen,
When the first image size determining means determines that the image size is less than the first threshold and the second image size determining means determines that the image size is equal to or greater than the second threshold, 7. The conference apparatus according to claim 5, wherein the first operation information and the second operation information are displayed on the display screen.   The conference apparatus according to claim 1, wherein the result information includes the operation image. In the result information storage means,
9. The conference apparatus according to claim 1, wherein agenda information, which is information on the agenda, and the result information related to the agenda information are stored in association with each other. There is a conference support method for a conference device that can be connected to a plurality of terminals via a communication line.
A display step of combining the captured images of the users output from the imaging means of the plurality of terminals and displaying a multi-image combining the captured images of each terminal on a display screen;
In a conference held between a plurality of users of the terminals, when the conference device functions as the chairman of the conference, a majority instruction step for instructing the terminal to perform a majority vote;
When assigning a majority vote to the terminal in the majority decision instruction step, an operation assignment step for assigning an action for displaying each will for each type of user opinion when making a majority vote,
A result information acquisition step of acquiring result information that is information on a result of a user's opinion totalization displayed by the operation assigned by the assigning unit from the terminal;
A result information storage step for storing the result information acquired in the result information acquisition step in a result information storage means;
A result information output step of outputting the result information stored in the result information storage means in the result information storage step.   A conference support program for a conference apparatus, comprising: a computer functioning as the various processing means of the conference apparatus according to claim 1.

Images