JP2009171119A - Remote conference management system, remote conference management method, and remote conference management program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a remote conference management system, a remote conference management method, and a remote conference management program for interconnecting conference terminals which are installed at remote places through a network, and certainly hold a conference using a video image and voice. <P>SOLUTION: A control section 21 of a conference management server 20 acquires images of attendants from TV conference terminals (10A, 10B), and performs authentication of the attendants of the remote conference. If the number of persons whose authentication is completed reaches a required number, the control section 21 of the conference management server 20 executes action permission processing. Moreover, the control section 21 analyzes the images from the TV conference terminals (10A, 10B), and performs attendant detection by tracing the attendants images. When attendants images cannot be followed, the control section 21 executes discrimination processing on a conference holding condition. When the number of attendants is small than a required number, and the conference holding condition cannot be maintained, the control section 21 outputs an alarm to respective TV conference terminals (10A, 10B). <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a remote conference management system, a remote conference management method, and a remote conference management program for connecting conference terminals installed at remote locations via a network and performing a conference using video and audio.

  2. Description of the Related Art Today, a teleconference management system is used in which companies can connect different offices via a network and perform a dialogue. However, with such a remote conference management system, it is difficult to authenticate that the conference participant is the person himself / herself and prove that the conference participant is actually participating in the conference.

  Therefore, a conference system for improving the reliability of the conference has been studied (for example, see Patent Documents 1 and 2). In the conference system described in Document 1, the authentication unit stores the consistency between the biometric information from the organizer transmitted from the conference host terminal, the identifier information, and the password, in the user data storage unit. The organizer is authenticated by checking the authentication data. Also, the minutes creation means creates the minutes from the contents recorded in the meeting content storage unit, and the signature reception means accepts the sent signature and records it together with the minutes in the meeting content storage unit.

Also, in the conference system described in Document 2, authentication information used to indicate to the participant terminal that the host is correct is generated on the organizer terminal side, and the authentication information is provided to the participant terminal. And request to join the meeting. Then, it waits for a meeting participation request from the organizer terminal, and for the meeting participation request from the organizer terminal, the operator determines whether or not the meeting connection is possible based on the authentication information provided and The connection process is performed.
JP 2002-271498 A (first page) JP 2003-167856 A (first page)

  Such a remote conference management system may be used even in situations where more stringency is required. For example, the Company Law states that the date and place of a board meeting (a director, executive officer, accounting advisor, corporate auditor, accounting auditor or shareholder who does not exist at the board meeting) The method of attendance in the case of attending at the same time is included ”(Enforcement Regulations, Article 101, Paragraph 3, Item 1), and it is assumed that the Board of Directors will be held by the video conference system. Meetings require strict requirements for meeting conditions, but in remote meetings, we recognize that all attendees are surely obtaining confirmations comparable to the various perceptions achieved by physical attendance. Specifically, the following conditions are required for each participant.

・ The existence of each attendee is mutually recognized by other attendees and can be confirmed.
・ Each participant's remarks must be heard accurately and completely by other participants and can be confirmed.

In Patent Documents 1 and 2, the authentication process is performed to improve the reliability of the remote conference management system. However, authentication and signature acts are acts at specific points in the conference, and authentication is performed. The viewpoint of continuity is not considered. That is, in such a meeting, it is necessary to obtain the recognition that all attendees have obtained confirmation that the above-mentioned conditions have been consistently maintained throughout the meeting. However, in Patent Documents 1 and 2, it cannot be proved that the condition that the condition during the conference is continued is maintained.

  The present invention has been made in view of the above-described problems, and the object of the present invention is to connect conference terminals installed at remote locations via a network and perform a conference reliably using video and audio. A remote conference management system, a remote conference management method, and a remote conference management program.

  In order to solve the above-mentioned problems, the invention described in claim 1 is characterized in that, for each user identifier, authentication information storage means for recording data relating to a feature amount for performing user authentication, and participation plans for each conference identifier. A system for managing a remote conference using a conference operation information storage unit that records data relating to a user identifier and a conference holding condition of a user, and a control unit connected to a conference terminal at a remote location, the control unit Acquires an image of a participant from the conference terminal, calculates a feature amount of a person in the image, and compares the feature amount with a feature amount of a scheduled participant recorded in the conference management information storage unit Then, an authentication processing means for executing a participant authentication process for specifying a participant, a participation status of the specified participant is checked against a meeting holding condition recorded in the conference management information storage means, and the participation status is Holding a meeting If the condition is satisfied, the start management means for outputting the meeting start permission, the participant is monitored in the image acquired from the conference terminal, and the participant image specified by the participant authentication process cannot be tracked. In this case, the gist of the present invention is to provide a continuation monitoring means for collating the participation status at this time with the conference holding condition and outputting a conference interruption instruction when the conference holding condition is not satisfied.

  According to a second aspect of the present invention, in the remote conference management system according to the first aspect, the conference management information storage unit records a necessary number of participants as a conference holding condition, and the start management unit and The gist is that the continuation monitoring means specifies the number of participants as the participation status and collates the number of participants with the necessary number recorded in the conference management information storage means.

  According to a third aspect of the present invention, in the remote conference management system according to the first or second aspect, a user identifier of an essential participant is recorded as a conference holding condition in the conference management information storage unit, The start management means and the continuation monitoring means specify the user identifier of the participant as the participation status, and the user identifier of the essential participant recorded in the conference management information storage means is included in the user identifier The gist is to check whether or not

  According to a fourth aspect of the present invention, in the remote conference management system according to any one of the first to third aspects, the control means determines a time interval for performing authentication again according to an image state, and the continuation The gist is characterized in that the monitoring means repeats the participant authentication process in accordance with the time interval.

  According to a fifth aspect of the present invention, in the remote conference management system according to any one of the first to fourth aspects, the continuation monitoring unit is acquired from the conference terminal based on a result of the participant authentication process. The gist of the present invention is to trace the image of the participant identified by the participant authentication process in the obtained image.

According to the sixth aspect of the present invention, there is provided an authentication information storage means for recording data relating to a feature quantity for performing user authentication for each user identifier, a user identifier of a prospective participant and a conference holding condition for each conference identifier. A method for managing a remote conference using a conference management information storage unit that records data related to the remote control unit and a control unit connected to a conference terminal at a remote location, wherein the control unit is connected to the participant from the conference terminal. Participation that acquires an image, calculates a feature amount of a person in the image, compares the feature amount with a feature amount of a prospective participant recorded in the conference management information storage unit, and identifies a participant Authentication process stage for executing the person authentication process, when the participation status of the identified participant is compared with the conference holding condition recorded in the conference management information storage means, and the participation status satisfies the conference holding condition , Start meeting Start management stage to output yes, participants are monitored in the image acquired from the conference terminal, and if it becomes impossible to track the image of the participant specified by the participant authentication process, the participation status at this time and the conference The gist is to execute a continuation monitoring stage that outputs a conference interruption instruction when the holding conditions are not met by meeting the holding conditions.

  The invention according to claim 7 is the authentication information storage means for recording the data relating to the characteristic amount for performing the identity authentication for each user identifier, the user identifier of the prospective participant and the meeting holding condition for each conference identifier. A program for managing a remote conference using a conference management information storage unit that records data related to and a control unit connected to a remote conference terminal, wherein the control unit is connected to the participant from the conference terminal. Participation that acquires an image, calculates a feature amount of a person in the image, compares the feature amount with a feature amount of a prospective participant recorded in the conference management information storage unit, and identifies a participant Authentication processing means for executing person authentication processing, the participation status of the identified participant is checked against the meeting holding conditions recorded in the conference management information storage means, and if the participation status satisfies the meeting holding conditions , Meeting Start management means for outputting a start permission, monitoring participants in the image acquired from the conference terminal, and if it becomes impossible to track the image of the participant identified by the participant authentication process, The gist is to make it function as a continuation monitoring means for outputting a conference interruption instruction when the conference holding conditions are collated and the conference holding conditions are not satisfied.

(Function)
According to the first, sixth, or seventh aspect of the invention, the control means acquires the participant's image from the conference terminal, calculates the feature amount of the person in the image, and stores the feature amount and the conference management information storage. Participant authentication processing for identifying the participant is executed by comparing the feature amount of the prospective participant recorded in the means. Then, the participation status of the participant is checked against the conference holding conditions recorded in the conference management information storage means, and if the participation status satisfies the conference holding conditions, a conference start permission is output. If the participant is monitored in the image acquired from the conference terminal and the participant's image specified by the participant authentication process cannot be tracked, the participation status at this time is compared with the conference holding condition, and the conference holding condition is determined. When it is not satisfied, a meeting interruption instruction is output. Thereby, a participant situation can be continuously grasped using an image, and a meeting can be strictly implemented.

  According to the second aspect of the present invention, the control means specifies the number of participants as the participation situation, and collates the number of participants with the required number recorded in the conference management information storage means. Thereby, a meeting can be strictly implemented, maintaining the number of participants required when holding a meeting continuously.

  According to the invention described in claim 3, the control means identifies the user identifier of the participant as the participation situation, and the use of the essential participant recorded in the conference management information storage means is included in this user identifier. Check whether the user identifier is included. Thereby, a meeting can be strictly implemented, maintaining the participation state of the person required when holding a meeting continuously.

  According to the invention described in claim 4, the control means determines the time interval for performing the authentication again according to the image state. Then, the participant authentication process is repeated according to the time interval. Thereby, a meeting can be strictly implemented while reducing the system processing load.

According to the fifth aspect of the present invention, the control unit tracks the image of the participant specified by the participant authentication process in the image acquired from the conference terminal based on the result of the participant authentication process. This makes it possible to maintain the conditions necessary for holding the conference while reducing the load of authentication processing.

  ADVANTAGE OF THE INVENTION According to this invention, the conference terminals installed in the remote place can be connected via a network, and a meeting can be reliably performed using an image | video and an audio | voice.

  Hereinafter, an embodiment of a remote conference management system embodying the present invention will be described with reference to FIGS. In this embodiment, as shown in FIG. 1, remote conference management used when a conference is performed by a remote participant using a plurality of TV conference terminals (10A, 10B) installed in a remote conference room. The system, the remote conference management method, and the remote conference management program will be described. Each TV conference terminal (10A, 10B) is mutually connected to the conference management server 20 via a network. Such a TV conference terminal (10A, 10B) and the conference management server 20 are each assigned an address (for example, an IP address) for specifying a communication destination, and using this address, another TV conference terminal is provided. And communication with the conference management server 20.

  Each TV conference terminal (10A, 10B) is a computer terminal for each user to conduct a remote conference. Specifically, the TV conference terminal (10A, 10B) includes a control unit 11 as a control unit, an operation unit 12, an input unit 13 as an input unit, and an output unit as an output unit. 14.

The operation unit 12 executes processing for receiving various user operations. The operation unit 12 includes a keyboard and a pointing device.
The input unit 13 executes a process of acquiring audio data and image data on the terminal side. Specifically, it consists of a microphone and an imaging camera.

The output unit 14 executes a process of outputting the other party's voice data and image data. Specifically, it consists of a speaker and a display.
As will be described later, the control unit 11 performs a process of transmitting audio data and image data acquired via the input unit 13 to the conference management server 20 and other TV conference terminals (10A, 10B). Furthermore, a process of outputting audio data and image data received from other TV conference terminals (10A, 10B) via the output unit 14 is executed.

  In addition, each of the TV conference terminals (10A, 10B) receives audio data and image data transmitted from one TV conference terminal (for example, the TV conference terminal 10A), and the other party's TV conference terminal (for example, the TV conference terminal 10B). Execute the operation check process to check whether it is output securely. The control unit 11 of the TV conference terminal (10A, 10B) executes the operation confirmation program, thereby transmitting a data to the partner terminal, receiving a data from the partner terminal, receiving an echo, and outputting an alarm. Each process is executed. For this reason, each TV conference terminal (10A, 10B) includes an audio / image data storage unit 15 as shown in FIG. The control unit 11 of each TV conference terminal (10A, 10B) includes an audio / image acquisition unit 11a, a recording unit 11b, an audio / image transfer unit 11c, an audio / image acquisition unit 11d, an echo extraction unit 11e, and an alarm output unit. 11f, echo canceling means 11g, recording means 11h, and sound / image output means 11i.

The voice / image data storage unit 15 temporarily stores voice data and image data acquired via the input unit 13 of the terminal itself.
The sound / image acquisition unit 11a executes a process of acquiring sound data and image data on the own terminal side via the input unit 13 of the own terminal.

The recording unit 11 b executes a process of recording the acquired audio data and image data in the audio / image data storage unit 15 via the input unit 13 of the own terminal.
The audio / image transfer means 11c sends the acquired audio data and image data to the TV conference terminals (10A, 10B) and the conference management server 20 that are the counterpart terminals via the network via the input unit 13 of the own terminal. Execute the process to send.

The sound / image acquisition unit 11d executes processing for acquiring sound data and image data from the counterpart terminal via the network.
The echo extraction unit 11e executes a process of extracting an echo signal of a transmission signal (audio data or image data) previously transmitted from the own terminal from the audio data or image data acquired from the counterpart terminal.

  The alarm output means 11f compares the intensity of the transmission signal (original signal level) and the intensity of the echo signal (echo signal level), and executes processing for outputting an alarm when the comparison value is equal to or less than the reference value.

The echo erasure unit 11g executes a process of erasing the echo signal of the audio data or image data transmitted from the own terminal from the audio data or image data acquired from the counterpart terminal.
The recording unit 11 h executes a process of recording, in the audio / image data storage unit 15, data obtained by deleting the echo signal from the audio data or image data acquired from the counterpart terminal.
The sound / image output unit 11 i executes a process of outputting the sound data or image data from which the echo signal has been deleted via the output unit 14.

  The TV conference terminals (10A, 10B) are connected to the conference management server 20 via a network as shown in FIG.

  The conference management server 20 is a computer system that manages a remote conference between the TV conference terminals (10A, 10B). The conference management server 20 includes a conference management data storage unit 22, a user data storage unit 23, a live data storage unit 24, and a minutes data storage unit 25.

  The conference management data storage unit 22 functions as a conference management information storage unit, and as shown in FIG. 2A, a conference management record 220 regarding the conference implementation schedule is recorded. This conference management record 220 is recorded when a participant is determined and a conference is set and registered. The conference management record 220 includes a data area for recording data relating to a conference management identifier, a scheduled start date and time, a conference holding condition, a user identifier, and a participation flag.

In the conference management identifier data area, data relating to an identifier for specifying each conference is recorded.
In the scheduled start date and time data area, data related to the scheduled date and time when this conference is started is recorded.

  In the meeting holding condition data area, data related to holding conditions for establishing this meeting is recorded. In the present embodiment, data relating to the number of participants (necessary number) necessary for establishing a conference is recorded.

In the user identifier data area, data relating to an identifier for specifying a user who participates in the conference is recorded.
In the participation flag data area, a flag for identifying a user who has participated in the conference is recorded.

  The user data storage unit 23 functions as an authentication information storage unit, and as shown in FIG. 2B, a user record 230 for specifying a user participating in the conference is recorded. This user record 230 is recorded when a user who uses this system is registered. The user record 230 includes a data area for recording pre-registration authentication data, authentication time, and sequential authentication data for the user identifier.

In the user identifier data area, data relating to an identifier for specifying a user who can use this system is recorded.
In the pre-registration authentication data area, authentication data registered in advance for performing personal authentication is recorded. In the pre-registration authentication data area, data relating to the face image of the person and the feature amount of the sound are recorded. In the participant confirmation process of the present embodiment, face authentication is used.

In the authentication time data area, data related to the date and time when authentication is performed sequentially by continuous authentication processing described later is recorded.
In the sequential authentication data area, data relating to the feature amount of the current face image calculated by the continuous authentication process is recorded.

  Audio data and image data acquired from each TV conference terminal (10A, 10B) are recorded in the live data storage unit 24 in time series together with a time stamp. With this time stamp, the date and time when each data was recorded can be specified. Furthermore, the live data storage unit 24 records hash values of audio data and image data.

  In the minutes data storage unit 25, the text data generated using the voice data acquired from each TV conference terminal (10A, 10B) is recorded in time series in association with the time stamp and the user identifier of the speaker. The

  The control unit 21 of the conference management server 20 executes each process such as an authentication process stage, a start management stage, and a continuous monitoring stage by executing the remote conference management program. Then, the control unit 21 includes a live data acquisition unit 21a, a live data recording unit 21b, a hash processing unit 21c, a conference progress management unit 21d, a person authentication unit 21e, an authentication management unit 21f, a speech confirmation unit 21g, a voice recognition unit 21h, It functions as the participant image tracking means 21i. In this embodiment, the conference progress management unit 21d functions as a start management unit, the person authentication unit 21e functions as an authentication processing unit, and the authentication management unit 21f and the participant image tracking unit 21i function as a continuation monitoring unit.

The actual condition data acquisition unit 21a executes processing for acquiring audio data and image data from each TV conference terminal (10A, 10B).
The live data recording unit 21b executes a process of sequentially recording audio data and image data acquired from each TV conference terminal (10A, 10B) in the live data storage unit 24.

  The hash processing means 21 c executes a process for calculating the hash value of the audio data and the image data stored in the live data storage unit 24. The hash processing unit 21c calculates a hash value of continuous audio data and image data in the remote conference together with the time stamp using a hash algorithm, and records the hash value in the live data storage unit 24. By using this hash value, it is possible to verify whether audio data or image data has been tampered with.

The conference progress management means 21d executes processing for managing the progress of the conference while confirming the participation status of the participants.
The personal authentication means 21e performs the personal authentication of the participants by using the actual data and the pre-registration authentication data acquired from each TV conference terminal (10A, 10B). In the present embodiment, personal authentication is performed using a face image included in the image data.

  The authentication management unit 21f executes processing for specifying the next authentication time according to the communication state. For this reason, the authentication management unit 21f retains an initial value of the authentication interval in the memory in order to determine a time interval (authentication interval) for continuous authentication. Further, the authentication management unit 21f records an authentication interval correction value (communication status correction value) corresponding to the communication speed and an authentication interval correction value (authentication status correction value) corresponding to the degree of coincidence of the feature amounts of the face images. Hold the correction table.

The speech confirmation means 21g executes processing for confirming the speech status of the participant.
The voice recognition means 21h specifies a speaker and executes a process of generating text data from voice data by voice recognition processing.
The participant image tracking means 21i executes processing for tracking an image of a user who has been authenticated by using an image analysis technique.

  Next, processing executed using the above system will be described with reference to FIGS. In the present embodiment, it is assumed that a remote conference is performed using two TV conference terminals (10A, 10B). First, the TV conference terminals (10A, 10B) used for the video conference are connected via the network. In this case, each TV conference terminal (10A, 10B) is relayed through the conference management server 20 and connected to the other TV conference terminal. Then, the control unit 11 of each TV conference terminal (10A, 10B) starts communication with the conference management server 20 and the other party's TV conference terminal (10A, 10B) via the network.

  In this embodiment, the participant confirmation process is executed at the start of the conference, and the continuity confirmation process is executed after the conference starts. In this continuity confirmation processing, continuation authentication processing, image tracking processing, and speech confirmation processing are executed. Further, the operation check process is continuously executed during the period from the start of the conference to the end of the conference.

(Participant confirmation process)
First, the participant confirmation process will be described with reference to FIG.
Here, the control unit 21 of the conference management server 20 executes a conference start request acquisition process (step S1-1). Specifically, the conference organizer accesses the conference management server 20 using a TV conference terminal (for example, the TV conference terminal 10A). In this case, the conference progress management unit 21d of the control unit 21 uses the current date and time to specify a conference in which the video conference is possible in the conference management data storage unit 22. Then, the conference progress management unit 21d generates a screen for displaying a list of identified conferences, and transmits the screen to the TV conference terminal 10A.

  Using the list display screen output to the output unit 14 of the TV conference terminal 10A, the conference organizer selects a conference to be held and inputs a conference start request. In this case, the TV conference terminal 10 </ b> A transmits a conference start request to the conference management server 20. This conference start request includes data related to the conference identifier of the conference designated by the TV conference terminal 10A. Thereby, the conference progress management means 21d acquires the conference start request.

  Next, the control unit 21 of the conference management server 20 executes a remote conference participant information acquisition process (step S1-2). Specifically, the conference progress management unit 21 d of the control unit 21 extracts the user identifier associated with the designated conference identifier from the conference management data storage unit 22. Then, the conference progress management means 21d issues an authentication instruction to the person authentication means 21e.

  Therefore, the control unit 21 of the conference management server 20 executes a participant image acquisition process (step S1-3). Specifically, the personal authentication unit 21e of the control unit 21 acquires the image data captured by the input unit 13 of each TV conference terminal (10A, 10B) via the live data acquisition unit 21a. Then, the personal authentication means 21e trims the face image portion for all persons included in the acquired photographed image.

  Next, the control unit 21 of the conference management server 20 executes a process for acquiring participant authentication information (step S1-4). Specifically, the personal authentication means 21 e of the control unit 21 extracts the user record 230 associated with the specified user identifier from the user data storage unit 23. And the personal authentication means 21e acquires the pre-registration authentication data of the user record 230.

  And the control part 21 of the meeting management server 20 performs an authentication process (step S1-5). Specifically, the personal authentication means 21e of the control unit 21 calculates the feature amount of the face image acquired from each TV conference terminal (10A, 10B). Next, the personal authentication means 21e compares the calculated feature amount with the feature amount of the pre-registration authentication data recorded in the user data storage unit 23 to authenticate the participant. Then, the personal authentication unit 21e notifies the conference progress management unit 21d of the authentication result. This authentication result includes the calculated feature value and the data relating to whether authentication is possible or not for the user identifier of the participant.

  Therefore, the control unit 21 of the conference management server 20 executes a determination process as to whether or not the conference holding condition is satisfied (step S1-6). Specifically, the conference progress management unit 21d of the control unit 21 calculates the number of participants who have been authenticated based on the authentication result acquired from the personal authentication unit 21e. And it compares with the required number as meeting holding conditions currently recorded on the meeting management data storage part 22. FIG.

  If the number of participants who have been authenticated is less than the required number and the meeting holding conditions are not satisfied (“NO” in step S1-6), the control unit 21 of the meeting management server 20 executes standby processing. (Step S1-7). Specifically, the conference progress management means 21d of the control unit 21 notifies each TV conference terminal (10A, 10B) that the required number has not been reached. Then, the conference progress management unit 21d repeats the participant image acquisition process (step S1-3) after waiting for a predetermined time.

  On the other hand, when the required number of persons has reached the required number (in the case of “YES” in step S1-6), the control unit 21 of the conference management server 20 executes a conference participant registration process (step S1-6). S1-8). Specifically, the conference progress management unit 21d of the control unit 21 records a participation flag in the conference management record 220 for the authenticated participant.

  And the control part 21 of the meeting management server 20 performs an action permission process (step S1-9). Specifically, the conference progress management means 21d of the control unit 21 notifies each TV conference terminal (10A, 10B) that the conference can be started (action).

(Continuity check processing)
Next, the conference continuity confirmation process will be described with reference to FIG. When the conference is started, the audio data and the image data acquired at the input unit 13 of each TV conference terminal (10A, 10B) are transmitted to the conference management server 20 via the network. In this case, the control unit 21 of the conference management server 20 executes a live data recording process (step S2). Specifically, the live data acquisition unit 21a of the control unit 21 records voice data and image data acquired from each TV conference terminal (10A, 10B) in the live data storage unit 24 via the live data recording unit 21b. To do. In this embodiment, the continuity confirmation process (continuation authentication process (
Steps S3-1 to S3-4), an image tracking process (step S4), and a speech confirmation process (step S5) described later are executed.

(Continuous authentication process)
Here, the continuous authentication process will be described with reference to FIG. As will be described later, this continuous authentication process is periodically executed at a predetermined timing.

  First, the control unit 21 of the conference management server 20 executes image verification processing for each participant (step S3-1). Here, motion detection according to the change state of the feature amount of the face image and plane detection by defocusing are performed.

  In the motion detection, the authentication management unit 21f of the control unit 21 instructs the personal authentication unit 21e to execute the personal authentication process for the image data acquired from each TV conference terminal (10A, 10B). The personal authentication means 21e compares the feature amount calculated from the face image with the feature amount of the pre-registration authentication data recorded in the user data storage unit 23 to authenticate the participant. Then, the personal authentication means 21e notifies the authentication management means 21f of the authentication result.

  Here, the authentication management unit 21f determines whether or not the feature amount of the face image acquired from the user authentication unit 21e matches the feature amount of the face image at the time of the previous user authentication recorded in the user data storage unit 23. To verify. Here, when the feature amount calculated at the time of the previous personal authentication matches the current feature amount, there is no movement in the face image.

  On the other hand, in the plane detection, the authentication management unit 21f executes focus change processing using the input unit 13 of the TV conference terminal (10A, 10B). Then, the authentication management unit 21f verifies the defocus status of the face and background in the acquired image data. If the entire image is defocused at a certain rate, a plane image is captured.

  As described above, when an abnormality is detected as a result of the authentication, such as a change state of the feature amount or the photographing of the planar image, the authentication management unit 21f outputs an alarm to the TV conference terminal (10A, 10B) on the other side.

  Next, the control unit 21 of the conference management server 20 executes authentication information recording processing (step S3-2). Specifically, the authentication management unit 21f of the control unit 21 records the feature amount of the face image acquired this time as authentication data in the user data storage unit 23 together with the authentication time.

  And the control part 21 of the meeting management server 20 performs the setting process of the next authentication time (step S3-3). Specifically, the authentication management unit 21f of the control unit 21 acquires a communication state (communication speed here) in the network, and acquires a communication state correction value in this communication state using a correction table. Further, the authentication management unit 21f acquires an authentication status correction value from the correction table using the degree of coincidence of the feature amounts of the face image. Then, the authentication management unit 21f calculates a time interval by multiplying both the correction value and the initial value, and determines the next authentication time using this time interval.

  Then, the control unit 21 of the conference management server 20 executes standby processing until the next authentication time (step S3-4). Specifically, the authentication management means 21f of the control unit 21 uses a timer to wait for the next authentication time. Then, when the authentication time has come, the image verification process (step S3-1) is re-executed.

(Image tracking process)
Next, an image tracking process is executed using FIG.
Here, the control unit 21 of the conference management server 20 executes location detection processing by image tracking (step S4-1). Specifically, the participant image tracking unit 21i of the control unit 21 performs image analysis on the image data obtained from each TV conference terminal (10A, 10B), where the location of the participant's image authenticated by the authentication management unit 21f is acquired. Identify by technology. Thereby, even when a plurality of participants are photographed in one image, each participant can be specified.

  And the control part 21 of the meeting management server 20 performs the determination process whether this face image can be tracked (step S4-2). Specifically, the participant image tracking means 21i of the control unit 21 has the face image specified by the image analysis technique within the imaging range of the image data acquired from the input unit 13 of the TV conference terminal (10A, 10B). Determine whether to do. Here, when the face image can be followed (in the case of “YES” in step S4-2), the location detection (step S4-1) by image tracking is continued.

  On the other hand, when the face image cannot be tracked (in the case of “NO” in step S4-2), the control unit 21 of the conference management server 20 executes a determination process as to whether or not the conference holding condition is maintained ( Step S4-3). Specifically, the participant image tracking means 21i of the control unit 21 calculates the number of participants who can follow the face image. Then, the participant image tracking means 21i compares the required number recorded in the conference management data storage unit 22 with the calculated number of participants.

  Here, when the number of participants is more than the required number and the meeting holding conditions are maintained (in the case of “YES” in step S4-3), the control unit 21 of the meeting management server 20 detects the location by image tracking. (Step S4-1) is continued.

  On the other hand, when the number of participants is less than the required number and the conference holding conditions cannot be maintained (in the case of “NO” in step S4-3), the control unit 21 of the conference management server 20 executes an alarm process ( Step S4-4). Specifically, the participant image tracking means 21i of the control unit 21 notifies each TV conference terminal (10A, 10B) that the meeting holding condition cannot be satisfied. In this case, the organizer interrupts the meeting.

(Speak confirmation process)
Next, the speech confirmation process will be described with reference to FIG.
Here, first, the control unit 21 of the conference management server 20 executes a determination process as to whether or not a speech by a participant has been detected (step S5-1). Specifically, the utterance confirmation unit 21 g of the control unit 21 acquires the participant's utterances from the voice data sequentially recorded in the live data storage unit 24. When the speech of the participant is not recorded in the voice data (in the case of “NO” in step S5-1), the speech confirmation unit 21g stands by.

  On the other hand, when the speech of the participant is detected in the voice data (in the case of “YES” in step S5-1), the control unit 21 of the conference management server 20 executes a voice recognition process (step S5-2). Specifically, the speech confirmation unit 21g of the control unit 21 instructs the speech recognition unit 21h to perform speech recognition processing on the participant's speech data.

  Then, the voice recognition unit 21 h performs voice recognition processing of the voice data using the voice feature amount of the pre-registration authentication data recorded in the user data storage unit 23. In this voice recognition process, a speaker is recognized and voice data is converted into text. Then, the voice recognition unit 21h records the recognition result (text data) in the minutes data storage unit 25 in association with the time stamp and the user identifier of the speaker.

  Next, the control unit 21 of the conference management server 20 executes face image acquisition processing at the time of speaking (step S5-3). Specifically, the utterance confirmation unit 21g of the control unit 21 specifies the image data of the utterer at the same timing as the time when the utterance was detected in the actual data storage unit 24, and acquires the face image.

  Further, the control unit 21 of the conference management server 20 executes a lip sync confirmation process (step S5-4). Specifically, the speech confirmation unit 21g of the control unit 21 trims the movement of the mouth in the face image acquired at the same timing as the speech sound.

  Then, the speech confirmation unit 21g performs a determination process as to whether or not the speech and the movement of the mouth are synchronized (step S5-5). Here, when the speech and the movement of the mouth are synchronized (lip sync) (in the case of “YES” in step S5-5), the control unit 21 of the conference management server 20 waits for the next speech. (Step S5-1).

  On the other hand, when the remark and the movement of the mouth are not synchronized (in the case of “NO” in step S5-5), the control unit 21 of the conference management server 20 executes an alarm process (step S5-6). Specifically, the speech confirmation unit 21g of the control unit 21 notifies each TV conference terminal (10A, 10B) that the sound and the image are not synchronized.

(Operation check process)
Next, the operation confirmation process will be described with reference to FIG. This process is executed independently at each TV conference terminal (10A, 10B). Here, it is assumed that audio data and image data are transmitted from the TV conference terminal 10A, which is a transmission source terminal, to the TV conference terminal 10B, which is a transmission destination terminal.

  First, the control unit 11 of the TV conference terminal 10A executes an original sound / image acquisition process in the transmission source terminal (step S6-1). Specifically, the sound / image acquisition unit 11 a of the control unit 11 acquires sound data and image data via the input unit 13.

  Next, the control unit 11 of the TV conference terminal 10A executes an original sound / image recording process (step S6-2). Specifically, the recording unit 11 b of the control unit 11 temporarily records the audio data and image data acquired by the audio / image acquisition unit 11 a in the audio / image data storage unit 15.

  Next, the control unit 11 of the TV conference terminal 10A executes a process of transferring the original sound / image to the transmission destination terminal (step S6-3). Specifically, the audio / image transfer unit 11c of the control unit 11 transmits audio data and image data to the TV conference terminal 10B via the network.

  Then, the TV conference terminal 10 </ b> B outputs the acquired audio data and image data via the output unit 14. In this case, the input unit 13 of the TV conference terminal 10 </ b> B acquires the image and sound output from the output unit 14. Then, the control unit 11 of the TV conference terminal 10B transmits the audio data and the image data acquired by the input unit 13 of the TV conference terminal 10B to the TV conference terminal 10A without performing echo cancellation processing.

  And the control part 11 of 10 A of video conference terminals performs the acquisition process of the audio | voice and image from a transmission destination terminal (step S6-4). Specifically, the audio / image acquisition unit 11d of the control unit 11 acquires audio data and image data from the TV conference terminal 10B via the network.

Next, the control unit 11 of the TV conference terminal 10 </ b> A executes an echo sound / image extraction process (
Step S6-5). Specifically, the echo extraction means 11e of the control unit 11 uses the audio / image data storage unit 15 to transmit the original audio data transmitted from the TV conference terminal 10A in the audio data and image data acquired from the TV conference terminal 10B. The signal component of the echo sound / image corresponding to the image data is extracted. The echo extraction unit 11e supplies the extracted echo signal to the alarm output unit 11f.

  Next, the control unit 11 of the TV conference terminal 10A performs a determination process as to whether or not the level of the echo signal is equal to or higher than a reference value (step S6-6). Specifically, the alarm output unit 11f of the control unit 11 calculates the echo signal level acquired from the echo extraction unit 11e. Further, the alarm output means 11 f uses the audio / image data storage unit 15 to calculate the signal level of the original audio / image. Then, the original signal level is compared with the echo signal level. Here, the alarm output means 11f calculates a comparison value obtained by dividing the echo signal level by the original signal level.

If this comparison value is greater than or equal to the reference value (“YES” in step S6-6), the control unit 11 of the TV conference terminal 10A continues the operation confirmation process.
On the other hand, when the comparison value does not reach the reference value (in the case of “NO” in step S6-6), the control unit 11 of the TV conference terminal 10A executes an alarm output process (step S6-7). . Specifically, the alarm output means 11f of the control unit 11 appropriately sends audio data and image data to the destination terminal (here, the TV conference terminal 10B) with respect to each TV conference terminal (10A, 10B). Notify that you have not. The above processing is executed in the same manner in the TV conference terminal 10B.

According to the remote conference management system of this embodiment, the following effects can be obtained.
In the present embodiment, the control unit 21 of the conference management server 20 acquires the participant image acquisition process (step S1-3), the participant authentication information acquisition process (step S1-4), and the authentication process (step S1-). 5) is executed. And the determination process of whether meeting meeting conditions are satisfied is performed (step S1-6). If the number of authenticated persons reaches the required number (in the case of “YES” in step S1-6), the control unit 21 of the conference management server 20 executes an action permission process (step S1-9). ). Thereby, even when the participant is at a remote place away from the meeting place, the meeting holding conditions can be confirmed and the meeting can be performed strictly.

  -In this embodiment, the control part 21 of the meeting management server 20 performs an image verification process (step S3-1). Here, motion detection according to the degree of coincidence of feature quantities of face images and plane detection by defocusing are performed. When an abnormality such as a feature amount change state or planar image shooting is detected, the authentication management unit 21f outputs an alarm to the TV conference terminal (10A, 10B) on the other side. Thereby, participation in the meeting in an unauthorized form can be suppressed.

  -In this embodiment, the control part 21 of the meeting management server 20 performs the setting process of the next authentication time using the communication state in a network, and the matching degree of the feature-value of a face image (step S3-3). Thereby, according to the situation, the meeting can be strictly performed while reducing the load of the authentication process. That is, for example, when the communication state is bad or the degree of coincidence of the feature amounts of the face images is bad, it is possible to increase the frequency of the authentication process and easily detect fraud. Furthermore, there is a possibility of fraud using a photo or the like when the feature amounts of the face images are completely matched. However, the fraud can be easily detected by increasing the frequency of authentication processing.

-In this embodiment, the control part 21 of the meeting management server 20 performs the location detection process by image tracking (step S4-1). When the face image cannot be followed (in the case of “NO” in step S4-2), the control unit 21 performs a determination process as to whether or not the meeting holding condition is maintained (step S4-3). . When the number of participants is less than the required number and the meeting holding conditions cannot be maintained (in the case of “NO” in step S4-3), the control unit 21 executes an alarm process (step S4-4). As a result, it is possible to continuously monitor the participation status of the participant by image analysis, taking over the result of the personal authentication performed intermittently. And when there is a problem in the participation situation, the conference can be interrupted, so that the conference can be managed strictly.

  In the present embodiment, when a participant's speech is detected in the voice data (in the case of “YES” in step S5-1), the control unit 21 of the conference management server 20 executes a voice recognition process (step S5- 2). Thereby, the burden of making minutes can be reduced. Furthermore, the control unit 21 of the conference management server 20 acquires a face image at the time of speaking (step S5-3), and executes a lip synchronization confirmation process (step S5-4). As a result, it is possible to reliably record a utterance using sound and an image. If there is a problem in the synchronization status between the sound and the image, the conference can be interrupted, so that the conference can be managed strictly.

  In this embodiment, the control unit 11 of the TV conference terminal 10A executes the original voice / image recording process (step S6-2) and the original voice / image transfer process to the destination terminal (step S6-3). To do. And the control part 11 of 10 A of video conference terminals performs the extraction process of an echo sound and image, when the audio | voice and image from a transmission destination terminal are acquired (step S6-4) (step S6-5). Here, when the comparison value obtained by dividing the echo signal level by the original signal level does not reach the reference value (in the case of “NO” in step S6-6), the control unit 11 of the TV conference terminal 10A performs alarm processing. Is executed (step S6-7). As a result, it is possible to confirm the communication status of the network and the operating status of the counterpart terminal using the echo signal, and to reliably supply voice and images to the counterpart side.

Moreover, you may change the said embodiment as follows.
In the above embodiment, the participant confirmation process is executed when a meeting start request is acquired. The timing of execution of the participant confirmation process is not limited to this, and it can be configured to be executed at the voting stage.

  In the above embodiment, data related to the necessary number for establishing a conference is recorded as a conference holding condition. Here, it is possible to set an essential participant as a meeting holding condition. In this case, data relating to the user identifier of the essential participant is recorded in the conference management data storage unit 22 as a conference holding condition. In step S1-6 or step S4-3, when confirming the meeting holding conditions, it is confirmed whether or not the participant has been authenticated.

  In the above embodiment, the operation check process is executed independently at each TV conference terminal (10A, 10B). Instead of this, the control unit 21 of the conference management server 20 may be configured to execute an operation confirmation process. In this case, the audio / image data storage unit 15 is provided in the conference management server 20. Then, the control unit 21 controls the sound / image acquisition means 11a, the recording means 11b, the sound / image transfer means 11c, the sound / image acquisition means 11d, the echo extraction means 11e, the alarm output means 11f, the echo cancellation means 11g, and the recording means 11h. And function as the sound / image output means 11i. In this case, when there is no failure in the communication status of the network, the conference management server 20 can check the operation status of each TV conference terminal (10A, 10B). Therefore, it is possible to reduce the processing load of each TV conference terminal (10A, 10B).

○ In the above embodiment, personal authentication is performed using face image authentication, but the personal authentication method is not limited to this, and other biometric authentication technologies such as fingerprint authentication and vein authentication can also be used. It is. In this case, a user authentication image (biometrics information) reading device is connected to each TV conference terminal and transmitted to the conference management server 20 via the network. In the conference management server 20, the user authentication process is executed using each user authentication image instead of the face image.

The system schematic of embodiment of this invention. It is explanatory drawing of the data recorded on each data memory | storage part, Comprising: (a) is a meeting management data memory | storage part, (b) is a user data memory | storage part. Explanatory drawing of the functional block of a video conference terminal. Explanatory drawing of the process sequence of this embodiment. Explanatory drawing of the process sequence of this embodiment. Explanatory drawing of the process sequence of this embodiment. Explanatory drawing of the process sequence of this embodiment. Explanatory drawing of the process sequence of this embodiment.

Explanation of symbols

  10A, 10B ... TV conference terminal, 11 ... control unit, 11a ... sound / image acquisition means, 11b ... recording means, 11c ... sound / image transfer means, 11d ... sound / image acquisition means, 11e ... echo extraction means, 11f ... Alarm output means, 11g ... echo canceling means, 11h ... recording means, 11i ... sound / image output means, 12 ... operation part, 13 ... input part, 14 ... output part, 20 ... conference management server, 21 ... control part, 21a ... Live data acquisition means, 21b ... Live data recording means, 21c ... Hash processing means, 21d ... Conference progress management means, 21e ... Personal authentication means, 21f ... Authentication management means, 21g ... Speech confirmation means, 21h ... Voice recognition means, 21i ... Participant image tracking means, 22 ... Conference management data storage unit, 23 ... User data storage unit, 24 ... Actual data storage unit, 25 ... Minutes data recording Part.

Claims (7)

Authentication information storage means for recording data relating to a feature amount for performing user authentication for each user identifier;
For each conference identifier, a conference management information storage means for recording data on the user identifier of the prospective participant and the conference holding conditions,
A system for managing a remote conference using a control means connected to a conference terminal at a remote location,
The control means is
An image of a participant is obtained from the conference terminal, a feature amount of a person in the image is calculated, and the feature amount is compared with a feature amount of a scheduled participant recorded in the conference management information storage unit. , An authentication processing means for executing a participant authentication process for identifying a participant,
A start management unit that compares the participation status of the identified participant with the conference holding condition recorded in the conference management information storage unit, and outputs a conference start permission when the participation status satisfies the conference holding condition ,
If the participant is monitored in the image acquired from the conference terminal and the participant's image specified by the participant authentication process cannot be tracked, the participation status at this time is compared with the conference holding condition, A remote conference management system comprising continuation monitoring means for outputting a conference interruption instruction when the holding conditions are not satisfied. In the conference management information storage means, the required number of participants is recorded as a conference holding condition,
2. The start management unit and the continuation monitoring unit specify the number of participants as a participation status, and collate the number of participants with the necessary number recorded in the conference management information storage unit. The remote conference management system described in 1. In the conference management information storage means, a user identifier of an essential participant is recorded as a conference holding condition,
The start management unit and the continuation monitoring unit specify the user identifier of the participant as a participation situation, and the user identifier of the essential participant recorded in the conference management information storage unit is included in the user identifier. 3. The remote conference management system according to claim 1, wherein the remote conference management system checks whether or not it is included. The control means determines a time interval for performing authentication again according to the image state,
The remote conference management system according to claim 1, wherein the continuation monitoring unit repeats the participant authentication process according to the time interval.   The said continuous monitoring means tracks the image of the participant specified by the said participant authentication process in the image acquired from the said conference terminal based on the result of the said participant authentication process. 5. The remote conference management system according to any one of 4 above. Authentication information storage means for recording data relating to a feature amount for performing user authentication for each user identifier;
For each conference identifier, a conference management information storage means for recording data on the user identifier of the prospective participant and the conference holding conditions,
A method for managing a remote conference using a control means connected to a remote conference terminal,
The control means is
An image of a participant is obtained from the conference terminal, a feature amount of a person in the image is calculated, and the feature amount is compared with a feature amount of a scheduled participant recorded in the conference management information storage unit. , An authentication process stage for performing a participant authentication process for identifying a participant,
A start management step of collating the participation status of the identified participant with the conference holding condition recorded in the conference management information storage means, and outputting a conference start permission if the participation status satisfies the conference holding condition ,
If the participant is monitored in the image acquired from the conference terminal and the participant's image specified by the participant authentication process cannot be tracked, the participation status at this time is compared with the conference holding condition, A remote conference management method, comprising: executing a continuous monitoring step of outputting a conference interruption instruction when the holding condition is not satisfied. Authentication information storage means for recording data relating to a feature amount for performing user authentication for each user identifier;
For each conference identifier, a conference management information storage means for recording data on the user identifier of the prospective participant and the conference holding conditions,
A program for managing a remote conference using a control means connected to a remote conference terminal,
The control means;
An image of a participant is obtained from the conference terminal, a feature amount of a person in the image is calculated, and the feature amount is compared with a feature amount of a scheduled participant recorded in the conference management information storage unit. , An authentication processing means for executing a participant authentication process for identifying a participant,
A start management unit that compares the participation status of the identified participant with the conference holding condition recorded in the conference management information storage unit, and outputs a conference start permission when the participation status satisfies the conference holding condition ,
If the participant is monitored in the image acquired from the conference terminal and the participant's image specified by the participant authentication process cannot be tracked, the participation status at this time is compared with the conference holding condition, A remote conference management program which functions as a continuous monitoring means for outputting a conference interruption instruction when the holding conditions are not satisfied.

Images