JP4728357B2 - Remote conference management system, remote conference management method, and remote conference operation confirmation program - Google Patents

Description

  The present invention relates to a remote conference management system, a remote conference management method, and a remote conference management program for connecting conference terminals installed at remote locations via a network and performing a conference using video and audio.

  2. Description of the Related Art Today, a teleconference management system is used in which companies can connect different offices via a network and perform a dialogue. However, with such a remote conference management system, it is difficult to authenticate that the conference participant is the person himself / herself and prove that the conference participant is actually participating in the conference.

  Therefore, a conference system for improving the reliability of the conference has been studied (for example, see Patent Documents 1 and 2). In the conference system described in Document 1, the authentication unit stores the consistency between the biometric information from the organizer transmitted from the conference host terminal, the identifier information, and the password, in the user data storage unit. The organizer is authenticated by checking the authentication data. Also, the minutes creation means creates the minutes from the contents recorded in the meeting content storage unit, and the signature reception means accepts the sent signature and records it together with the minutes in the meeting content storage unit.

Also, in the conference system described in Document 2, authentication information used to indicate to the participant terminal that the host is correct is generated on the organizer terminal side, and the authentication information is provided to the participant terminal. And request to join the meeting. Then, it waits for a meeting participation request from the organizer terminal, and for the meeting participation request from the organizer terminal, the operator determines whether or not the meeting connection is possible based on the authentication information provided and The connection process is performed.
JP 2002-271498 A (first page) JP 2003-167856 A (first page)

  Such a remote conference management system may be used even in situations where more stringency is required. For example, the Companies Act states that the date and place of a board meeting (a director, executive officer, accounting advisor, auditor, accounting auditor or shareholder who does not exist at the board meeting) The method of attendance in the case of attending at the same time is included ”(Enforcement Regulations, Article 101, Paragraph 3, Item 1), and it is assumed that the Board of Directors will be held by the video conference system. Meetings require strict requirements for meeting conditions, but in remote meetings, we recognize that all attendees are surely obtaining confirmations comparable to the various perceptions achieved by physical attendance. Specifically, the following conditions are required for each participant.

・ The existence of each attendee is mutually recognized by other attendees and can be confirmed.
・ Each participant's remarks must be heard accurately and completely by other participants and can be confirmed.

In Patent Documents 1 and 2, the authentication process is performed to improve the reliability of the remote conference management system. However, the existence and remarks of each attendee described above are confirmed by the conference terminal and the network. Realized in operation. Therefore, if any of these are defective,
A reliable remote conference cannot be realized.

  The present invention has been made in view of the above-described problems, and the object of the present invention is to connect conference terminals installed at remote locations via a network and perform a conference reliably using video and audio. A remote conference management system, a remote conference management method, and a remote conference operation confirmation program.

In order to solve the above problems, a first aspect of the present invention, the operation confirmation control means for operation check of the far septum conference, and the reference value data storage means for storing a reference value of the echo signal, the transmission data storage Means and a first conference terminal provided with an input unit; and a second conference terminal connected to the first conference terminal via a network and provided with an input unit and an output unit.
And a teleconference management system, the operation confirmation control means, the first conference terminal first commentary recording means for recording the data in the transmission data storage unit acquired via the input unit of the second transmitting means for transmitting the first Commentary data to the conference terminals, an output of the first Commentary data at the output of the second conference terminal is input at the input of the second conference terminal, the first Receiving means for receiving second live data transmitted from the two conference terminals via the network, and extracting an echo signal of the first live data recorded in the transmission data storage means from the second live data. echo extraction means, wherein when the magnitude of the echo signal has been equal to or less than the reference value stored in the reference value data storage means, Abstract, further comprising an alarm output means for outputting an alarm To.

Invention according to claim 2, in the remote conference management system according to claim 1, wherein the first conference terminal may further include the operation confirmation control unit output unit, before Symbol from said second Commentary Statistics The gist of the present invention is that it further comprises means for deleting the echo signal of the first live data and outputting it from the output unit of the first conference terminal.

According to a third aspect of the present invention, in the remote conference management system according to the first or second aspect, the first conference terminal further includes an actual data storage unit that records actual data of the remote conference, and the operation check The gist of the present invention is that the control means further comprises means for recording the second actual data from which the echo signal has been deleted in the actual data storage means.

A fourth aspect of the present invention is the remote conference management system according to any one of the first to third aspects, wherein the first and second live data include voice data.
A fifth aspect of the present invention is the teleconference management system according to any one of the first to fourth aspects, wherein the first and second live data include image data.

The invention according to claim 6, comprising an operation confirmation control means for performing an operation check of the far septum conference, and the reference value data storage means for storing a reference value of the echo signal, the transmission data storage unit, an input unit Using a remote conference management system comprising a first conference terminal and a second conference terminal connected to the first conference terminal via a network and having an input unit and an output unit , a method for managing the remote conference, the operation confirmation control means, a recording step of recording the first commentary data acquired through the input portion of the first conference terminal to the transmitting data storage means and the transmission step of transmitting the first Commentary data to the second conference terminal, an output of the first Commentary data at the output of the second conference terminal, the input portion of the second conference terminal Entered in Extracting a second commentary data transmitted from the second conference terminal, a receiving step of receiving via the network, from the second Commentary data, an echo signal of the first commentary data recorded in the transmission data storage unit The gist of the present invention is to execute an echo extraction step of performing an alarm and an alarm output step of outputting an alarm when the magnitude of the echo signal is equal to or smaller than a reference value stored in the reference value data storage means .

The invention according to claim 7, comprising an operation confirmation control means for performing an operation check of the far septum conference, and the reference value data storage means for storing a reference value of the echo signal, the transmission data storage unit, an input unit Using a remote conference management system comprising a first conference terminal and a second conference terminal connected to the first conference terminal via a network and having an input unit and an output unit , a program for performing an operation check during teleconferencing, the operation confirmation control unit records the first commentary data acquired through the input portion of the first conference terminal to the transmitting data storage means recording means, wherein the relative second conference terminal first commentary transmitting means for transmitting data, the output of the first Commentary data at the output of the second conference terminal is the input of the second conference terminal Input in the department Is the second commentary data transmitted from the second conference terminal, receiving means for receiving via the network, from the second Commentary data, first Commentary echo data stored in the transmission data storage unit The gist is to function as an alarm output means for outputting an alarm when an echo extraction means for extracting a signal and the magnitude of the echo signal is equal to or smaller than a reference value stored in the reference value data storage means .

(Function)
According to the invention described in claim 1, 6 or 7 , the operation confirmation control means records the first live data acquired in the first conference terminal in the transmission data storage means, and Send the first live data. Then, the second actual data acquired at the second conference terminal is received, and an echo signal of the first actual data recorded in the transmission data storage means is extracted from the second actual data. When the magnitude of this echo signal is below the reference value, an alarm is output. Thereby, the output state of the actual condition data from the 1st conference terminal in the 2nd conference terminal can be checked using an echo signal. If there is a problem, the alarm can be output so that the conference can be carried out reliably.

According to the second aspect of the present invention, the operation confirmation control means deletes the echo signal of the first live data from the second live data and outputs it from the first conference terminal in the first conference terminal. Thereby, a meeting can be implemented using the actual condition data which does not contain an echo signal.

According to the third aspect of the present invention, the operation confirmation control means records the second live data from which the echo signal has been deleted in the live data storage means. As a result, it is possible to record live data that does not include an echo signal.

According to the fourth aspect of the present invention, the first and second live data include voice data. Thereby, it is possible to confirm the transmission of voice.
According to the fifth aspect of the present invention, the first and second live data include image data. Thereby, transmission of an image can be confirmed.

  ADVANTAGE OF THE INVENTION According to this invention, the conference terminals installed in the remote place can be connected via a network, and a meeting can be reliably performed using a video and an audio | voice.

  Hereinafter, an embodiment of a remote conference management system embodying the present invention will be described with reference to FIGS. In this embodiment, as shown in FIG. 1, remote conference management used when a conference is performed by a remote participant using a plurality of TV conference terminals (10A, 10B) installed in a remote conference room. The system, the remote conference management method, and the remote conference operation confirmation program will be described. Each TV conference terminal (10A, 10B) is mutually connected to the conference management server 20 via a network. Such a TV conference terminal (10A, 10B) and the conference management server 20 are each assigned an address (for example, an IP address) for specifying a communication destination, and using this address, another TV conference terminal is provided. And communication with the conference management server 20.

  Each TV conference terminal (10A, 10B) is a computer terminal for each user to conduct a remote conference. Specifically, the TV conference terminal (10A, 10B) includes a control unit 11 as an operation confirmation control unit, an operation unit 12, an input unit 13 as an input unit, and an output unit configured by a CPU, a RAM, and a ROM. An output unit 14 is provided.

The operation unit 12 executes processing for receiving various user operations. The operation unit 12 includes a keyboard and a pointing device.
The input unit 13 executes a process of acquiring audio data and image data on the terminal side. Specifically, it consists of a microphone and an imaging camera.

The output unit 14 executes a process of outputting the other party's voice data and image data. Specifically, it consists of a speaker and a display.
As will be described later, the control unit 11 performs a process of transmitting audio data and image data acquired via the input unit 13 to the conference management server 20 and other TV conference terminals (10A, 10B). Furthermore, a process of outputting audio data and image data received from other TV conference terminals (10A, 10B) via the output unit 14 is executed.

  In addition, each of the TV conference terminals (10A, 10B) receives audio data and image data transmitted from one TV conference terminal (for example, the TV conference terminal 10A), and the other party's TV conference terminal (for example, the TV conference terminal 10B). Execute the operation check process to check whether it is output securely. The control unit 11 of the TV conference terminal (10A, 10B) executes the remote conference operation confirmation program, so that the data recording stage, transmission stage, data reception stage from the counterpart terminal, echo extraction to the counterpart terminal are executed. Each process such as a stage and an alarm output stage is executed. Therefore, each TV conference terminal (10A, 10B) includes a voice / image data storage unit 15 as a transmission data storage unit and an actual data storage unit as shown in FIG. The control unit 11 of each TV conference terminal (10A, 10B) includes an audio / image acquisition unit 11a, a recording unit 11b, an audio / image transfer unit 11c, an audio / image acquisition unit 11d, an echo extraction unit 11e, and an alarm output unit. 11f, echo canceling means 11g, recording means 11h, and sound / image output means 11i.

The voice / image data storage unit 15 temporarily stores voice data and image data acquired via the input unit 13 of the terminal itself.
The sound / image acquisition unit 11a executes a process of acquiring sound data and image data on the own terminal side via the input unit 13 of the own terminal.

The recording unit 11 b executes a process of recording the acquired audio data and image data in the audio / image data storage unit 15 via the input unit 13 of the own terminal.
The sound / image transfer means 11c functions as a transmission means for transmitting the first live data. Specifically, the process of transmitting the acquired audio data and image data to the TV conference terminals (10A, 10B) and the conference management server 20 that are the counterpart terminals via the network via the input unit 13 of the own terminal. Execute.

  The sound / image acquisition unit 11d functions as a reception unit that receives the second live data. Specifically, processing for acquiring audio data and image data from the counterpart terminal is executed via the network.

  The echo extraction unit 11e executes a process of extracting an echo signal of a transmission signal (audio data or image data) previously transmitted from the own terminal from the audio data or image data acquired from the counterpart terminal.

  The alarm output unit 11f includes a reference value data storage unit that stores a reference value of the echo signal. Then, the alarm output means 11f compares the intensity of the transmission signal (original signal level) with the intensity of the echo signal (echo signal level), and executes a process of outputting an alarm if the comparison value is below the reference value. To do.

The echo erasure unit 11g executes a process of erasing the echo signal of the audio data or image data transmitted from the own terminal from the audio data or image data acquired from the counterpart terminal.
The recording unit 11 h executes a process of recording, in the audio / image data storage unit 15, data obtained by deleting the echo signal from the audio data or image data acquired from the counterpart terminal.
The sound / image output unit 11 i executes a process of outputting the sound data or image data from which the echo signal has been deleted via the output unit 14.

  The TV conference terminals (10A, 10B) are connected to the conference management server 20 via a network as shown in FIG.

  The conference management server 20 is a computer system that manages a remote conference between the TV conference terminals (10A, 10B). The conference management server 20 includes a conference management data storage unit 22, a user data storage unit 23, a live data storage unit 24, and a minutes data storage unit 25.

  The conference management data storage unit 22 functions as a conference management information storage unit, and as shown in FIG. 2A, a conference management record 220 regarding the conference implementation schedule is recorded. This conference management record 220 is recorded when a participant is determined and a conference is set and registered. The conference management record 220 includes a data area for recording data relating to a conference management identifier, a scheduled start date and time, a conference holding condition, a user identifier, and a participation flag.

In the conference management identifier data area, data relating to an identifier for specifying each conference is recorded.
In the scheduled start date and time data area, data related to the scheduled date and time when this conference is started is recorded.

  In the meeting holding condition data area, data related to holding conditions for establishing this meeting is recorded. In the present embodiment, data relating to the number of participants (necessary number) necessary for establishing a conference is recorded.

In the user identifier data area, data relating to an identifier for specifying a user who participates in the conference is recorded.
In the participation flag data area, a flag for identifying a user who has participated in the conference is recorded.

  The user data storage unit 23 functions as an authentication information storage unit, and as shown in FIG. 2B, a user record 230 for specifying a user participating in the conference is recorded. This user record 230 is recorded when a user who uses this system is registered. The user record 230 includes a data area for recording pre-registration authentication data, authentication time, and sequential authentication data for the user identifier.

In the user identifier data area, data relating to an identifier for specifying a user who can use this system is recorded.
In the pre-registration authentication data area, authentication data registered in advance for performing personal authentication is recorded. In the pre-registration authentication data area, data relating to the face image of the person and the feature amount of the sound are recorded. In the participant confirmation process of the present embodiment, face authentication is used.

In the authentication time data area, data related to the date and time when authentication is performed sequentially by continuous authentication processing described later is recorded.
In the sequential authentication data area, data relating to the feature amount of the current face image calculated by the continuous authentication process is recorded.

  Audio data and image data acquired from each TV conference terminal (10A, 10B) are recorded in the live data storage unit 24 in time series together with a time stamp. With this time stamp, the date and time when each data was recorded can be specified. Furthermore, the live data storage unit 24 records hash values of audio data and image data.

  In the minutes data storage unit 25, the text data generated using the voice data acquired from each TV conference terminal (10A, 10B) is recorded in time series in association with the time stamp and the user identifier of the speaker. The

  The control unit 21 of the conference management server 20 executes each process such as an authentication process stage, a start management stage, and a continuous monitoring stage by executing the remote conference management program. Then, the control unit 21 includes a live data acquisition unit 21a, a live data recording unit 21b, a hash processing unit 21c, a conference progress management unit 21d, a person authentication unit 21e, an authentication management unit 21f, a speech confirmation unit 21g, a voice recognition unit 21h, It functions as the participant image tracking means 21i. In this embodiment, the conference progress management unit 21d functions as a start management unit, the person authentication unit 21e functions as an authentication processing unit, and the authentication management unit 21f and the participant image tracking unit 21i function as a continuation monitoring unit.

The actual condition data acquisition unit 21a executes processing for acquiring audio data and image data from each TV conference terminal (10A, 10B).
The live data recording unit 21b executes a process of sequentially recording audio data and image data acquired from each TV conference terminal (10A, 10B) in the live data storage unit 24.

  The hash processing means 21 c executes a process for calculating the hash value of the audio data and the image data stored in the live data storage unit 24. The hash processing unit 21c calculates a hash value of continuous audio data and image data in the remote conference together with the time stamp using a hash algorithm, and records the hash value in the live data storage unit 24. By using this hash value, it is possible to verify whether audio data or image data has been tampered with.

The conference progress management means 21d executes processing for managing the progress of the conference while confirming the participation status of the participants.
The personal authentication means 21e performs the personal authentication of the participants by using the actual data and the pre-registration authentication data acquired from each TV conference terminal (10A, 10B). In the present embodiment, personal authentication is performed using a face image included in the image data.

  The authentication management unit 21f executes processing for specifying the next authentication time according to the communication state. For this reason, the authentication management unit 21f retains an initial value of the authentication interval in the memory in order to determine a time interval (authentication interval) for continuous authentication. Further, the authentication management unit 21f records an authentication interval correction value (communication status correction value) corresponding to the communication speed and an authentication interval correction value (authentication status correction value) corresponding to the degree of coincidence of the feature amounts of the face images. Hold the correction table.

The speech confirmation means 21g executes processing for confirming the speech status of the participant.
The voice recognition means 21h specifies a speaker and executes a process of generating text data from voice data by voice recognition processing.
The participant image tracking means 21i executes processing for tracking an image of a user who has been authenticated by using an image analysis technique.

  Next, processing executed using the above system will be described with reference to FIGS. In the present embodiment, it is assumed that a remote conference is performed using two TV conference terminals (10A, 10B). First, the TV conference terminals (10A, 10B) used for the video conference are connected via the network. In this case, each TV conference terminal (10A, 10B) is relayed through the conference management server 20 and connected to the other TV conference terminal. Then, the control unit 11 of each TV conference terminal (10A, 10B) starts communication with the conference management server 20 and the other party's TV conference terminal (10A, 10B) via the network.

  In this embodiment, the participant confirmation process is executed at the start of the conference, and the continuity confirmation process is executed after the conference starts. In this continuity confirmation processing, continuation authentication processing, image tracking processing, and speech confirmation processing are executed. Further, the operation check process is continuously executed during the period from the start of the conference to the end of the conference.

(Participant confirmation process)
First, the participant confirmation process will be described with reference to FIG.
Here, the control unit 21 of the conference management server 20 executes a conference start request acquisition process (step S1-1). Specifically, the conference organizer accesses the conference management server 20 using a TV conference terminal (for example, the TV conference terminal 10A). In this case, the conference progress management unit 21d of the control unit 21 uses the current date and time to specify a conference in which the video conference is possible in the conference management data storage unit 22. Then, the conference progress management unit 21d generates a screen for displaying a list of identified conferences, and transmits the screen to the TV conference terminal 10A.

  Using the list display screen output to the output unit 14 of the TV conference terminal 10A, the conference organizer selects a conference to be held and inputs a conference start request. In this case, the TV conference terminal 10 </ b> A transmits a conference start request to the conference management server 20. This conference start request includes data related to the conference identifier of the conference designated by the TV conference terminal 10A. Thereby, the conference progress management means 21d acquires the conference start request.

  Next, the control unit 21 of the conference management server 20 executes a remote conference participant information acquisition process (step S1-2). Specifically, the conference progress management unit 21 d of the control unit 21 extracts the user identifier associated with the designated conference identifier from the conference management data storage unit 22. Then, the conference progress management means 21d issues an authentication instruction to the person authentication means 21e.

  Therefore, the control unit 21 of the conference management server 20 executes a participant image acquisition process (step S1-3). Specifically, the personal authentication unit 21e of the control unit 21 acquires the image data captured by the input unit 13 of each TV conference terminal (10A, 10B) via the live data acquisition unit 21a. Then, the personal authentication means 21e trims the face image portion for all persons included in the acquired photographed image.

Next, the control unit 21 of the conference management server 20 executes a process for acquiring participant authentication information (step S1-4). Specifically, the personal authentication means 21 e of the control unit 21 extracts the user record 230 associated with the specified user identifier from the user data storage unit 23. And the personal authentication means 21e acquires the pre-registration authentication data of the user record 230.

  And the control part 21 of the meeting management server 20 performs an authentication process (step S1-5). Specifically, the personal authentication means 21e of the control unit 21 calculates the feature amount of the face image acquired from each TV conference terminal (10A, 10B). Next, the personal authentication means 21e compares the calculated feature amount with the feature amount of the pre-registration authentication data recorded in the user data storage unit 23 to authenticate the participant. Then, the personal authentication unit 21e notifies the conference progress management unit 21d of the authentication result. This authentication result includes the calculated feature value and the data relating to whether authentication is possible or not for the user identifier of the participant.

  Therefore, the control unit 21 of the conference management server 20 executes a determination process as to whether or not the conference holding condition is satisfied (step S1-6). Specifically, the conference progress management unit 21d of the control unit 21 calculates the number of participants who have been authenticated based on the authentication result acquired from the personal authentication unit 21e. And it compares with the required number as meeting holding conditions currently recorded on the meeting management data storage part 22. FIG.

  If the number of participants who have been authenticated is less than the required number and the meeting holding conditions are not satisfied (“NO” in step S1-6), the control unit 21 of the meeting management server 20 executes standby processing. (Step S1-7). Specifically, the conference progress management means 21d of the control unit 21 notifies each TV conference terminal (10A, 10B) that the required number has not been reached. Then, the conference progress management unit 21d repeats the participant image acquisition process (step S1-3) after waiting for a predetermined time.

  On the other hand, when the required number of persons has reached the required number (in the case of “YES” in step S1-6), the control unit 21 of the conference management server 20 executes a conference participant registration process (step S1-6). S1-8). Specifically, the conference progress management unit 21d of the control unit 21 records a participation flag in the conference management record 220 for the authenticated participant.

  And the control part 21 of the meeting management server 20 performs an action permission process (step S1-9). Specifically, the conference progress management means 21d of the control unit 21 notifies each TV conference terminal (10A, 10B) that the conference can be started (action).

(Continuity check processing)
Next, the conference continuity confirmation process will be described with reference to FIG. When the conference is started, the audio data and the image data acquired at the input unit 13 of each TV conference terminal (10A, 10B) are transmitted to the conference management server 20 via the network. In this case, the control unit 21 of the conference management server 20 executes a live data recording process (step S2). Specifically, the live data acquisition unit 21a of the control unit 21 records voice data and image data acquired from each TV conference terminal (10A, 10B) in the live data storage unit 24 via the live data recording unit 21b. To do. In this embodiment, as the continuity confirmation process, a continuation authentication process (steps S3-1 to S3-4), an image tracking process (step S4), and a statement confirmation process (step S5) described later are executed.

(Continuous authentication process)
Here, the continuous authentication process will be described with reference to FIG. As will be described later, this continuous authentication process is periodically executed at a predetermined timing.

  First, the control unit 21 of the conference management server 20 executes image verification processing for each participant (step S3-1). Here, motion detection according to the change state of the feature amount of the face image and plane detection by defocusing are performed.

  In the motion detection, the authentication management unit 21f of the control unit 21 instructs the personal authentication unit 21e to execute the personal authentication process for the image data acquired from each TV conference terminal (10A, 10B). The personal authentication means 21e compares the feature amount calculated from the face image with the feature amount of the pre-registration authentication data recorded in the user data storage unit 23 to authenticate the participant. Then, the personal authentication means 21e notifies the authentication management means 21f of the authentication result.

  Here, the authentication management unit 21f determines whether or not the feature amount of the face image acquired from the user authentication unit 21e matches the feature amount of the face image at the time of the previous user authentication recorded in the user data storage unit 23. To verify. Here, when the feature amount calculated at the time of the previous personal authentication matches the current feature amount, there is no movement in the face image.

  On the other hand, in the plane detection, the authentication management unit 21f executes focus change processing using the input unit 13 of the TV conference terminal (10A, 10B). Then, the authentication management unit 21f verifies the defocus status of the face and background in the acquired image data. If the entire image is defocused at a certain rate, a plane image is captured.

  As described above, when an abnormality is detected as a result of the authentication, such as a change state of the feature amount or the photographing of the planar image, the authentication management unit 21f outputs an alarm to the TV conference terminal (10A, 10B) on the other side.

  Next, the control unit 21 of the conference management server 20 executes authentication information recording processing (step S3-2). Specifically, the authentication management unit 21f of the control unit 21 records the feature amount of the face image acquired this time as authentication data in the user data storage unit 23 together with the authentication time.

  And the control part 21 of the meeting management server 20 performs the setting process of the next authentication time (step S3-3). Specifically, the authentication management unit 21f of the control unit 21 acquires a communication state (communication speed here) in the network, and acquires a communication state correction value in this communication state using a correction table. Further, the authentication management unit 21f acquires an authentication status correction value from the correction table using the degree of coincidence of the feature amounts of the face image. Then, the authentication management unit 21f calculates a time interval by multiplying both the correction value and the initial value, and determines the next authentication time using this time interval.

  Then, the control unit 21 of the conference management server 20 executes standby processing until the next authentication time (step S3-4). Specifically, the authentication management means 21f of the control unit 21 uses a timer to wait for the next authentication time. Then, when the authentication time has come, the image verification process (step S3-1) is re-executed.

(Image tracking process)
Next, an image tracking process is executed using FIG.
Here, the control unit 21 of the conference management server 20 executes location detection processing by image tracking (step S4-1). Specifically, the participant image tracking unit 21i of the control unit 21 performs image analysis on the image data obtained from each TV conference terminal (10A, 10B), where the location of the participant's image authenticated by the authentication management unit 21f is acquired. Identify by technology. Thereby, even when a plurality of participants are photographed in one image, each participant can be specified.

And the control part 21 of the meeting management server 20 performs the determination process whether this face image can be tracked (step S4-2). Specifically, the participant image tracking means 21i of the control unit 21 has the face image specified by the image analysis technique within the imaging range of the image data acquired from the input unit 13 of the TV conference terminal (10A, 10B). Determine whether to do. Here, when the face image can be followed (in the case of “YES” in step S4-2), the location detection (step S4-1) by image tracking is continued.

  On the other hand, when the face image cannot be tracked (in the case of “NO” in step S4-2), the control unit 21 of the conference management server 20 executes a determination process as to whether or not the conference holding condition is maintained ( Step S4-3). Specifically, the participant image tracking means 21i of the control unit 21 calculates the number of participants who can follow the face image. Then, the participant image tracking means 21i compares the required number recorded in the conference management data storage unit 22 with the calculated number of participants.

  Here, when the number of participants is more than the required number and the meeting holding conditions are maintained (in the case of “YES” in step S4-3), the control unit 21 of the meeting management server 20 detects the location by image tracking. (Step S4-1) is continued.

  On the other hand, when the number of participants is less than the required number and the conference holding conditions cannot be maintained (in the case of “NO” in step S4-3), the control unit 21 of the conference management server 20 executes an alarm process ( Step S4-4). Specifically, the participant image tracking means 21i of the control unit 21 notifies each TV conference terminal (10A, 10B) that the meeting holding condition cannot be satisfied. In this case, the organizer interrupts the meeting.

(Speak confirmation process)
Next, the speech confirmation process will be described with reference to FIG.
Here, first, the control unit 21 of the conference management server 20 executes a determination process as to whether or not a speech by a participant has been detected (step S5-1). Specifically, the utterance confirmation unit 21 g of the control unit 21 acquires the participant's utterances from the voice data sequentially recorded in the live data storage unit 24. When the speech of the participant is not recorded in the voice data (in the case of “NO” in step S5-1), the speech confirmation unit 21g stands by.

  On the other hand, when the speech of the participant is detected in the voice data (in the case of “YES” in step S5-1), the control unit 21 of the conference management server 20 executes a voice recognition process (step S5-2). Specifically, the speech confirmation unit 21g of the control unit 21 instructs the speech recognition unit 21h to perform speech recognition processing on the participant's speech data.

  Then, the voice recognition unit 21 h performs voice recognition processing of the voice data using the voice feature amount of the pre-registration authentication data recorded in the user data storage unit 23. In this voice recognition process, a speaker is recognized and voice data is converted into text. Then, the voice recognition unit 21h records the recognition result (text data) in the minutes data storage unit 25 in association with the time stamp and the user identifier of the speaker.

  Next, the control unit 21 of the conference management server 20 executes face image acquisition processing at the time of speaking (step S5-3). Specifically, the utterance confirmation unit 21g of the control unit 21 specifies the image data of the utterer at the same timing as the time when the utterance was detected in the actual data storage unit 24, and acquires the face image.

  Further, the control unit 21 of the conference management server 20 executes a lip sync confirmation process (step S5-4). Specifically, the speech confirmation unit 21g of the control unit 21 trims the movement of the mouth in the face image acquired at the same timing as the speech sound.

Then, the speech confirmation unit 21g performs a determination process as to whether or not the speech and the movement of the mouth are synchronized (step S5-5). Here, when the speech and the movement of the mouth are synchronized (lip sync) (in the case of “YES” in step S5-5), the control unit 21 of the conference management server 20 waits for the next speech. (Step S5-1).

  On the other hand, when the remark and the movement of the mouth are not synchronized (in the case of “NO” in step S5-5), the control unit 21 of the conference management server 20 executes an alarm process (step S5-6). Specifically, the speech confirmation unit 21g of the control unit 21 notifies each TV conference terminal (10A, 10B) that the sound and the image are not synchronized.

(Operation check process)
Next, the operation confirmation process will be described with reference to FIG. This process is executed independently at each TV conference terminal (10A, 10B). Here, audio data and image data are transmitted from the TV conference terminal 10A, which is the transmission source terminal (first conference terminal), to the TV conference terminal 10B, which is the transmission destination terminal (second conference terminal). Is assumed.

  First, the control unit 11 of the TV conference terminal 10A executes an original sound / image acquisition process in the transmission source terminal (step S6-1). Specifically, the voice / image acquisition unit 11 a of the control unit 11 acquires voice data and image data as the first live data via the input unit 13.

  Next, the control unit 11 of the TV conference terminal 10A executes an original sound / image recording process (step S6-2). Specifically, the recording unit 11 b of the control unit 11 temporarily records the audio data and image data acquired by the audio / image acquisition unit 11 a in the audio / image data storage unit 15.

  Next, the control unit 11 of the TV conference terminal 10A executes a process of transferring the original sound / image to the transmission destination terminal (step S6-3). Specifically, the audio / image transfer unit 11c of the control unit 11 transmits audio data and image data to the TV conference terminal 10B via the network.

  Then, the TV conference terminal 10 </ b> B outputs the acquired audio data and image data via the output unit 14. In this case, the input unit 13 of the TV conference terminal 10 </ b> B acquires the image and sound output from the output unit 14. Then, the control unit 11 of the TV conference terminal 10B transmits the audio data and the image data acquired by the input unit 13 of the TV conference terminal 10B to the TV conference terminal 10A without performing echo cancellation processing.

  And the control part 11 of 10 A of video conference terminals performs the acquisition process of the audio | voice and image from a transmission destination terminal (step S6-4). Specifically, the audio / image acquisition unit 11d of the control unit 11 acquires audio data and image data as second live data from the TV conference terminal 10B via the network.

  Next, the control unit 11 of the TV conference terminal 10A executes an echo sound / image extraction process (step S6-5). Specifically, the echo extraction means 11e of the control unit 11 uses the audio / image data storage unit 15 to transmit the original audio data transmitted from the TV conference terminal 10A in the audio data and image data acquired from the TV conference terminal 10B. And the signal component of the echo sound / image corresponding to the image data. The echo extraction unit 11e supplies the extracted echo signal to the alarm output unit 11f.

Next, the control unit 11 of the TV conference terminal 10A performs a determination process as to whether or not the level of the echo signal is equal to or higher than a reference value (step S6-6). Specifically, the alarm output unit 11f of the control unit 11 calculates the echo signal level acquired from the echo extraction unit 11e. Further, the alarm output means 11 f uses the audio / image data storage unit 15 to calculate the signal level of the original audio / image. Then, the original signal level is compared with the echo signal level. Here, the alarm output means 11f calculates a comparison value obtained by dividing the echo signal level by the original signal level.

If this comparison value is greater than or equal to the reference value (“YES” in step S6-6), the control unit 11 of the TV conference terminal 10A continues the operation confirmation process.
On the other hand, when the comparison value does not reach the reference value (in the case of “NO” in step S6-6), the control unit 11 of the TV conference terminal 10A executes an alarm output process (step S6-7). . Specifically, the alarm output means 11f of the control unit 11 appropriately sends audio data and image data to the destination terminal (here, the TV conference terminal 10B) with respect to each TV conference terminal (10A, 10B). Notify that you have not. The above processing is executed in the same manner in the TV conference terminal 10B.

According to the remote conference management system of this embodiment, the following effects can be obtained.
In the present embodiment, the control unit 21 of the conference management server 20 acquires the participant image acquisition process (step S1-3), the participant authentication information acquisition process (step S1-4), and the authentication process (step S1-). 5) is executed. And the determination process of whether meeting meeting conditions are satisfied is performed (step S1-6). If the number of authenticated persons reaches the required number (in the case of “YES” in step S1-6), the control unit 21 of the conference management server 20 executes an action permission process (step S1-9). ). Thereby, even when the participant is at a remote place away from the meeting place, the meeting holding conditions can be confirmed and the meeting can be performed strictly.

  -In this embodiment, the control part 21 of the meeting management server 20 performs an image verification process (step S3-1). Here, motion detection according to the degree of coincidence of feature quantities of face images and plane detection by defocusing are performed. When an abnormality such as a feature amount change state or planar image shooting is detected, the authentication management unit 21f outputs an alarm to the TV conference terminal (10A, 10B) on the other side. Thereby, participation in the meeting in an unauthorized form can be suppressed.

  -In this embodiment, the control part 21 of the meeting management server 20 performs the setting process of the next authentication time using the communication state in a network, and the matching degree of the feature-value of a face image (step S3-3). Thereby, according to the situation, the meeting can be strictly performed while reducing the load of the authentication process. That is, for example, when the communication state is bad or the degree of coincidence of the feature amounts of the face images is bad, it is possible to increase the frequency of the authentication process and easily detect fraud. Furthermore, there is a possibility of fraud using a photo or the like when the feature amounts of the face images are completely matched. However, the fraud can be easily detected by increasing the frequency of authentication processing.

  -In this embodiment, the control part 21 of the meeting management server 20 performs the location detection process by image tracking (step S4-1). When the face image cannot be followed (in the case of “NO” in step S4-2), the control unit 21 performs a determination process as to whether or not the meeting holding condition is maintained (step S4-3). . When the number of participants is less than the required number and the meeting holding conditions cannot be maintained (in the case of “NO” in step S4-3), the control unit 21 executes an alarm process (step S4-4). As a result, it is possible to continuously monitor the participation status of the participant by image analysis, taking over the result of the personal authentication performed intermittently. And when there is a problem in the participation situation, the conference can be interrupted, so that the conference can be managed strictly.

-In this embodiment, when a participant's speech is detected in audio | voice data (step S5)
In the case of “YES” in −1), the control unit 21 of the conference management server 20 executes a voice recognition process (step S5-2). Thereby, the burden of making minutes can be reduced. Furthermore, the control unit 21 of the conference management server 20 acquires a face image at the time of speaking (step S5-3), and executes a lip synchronization confirmation process (step S5-4). As a result, it is possible to reliably record a utterance using sound and an image. If there is a problem in the synchronization status between the sound and the image, the conference can be interrupted, so that the conference can be managed strictly.

  In this embodiment, the control unit 11 of the TV conference terminal 10A executes the original voice / image recording process (step S6-2) and the original voice / image transfer process to the destination terminal (step S6-3). To do. And the control part 11 of 10 A of video conference terminals performs the extraction process of an echo sound and image, when the audio | voice and image from a transmission destination terminal are acquired (step S6-4) (step S6-5). Here, when the comparison value obtained by dividing the echo signal level by the original signal level does not reach the reference value (in the case of “NO” in step S6-6), the control unit 11 of the TV conference terminal 10A performs alarm processing. Is executed (step S6-7). As a result, it is possible to confirm the communication status of the network and the operating status of the counterpart terminal using the echo signal, and to reliably supply voice and images to the counterpart side.

Moreover, you may change the said embodiment as follows.
In the above embodiment, the participant confirmation process is executed when a meeting start request is acquired. The timing of execution of the participant confirmation process is not limited to this, and it can be configured to be executed at the voting stage.

  In the above embodiment, data related to the necessary number for establishing a conference is recorded as a conference holding condition. Here, it is possible to set an essential participant as a meeting holding condition. In this case, data relating to the user identifier of the essential participant is recorded in the conference management data storage unit 22 as a conference holding condition. In step S1-6 or step S4-3, when confirming the meeting holding conditions, it is confirmed whether or not the participant has been authenticated.

  In the above embodiment, the operation check process is executed independently at each TV conference terminal (10A, 10B). Instead of this, the control unit 21 of the conference management server 20 may be configured to execute an operation confirmation process. In this case, the audio / image data storage unit 15 is provided in the conference management server 20. Then, the control unit 21 controls the sound / image acquisition means 11a, the recording means 11b, the sound / image transfer means 11c, the sound / image acquisition means 11d, the echo extraction means 11e, the alarm output means 11f, the echo cancellation means 11g, and the recording means 11h. And function as the sound / image output means 11i. In this case, when there is no failure in the communication status of the network, the conference management server 20 can check the operation status of each TV conference terminal (10A, 10B). Therefore, it is possible to reduce the processing load of each TV conference terminal (10A, 10B).

  ○ In the above embodiment, personal authentication is performed using face image authentication, but the personal authentication method is not limited to this, and other biometric authentication technologies such as fingerprint authentication and vein authentication can also be used. It is. In this case, a user authentication image (biometrics information) reading device is connected to each TV conference terminal and transmitted to the conference management server 20 via the network. In the conference management server 20, the user authentication process is executed using each user authentication image instead of the face image.

In the above embodiment, in the operation confirmation process, the voice data and image data as the first live data and the voice data and image data as the second live data are used. However, only one of the voice data and the image data is used. It is also possible to use.

The system schematic of embodiment of this invention. It is explanatory drawing of the data recorded on each data memory | storage part, Comprising: (a) is a meeting management data memory | storage part, (b) is a user data memory | storage part. Explanatory drawing of the functional block of a video conference terminal. Explanatory drawing of the process sequence of this embodiment. Explanatory drawing of the process sequence of this embodiment. Explanatory drawing of the process sequence of this embodiment. Explanatory drawing of the process sequence of this embodiment. Explanatory drawing of the process sequence of this embodiment.

Explanation of symbols

  10A, 10B ... TV conference terminal, 11 ... control unit, 11a ... sound / image acquisition means, 11b ... recording means, 11c ... sound / image transfer means, 11d ... sound / image acquisition means, 11e ... echo extraction means, 11f ... Alarm output means, 11g ... echo canceling means, 11h ... recording means, 11i ... sound / image output means, 12 ... operation part, 13 ... input part, 14 ... output part, 20 ... conference management server, 21 ... control part, 21a ... Live data acquisition means, 21b ... Live data recording means, 21c ... Hash processing means, 21d ... Conference progress management means, 21e ... Personal authentication means, 21f ... Authentication management means, 21g ... Speech confirmation means, 21h ... Voice recognition means, 21i ... Participant image tracking means, 22 ... Conference management data storage unit, 23 ... User data storage unit, 24 ... Actual data storage unit, 25 ... Minutes data recording Part.

Claims (7)

And operation confirming control means for performing an operation check of the far septum conference, and the reference value data storage means for storing a reference value of the echo signal, the transmission data storage unit, a first conference terminal having an input unit,
A remote conference management system configured to be connected to the first conference terminal via a network and configured to include a second conference terminal including an input unit and an output unit ;
The operation confirmation control means is
Recording means for recording the first commentary data acquired through the input portion of the first conference terminal to the transmitting data storage means,
Transmitting means for transmitting the first Commentary data to the second conference terminal,
The output of the first live data at the output unit of the second conference terminal is input at the input unit of the second conference terminal, and the second live data transmitted from the second conference terminal is used as the network. Receiving means for receiving via
An echo extraction means for extracting an echo signal of the first live data recorded in the transmission data storage means from the second live data;
A remote conference management system comprising alarm output means for outputting an alarm when the magnitude of the echo signal is equal to or less than a reference value stored in the reference value data storage means . The first conference terminal further includes an output unit,
The operation confirmation control means, before Symbol to cancel the echo signal of the first Commentary data from the second Commentary data, and further comprising a means for outputting the output of the first conference terminal according Item 4. The remote conference management system according to Item 1 . The first conference terminal further comprises a live data storage unit for recording live data of a remote conference,
3. The remote conference management system according to claim 1, wherein the operation confirmation control unit further includes a unit that records the second live state data from which the echo signal is deleted in the live state data storage unit. The remote conference management system according to any one of claims 1 to 3 , wherein the first and second live data include voice data. The remote conference management system according to any one of claims 1 to 4 , wherein the first and second live data include image data. And operation confirming control means for performing an operation check of the far septum conference, and the reference value data storage means for storing a reference value of the echo signal, the transmission data storage unit, a first conference terminal having an input unit,
In order to manage a remote conference using a remote conference management system that is connected to the first conference terminal via a network and includes a second conference terminal having an input unit and an output unit. The method of
The operation confirmation control means is
A recording step of recording the first commentary data acquired through the input portion of the first conference terminal to the transmitting data storage means,
A transmission step of transmitting the first Commentary data to the second conference terminal,
The output of the first live data at the output unit of the second conference terminal is input at the input unit of the second conference terminal, and the second live data transmitted from the second conference terminal is used as the network. A receiving stage for receiving via
An echo extraction step of extracting an echo signal of the first live data recorded in the transmission data storage means from the second live data;
A remote conference management method, comprising: executing an alarm output step of outputting an alarm when the magnitude of the echo signal is equal to or smaller than a reference value stored in the reference value data storage means . And operation confirming control means for performing an operation check of the far septum conference, and the reference value data storage means for storing a reference value of the echo signal, the transmission data storage unit, a first conference terminal having an input unit,
Operation confirmation at the time of a remote conference is performed using a remote conference management system that is connected to the first conference terminal via a network and is configured by a second conference terminal having an input unit and an output unit. A program for performing,
The operation check control means;
Recording means for recording the first commentary data acquired through the input portion of the first conference terminal to the transmitting data storage means,
Transmitting means for transmitting the first Commentary data to the second conference terminal,
The output of the first live data at the output unit of the second conference terminal is input at the input unit of the second conference terminal, and the second live data transmitted from the second conference terminal is used as the network. Receiving means for receiving via
An echo extraction means for extracting an echo signal of the first live data recorded in the transmission data storage means from the second live data;
A teleconference operation confirmation program that causes an alarm output unit to output an alarm when the magnitude of the echo signal is equal to or smaller than a reference value stored in the reference value data storage unit .

Images