JP2019117997A - Web conference system, control method of web conference system, and program - Google Patents

Abstract

To provide a web conference system capable of improving the visibility of a lip part of a speaking participant.SOLUTION: A web conference system capable of displaying a plurality of photographed images photographed by client terminals connected to a plurality of bases includes an identifying means for identifying a client terminal in which a speaking person is present, a detecting means for detecting a lip part of a speaking person from a captured image captured by the client terminal identified by the identifying means, and a display control means for displaying a message indicating that detection is not performed for the client terminal identified by the identifying means when the lip part of the speaking person is not detected by the detecting means.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a web conference system using a client terminal connected to a plurality of bases, a control method of the web conference system, and a program.

  2. Description of the Related Art Conventionally, there is known a web conference system that performs virtual conferences using a plurality of terminals connected via a network. Conference participants in the web conference system can connect to terminals located at different bases and can freely speak. On the other hand, when there are a large number of bases participating in the conference, there may be cases where the speaking base is unknown.

  Patent Document 1 discloses a video conference system that identifies a participant who is about to start speaking by capturing a movement of a lip portion and detecting a speech precursor action.

Unexamined-Japanese-Patent No. 2004-118314

  By the way, when a person who is hard to hear the noise participates in the web conference system, he / she sees the movement of the lips of the speakers at different bases and understands the contents. In this case, it is difficult for the deaf person or the like to communicate when the lips of the speaker are displayed smaller than the screen or when they are not displayed on the screen. However, although the technique described in Patent Document 1 can detect the start of speech, no method has been considered for detecting a motion during speech.

  Then, this invention is made in view of said subject, and an object of this invention is to provide the web meeting system which can improve the visibility of the lip part of the participant in speech.

  In order to achieve the above object, the present invention is a web conference system capable of displaying a plurality of photographed images photographed by client terminals connected to a plurality of bases, and a person speaking is present Specification means for specifying a client terminal, detection means for detecting a lip portion of a person speaking from a photographed image captured by the client terminal specified by the specifying means, Lip portion of a person speaking by the detection means And a display control means for displaying an indication that the detection has not been made to the client terminal specified by the specifying means, when the above is not detected.

  According to the present invention, it is possible to provide a web conference system capable of improving the visibility of the lip portion of a participant who is speaking.

It is a figure which shows an example of the system configuration | structure figure of the web meeting system concerning embodiment of this invention. It is a block diagram showing an example of hardware constitutions of an information processor in a web meeting system concerning an embodiment of the present invention. It is a functional block diagram which shows an example of a functional structure in the web meeting system which concerns on embodiment of this invention. It is a flowchart which shows an example of the control processing in the web meeting system which concerns on embodiment of this invention. It is a figure which shows an example of the database in the web meeting system which concerns on embodiment of this invention. It is a figure which shows an example of the display screen of the client terminal in the web conference system which concerns on embodiment of this invention. It is a figure which shows an example of the display screen of the client terminal in the web conference system which concerns on embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram showing an example of a system configuration diagram of a web conference system according to the present embodiment.

  In the web conference system of the present invention, for example, a web conference server (server apparatus) 200 that makes a web conference system available between client terminals 100 (300, 310) applicable to personal computers is connected via the network 110. It is done. The web conferencing system is also referred to as a web conferencing system and a video conferencing system. The client terminals 100 (300, 310) function as client terminals of the web conference server (server apparatus) 200. The web conference system can display an image (still image or moving image) captured by each client terminal connected to a plurality of bases on each of the plurality of client terminals. In addition, the web conference system can output voices emitted by each client terminal connected to a plurality of bases to each of the plurality of client terminals. The client terminal 100 is an aspect of an information processing apparatus represented by a PC or the like.

  Each device may be connected to the network 110 by a wired network, or may be connected by a wireless network or wireless communication.

  The web conference server 200 is a server for realizing a web conference using a web browser. The client terminal 100 can obtain a web conference screen (interface) and conduct a web conference by accessing the web conference server 200. The web conference server 200 includes a control circuit such as a CPU and a storage area such as a RAM and a ROM. Also, the web conference server 200 includes a communication circuit for communicating with each client terminal 100.

  In this embodiment, a tablet PC is described as an example of the client terminal 100 or 300, but a notebook PC such as the information processing apparatus 310 or a desktop PC (not shown) may be used. The information processing apparatus can display an image of content distributed from the web conference server in a predetermined display area.

  Content to be distributed (video, audio, volume, etc.) is an image or audio acquired by the camera 101 or the microphone 102, and is transmitted from the information processing apparatus to the web conference server at predetermined intervals. The web conference server 200 storing these pieces of information is distributed to the connected information processing apparatus.

  In these client terminals 100, a web browser for accessing the web conference server 200 and a dedicated module are installed. Note that this dedicated module is, for example, an ActiveX component downloaded from the web conference server 200 via a web browser.

  In addition, when transmitting a moving image of oneself in a web conference, the camera 101 such as a web camera which is a video device is connected. In the case of transmitting voice, the microphone 102 is connected, and in order to view the voice of the other party, the speaker 103 is connected. The microphone 102 and the speaker 103 are audio devices. For example, in the present embodiment, it is possible to determine a base to which attention should be paid by using a change in sound collected by the microphone 102.

  In addition, some devices are provided in the same housing hardware, and for example, there are devices provided with a camera function and a microphone function, a microphone function and a speaker function, and all functions.

  In addition, the web conference system can perform screen sharing in which the screen of the client terminal 100 is displayed on the screen of another information processing apparatus (300 or 310). The screen to be shared can be designated by the client terminal 100, and may be the screen of the designated application or the entire desktop.

  The present web conference system can set a plurality of virtual conference rooms on the web called a room, and can set a plurality of rooms to perform a plurality of conferences.

  FIG. 2 is a block diagram showing an example of the hardware configuration of the information processing apparatus according to the embodiment of the present invention.

  As shown in FIG. 2, in the client terminal 100, a central processing unit (CPU) 201, a read only memory (ROM) 202, a random access memory (RAM) 203, a storage device 204, an input controller 205, and a system bus 215. A voice input / output controller 206, a video controller 207, a memory controller 208, and a communication I / F controller 209 are connected.

  The CPU 201 centrally controls the devices and controllers connected to the system bus 215.

  The ROM 202 or the storage device 204 is a control program executed by the CPU 201, such as a BIOS (Basic Input / Output System) or an OS (Operating System), a computer readable and executable program for realizing the information processing method, and various necessary items. It holds data (including data tables).

  The RAM 203 functions as a main memory, a work area, and the like of the CPU 201. The CPU 201 loads programs and the like necessary for execution of processing from the ROM 202 or the storage device 204 to the RAM 203, and implements various operations by executing the loaded programs.

  The input controller 205 controls input from an input device 210 such as a keyboard / touch panel / camera. The input device is not limited to this, and may be a touch panel capable of detecting a position touched with a plurality of fingers, such as a mouse or a multi touch screen.

  The audio input / output controller 206 controls input / output from an audio input / output device 211 such as a microphone / speaker. It is possible to recognize speech input from the microphone. It is also possible to connect an earphone to the speaker.

  The video controller 207 controls display on an external output device such as the display 212. The display shall also include the display of a notebook computer integrated with the main unit. The external output device is not limited to the display, and may be, for example, a projector. Moreover, it becomes possible to receive the input from a keyboard / touch panel about the apparatus which can be received by the above-mentioned touch operation.

  The video controller 207 can control a video memory (VRAM) for display control, and can use a part of the RAM 203 as a video memory area, or can separately provide a dedicated video memory. It is possible.

  In the present invention, the first video memory area used for display when the user normally operates the information processing apparatus and the display content superimposed on the display content of the first video memory area when the predetermined screen is displayed And a second video memory area used for The video memory area is not limited to two, and can be plural as long as the resource of the information processing apparatus permits.

  The memory controller 208 controls access to the external memory 213. The external memory is connected via an adapter to an external storage device (hard disk) that stores a boot program, various applications, font data, user files, editing files, various data, etc., a flexible disk (FD), or a PCMCIA card slot. Compact flash (registered trademark) memory etc. can be used.

  The communication I / F controller 209 connects and communicates with an external device via the network 214, and executes communication control processing in the network. For example, communications using TCP / IP, telephone lines such as ISDN, and 3G lines of mobile phones are possible.

  The storage device 204 is a medium for permanently storing information, and the form thereof is not limited to a storage device such as a hard disk. For example, a medium such as a solid state drive (SSD) may be used.

  Moreover, it can also be used as a temporary memory area at the time of various processes performed by the communication terminal in the present embodiment.

  FIG. 3 is a functional configuration diagram showing an example of a functional configuration in the web conference system according to the present embodiment. Each function is a function executed by a CPU or the like included in the web conference server 200.

  The web conference server 200 of the web conference system includes a speaker identification unit 311, a captured image acquisition unit 312, a lip part detection unit 313, a display control unit 314, a conference mode setting unit 315, and a storage unit 316.

  The speaker specifying unit 311 specifies a client terminal in which a person who is speaking is present. Specifically, when voice is acquired from a client terminal participating in a conference, the speaker identifying unit 311 can identify the client terminal in which a person who is speaking is present. For example, when detecting that a person using the client terminal has made a sound, the speaker specifying unit 311 can specify that the client terminal is a client terminal in which a person who is speaking is present. The method for specifying the client terminal by the speaker specifying unit 311 is not limited to these. For example, the speaker identifying unit 311 may identify from the action of the person appearing in the captured image. In addition, the speaker identifying unit 311 may receive a signal other than voice indicating that the user is speaking, prior to speaking from the client terminal, and may specify the information based on the information.

  The captured image acquisition unit 312 acquires a captured image captured by a client terminal participating in a web conference. The captured image acquisition unit 312 acquires a moving image that is a captured image captured continuously.

  The lip part detection unit 313 detects the lip part of the person speaking from the captured image captured by the client terminal identified by the speaker identification unit 311. The web conference server 200 always acquires a moving image by the captured image acquisition unit 312 at the web conference. Here, the lip part detection unit 313 detects the face part of the person from the captured image of each frame of the moving image acquired from the client terminal identified as the person who has started speaking. Then, the lip part detection unit 313 detects a lip part from the captured image in which the face part is detected. Here, the lip portion includes a lip and its peripheral portion, and indicates a portion which moves when a person speaks. It is preferable that the lip portion includes a range that allows the user to read the statement by observing the movement of the mouth. In addition, it is preferable that the area of the lip is in the range from the nose to the chin of the face and between the cheeks. The lip part detection unit 313 specifies the lip part by a known image processing technique, pattern recognition process, or the like.

  The display control unit 314 displays the photographed image acquired from each client terminal on the client terminal participating in the conference. The display control unit 314 causes the client terminal specified by the speaker specification unit 311 to display a warning when the lip unit of the person speaking is not detected by the lip detection unit 313. Here, the warning display may be a display indicating that the lip portion detection unit 313 has not detected the lip portion. The display control unit 314 can alert the speaker by performing the display. The display control unit 314 does not perform the warning display when the lip part detection unit 313 detects the lip part of the person speaking. In addition, when the lip part of the person who is speaking is detected by the lip part detection part 313, the display control part 314 displays an enlarged image (enlarged image) of an enlarged lip part of the speaker displayed on the client terminal. Display on the client terminal of Furthermore, the display control unit 314 may display the enlarged image only to the client terminal for which the lip priority mode is set by the conference mode setting unit 315 described later. Note that, in this case, the display control unit 314 may perform control to display a normal captured image instead of the enlarged image for the client terminal in which the lip priority mode is not set.

  The conference mode setting unit 315 sets the conference mode of each client terminal. The participants of each meeting can select the meeting mode in advance from the participation screen (the meeting reservation screen 600 of FIG. 6) on the client terminal. The conference mode setting unit 315 stores the acquired setting information in the storage unit 316 of the web conference server as a database (conference mode database 500 in FIG. 5). Here, the conference mode indicates a display mode for each client terminal in the conference. The lip priority mode is provided for the user who determines the content of the other participant's speech based on the movement of the lip portion. The lip priority mode allows the deaf person to attend the meeting in real time. The normal mode is a mode in which the detection process of the lip portion and the display process of the enlarged image described above are not performed.

  The storage unit 316 is a functional unit that stores each database shown in FIG. 5 and manages these.

  In the present embodiment, the web conference server is described as having the functions shown in this functional configuration diagram, but some functions may be executed by each client terminal, or a plurality of servers may cooperate. You may make it function.

  Here, an example of a database in the web conference system according to the present embodiment will be described with reference to FIG.

  The conference mode database 500 is information in which a room ID 501, a user ID 502, and a conference mode 503 are associated. The room ID 501 is information for identifying a conference held in the web conference system. The user ID 502 is information for identifying a participant of the web conference system. The conference mode 503 is information for identifying a conference mode set by the user. If FIG. 5 is demonstrated to an example, room ID 501 will be "100,000", and user ID 502 will show that each user of "100000" and "100001" participates in the same web meeting. The user with the user ID 502 “100000” participates in the meeting in the “lip priority mode”, and the user with the user ID 502 “100001” participates in the meeting in the “normal mode”.

  The meeting management database 510 is information in which a room ID 511, a room name 512 which is the name of a meeting, and a use time 513 which is the use time thereof are associated with each other.

  The user management database 520 is information in which a user ID 521, a user name 522 indicating the name of a user, and a group name 523 to which the user belongs are associated.

  Next, control processing in the web conference system of the present embodiment will be described using FIG. FIG. 4 is a flowchart showing an example of control processing in the web conference system according to the embodiment of the present invention.

  In S401, the web conference server 200 receives conference participant information. Here, the participant specifically indicates the client terminal 100 operated by each user. The participant information is a request for participation of the user registered in the user management database 520. Furthermore, the participant information is information indicating whether to participate in the conference mode, which is the lip priority mode or the normal mode. In order to receive the participant information, the web conference server 200 receives the information input by the user from the conference reservation screen 600 (FIG. 6) displayed on the client terminal. Here, the conference reservation screen 600 will be described with reference to FIG. The conference reservation screen 600 is a screen displayed by the web conference server 200 on each client terminal. The conference reservation screen 600 includes a join button (601, 602) and a lip priority mode selection icon (603, 604). The join button (601, 602) is a button set for each conference, and when the selection is accepted, the client terminal requests the web conference server to participate. The lip priority mode selection icons (603, 604) are icons for selecting whether to participate in the lip priority mode. The lip priority mode selection icon 603 is a case where participation is selected in the lip priority mode, and the lip priority mode selection icon 604 shows a case where participation is not in the lip priority mode but in the normal mode.

  In S402, the web conference server 200 stores the conference mode of each participant. Specifically, the web conference server 200 stores in the conference mode database 500 whether each participant is in the lip priority mode or in the normal mode.

  In S403, the web conference server 200 transmits the conference screen to the client terminal of the participant. Here, a screen when a conference is performed will be described using the conference screen 610 of FIG. The conference screen 610 in FIG. 6 is a screen displayed on the client terminal 100 that has entered the conference room set as the web conference room (virtual conference room). In the present embodiment, a total of five users participate in the web conference at four sites (User 001 to 004) besides themselves. The conference screen 610 can further display its own captured image. The web conference server 200 displays an image captured by the camera of each client terminal in the image display area 613. The screen layout of the conference screen 610 in FIG. 6 is a layout in which four image display areas 613 are arranged. The participant name display unit 612 displays the name of the transmission source of the image displayed in the image display area 613. For example, the user name of the user who has logged in to the web conference server 200 on the client terminal 100 that is the transmission source of the image is displayed. In the button display area 611, a “document sharing” button for starting sharing of the designated image, a “exit” button 614 for exiting the meeting (exiting from the meeting room), and the like are displayed.

  In S404, the web conference server 200 receives the photographed image and the sound from each client terminal.

  In S405, the web conference server 200 determines whether there is a participant in the lip priority mode. When there is no participant in the lip priority mode, the web conference server 200 executes the processing of S406. On the other hand, when there is a participant in the lip priority mode, the web conference server 200 executes the process of S407.

  In step S406, the web conference server 200 transmits each captured image and sound received in step S404 to the client terminal of the participant. The web conference server 200 repeats the process while the conference continues.

  In step S407, the web conference server 200 identifies a client terminal in which the speaker is present.

  In S408, the web conference server 200 detects the lip portion of the person speaking from the photographed image transmitted from the client terminal specified in S407. If the web conference server 200 can not detect the lip portion, the web conference server 200 executes the processing of S409. On the other hand, the web conference server 200 executes the processing of S411 when the lip portion can be detected.

  In S409, the web conference server 200 transmits a warning display to the client terminal of the speaker. Here, warning display will be described using the warning display screen 700 of FIG. 7. The warning display is an aspect of a notification indicating that the lip portion is not detected for the client terminal identified in S407. The warning display screen 700 of FIG. 7 is a screen displayed with a warning display superimposed on the conference screen 610. The warning display screen 700 is displayed including at least the warning display 701. The warning display 701 can cause the client terminal 100 to recognize that the user's moving image does not show the mouth. Therefore, the warning display 701 can urge the user of the client terminal to adjust its own position and the position of the client terminal 100.

  In S410, the web conference server 200 determines again whether the lip portion is detected. That is, the web conference server 200 repeats the process of S409 until the lip portion is detected. On the other hand, the web conference server 200 executes the processing of S411 when the lip portion is detected.

  In step S411, the web conference server 200 transmits an enlarged image in which the lip portion is enlarged to the client terminal. Here, a magnified image will be described using the magnified display screen 710 of FIG. 7. The enlarged display screen 710 is one in which the enlarged image 711 is displayed on the conference screen 610 of FIG. Here, User 001 is a photographed image of the participant identified in S407. The web conference server 200 transmits the enlarged image only to the client terminal in the lip priority mode. The web conference server 200 transmits the sound together with the enlarged image. The web conference server 200 may control so as not to transmit the voice to the client terminal for which the lip priority mode is set. This is because there is a high possibility that the client terminal in which the lip priority mode is set is not speech but judging movement of the lip portion to determine the content of the speech.

  In step S412, the web conference server 200 transmits the received captured image and sound to other client terminals participating in the conference. The web conference server 200 transmits the enlarged display screen 710 of FIG. 7 and the voice to the client terminal 100.

  In the present embodiment, although the virtual meeting room used in the web meeting system is described as a web meeting room, a virtual meeting room, or simply a meeting room, it is the same.

  As described above, according to the above-described embodiment, the web conference system can display a plurality of captured images captured by client terminals connected to a plurality of bases. The web conference system of the present embodiment identifies a client terminal in which a person speaking is present, and detects a lip portion of the person speaking from the photographed image captured by the identified client terminal. Then, when the lip part of the person speaking is not detected, the web conference system displays a message indicating that the detection is not performed on the specified client terminal. This makes it possible to improve the visibility of the lip of the participant who is speaking. In other words, the web conference system can prompt the speaker to speak the information in which the lip portion is reflected by performing the display. As a result, the participant who is speaking will speak in such a way as to reflect his / her lip. Then, the web conference system can provide a moving image in which the lip is reflected, even when the participants in the conference have a person with hearing loss.

  Further, according to the above-described embodiment, the web conference system displays an enlarged image in which the lip portion of the speaker is enlarged, when the lip portion of the person speaking is detected. This enables the web conferencing system to further improve the visibility of the lips of the participant in question.

  Furthermore, according to the above-described embodiment, the web conferencing system can set either the lip priority mode or the conference mode in the normal mode. Then, the web conference system displays an enlarged image when there is a client terminal in which the lip priority mode is set to the participants of the conference. This enables the web conferencing system to perform appropriate display control in accordance with the conference mode. This is because the web conference system may prevent the progress of a prompt conference if a warning display or a magnified image is displayed even when there is no user who has set the lip priority mode. Therefore, the web conference system displays the enlarged image when there is a user who has set the lip priority mode.

  As described above, the recording medium recording the program for realizing the functions of the above-described embodiments is supplied to the system or apparatus, and the computer (or CPU or MPU) of the system or apparatus stores the program stored in the recording medium. It goes without saying that the object of the present invention can be achieved also by reading and executing.

  In this case, the program itself read out from the recording medium realizes the novel function of the present invention, and the recording medium storing the program constitutes the present invention.

  As a recording medium for supplying the program, for example, a flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, DVD-ROM, magnetic tape, non-volatile memory card, ROM, EEPROM, silicon A disk etc. can be used.

  Further, by executing the program read by the computer, not only the functions of the above-described embodiment are realized, but also an operating system (OS) or the like running on the computer is actually executed based on the instructions of the program. It goes without saying that the processing is partially or entirely performed, and the processing realizes the functions of the above-described embodiments.

  Furthermore, after the program read from the recording medium is written to the memory provided to the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function expansion board is read based on the instruction of the program code. It goes without saying that the case where the CPU or the like provided in the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiment are realized by the processing.

  Further, the present invention may be applied to a system constituted by a plurality of devices or to an apparatus comprising a single device. It goes without saying that the present invention can also be applied to the case where it is achieved by supplying a program to a system or apparatus. In this case, by reading a recording medium storing a program for achieving the present invention into the system or apparatus, the system or apparatus can receive the effects of the present invention.

  Furthermore, by downloading and reading a program for achieving the present invention from a server on a network, a database or the like by a communication program, the system or apparatus can receive the effects of the present invention.

100 information processing apparatus 101 camera 200 web conference server 311 speaker identification unit 312 captured image acquisition unit 313 lip detection unit 314 display control unit

Claims (7)

A web conference system capable of displaying a plurality of captured images captured by client terminals connected to a plurality of bases,
Specifying means for specifying a client terminal in which a person speaking is present;
Detection means for detecting a lip portion of a person speaking from the photographed image photographed by the client terminal specified by the specifying means;
Display control means for displaying a message indicating that the detection has not been made to the client terminal specified by the specifying means when the lip part of the person speaking is not detected by the detection means;
Web conferencing system characterized by having.   The display control means displays, on another client terminal, an enlarged image obtained by enlarging the lip part of the speaker displayed on the client terminal specified by the specifying means when the lip part is detected by the detection means. The web conference system according to claim 1, wherein the web conference system comprises: And conference mode setting means for setting one of the conference mode of the lip priority mode and the normal mode;
3. The web conference system according to claim 2, wherein the display control means displays the enlarged image when there is a client terminal in which the lip priority mode is set to the participant of the conference.   The web conference system according to claim 3, wherein the display control means displays the enlarged image on a client terminal in which the lip priority mode is set.   The web according to any one of claims 1 to 4, wherein the specifying means specifies the client terminal as a client terminal in which a person who is speaking is present when the sound is acquired from the client terminal. Conference system. A control method of a web conference system capable of displaying a plurality of photographed images photographed by a client terminal connected to a plurality of bases.
A specifying step of specifying a client terminal in which a person who is speaking is present;
A detection step of detecting a lip portion of a person speaking from the photographed image photographed by the client terminal specified in the specifying step;
A display control step of displaying a message indicating that the detection has not been made to the client terminal specified in the specifying step, when the lip part of the person speaking is not detected in the detecting step;
A control method of a web conference system characterized by having. The program for making a computer perform the control method of the web meeting system of Claim 6.

Images