JP2013153411A - Conference server, conference system, control method and program of conference server, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a conference system which presents the speech converted into character information, out of utterances made at the same time by using speech output and character output, on a screen plainly.SOLUTION: Speech data transmitted from a client device is converted into characters as a character output. The character output is combined with the image data of an utterer if it is displayed, otherwise the image data is generated in a character speech frame. The whole image data based on the image data combined with the character output, the image data received from the client device, and the character speech frame is synthesized as the entire image data, and transmitted to the client device.

Description

  The present invention relates to a technology for presenting the content of an utterance to a conference participant in an easily understandable manner when a plurality of utterances overlap in a conference system that connects a plurality of bases via a network.

  2. Description of the Related Art Conventionally, a conference system that performs a conference using a plurality of terminal devices connected via a network is known. A conference system allows users in remote locations to enter a single “meeting room” and send and receive images and audio data of participants and hold a meeting without wasting travel time Can do.

  However, in the conferencing system, it is harder to understand who is speaking, compared to the case of actually gathering at one place, or listening only to the necessary comments when multiple participants speak at the same time. There is a problem that is difficult.

  In the technique described in Patent Document 1, priorities are given to the participants in advance, or the priorities of the participants with the highest priorities are given, for example, by giving high priorities to the participants who have started speaking first. Is described in real-time, and the speech of other participants is converted into text information and output.

JP 2006-229903 A

  However, the technology described in Patent Document 1 does not describe a specific display configuration for a plurality of utterances, particularly utterances converted into character information. In particular, there is a utterance converted into character information. Therefore, there is no description about a configuration that is easy to understand on the screen.

  The present invention solves the above-described problem, and provides a conference system that uses speech output and character output to present on the screen an utterance that has been converted to character information in an easy-to-understand manner. With the goal.

  It is another object of the present invention to provide a conference system that switches between voice output and character output in an easy-to-understand manner with respect to participants' statements.

The present invention is a conference server capable of communicating with a client device via a network, capable of converting voice data from the client device into characters and displaying the characters, and transmitting a voice transmitted from the client device. Voice receiving means for receiving data, voice data registration means for registering the voice data received by the voice receiving means in the voice data storage means in association with the client device, and voice data corresponding to the client device, Audio data acquisition means to be acquired from audio data storage means, image reception means for receiving image data transmitted from the client device, and image data storage means in association with image data received by the image reception means in association with the client device Image data registration means to be registered with the voice acquisition If the voice data acquired by the stage is set to be displayed on the client device, the voice data character conversion means for converting the voice data into characters and the image data corresponding to the converted characters And, if the image data is not set to be displayed on the client device, generates image data based on a character utterance frame for displaying the character output, Further, when the image data combined with the character output, the image data received from the client device, and the character utterance frame are generated, all of the image data based on the generated character utterance frame Image synthesizing means for synthesizing the image as whole image data, and an entire image for transmitting the whole image data synthesized by the image synthesizing means to the client device And over data transmission means,
It is characterized by having.

  With the conference system of the present invention, it is possible to provide a conference system that uses speech output and character output to present on the screen an utterance converted into character information in an easy-to-understand manner.

  In addition, the conference system of the present invention makes it possible to switch between voice output and character output in an easy-to-understand manner with respect to participants' statements.

It is a figure which shows an example of the system configuration | structure concerning embodiment of this invention. It is a block diagram which shows an example of the hardware constitutions of the conference server and client apparatus concerning embodiment of this invention. It is a figure which shows the component in the conference system which concerns on embodiment of this invention. It is an example of the screen which registers the conference room displayed on the client apparatus concerning embodiment of this invention. It is an example of the screen which reserves the conference room displayed on the client apparatus concerning embodiment of this invention. It is an example of the screen which participates in the meeting displayed on the client apparatus concerning embodiment of this invention. It is an example of the screen in the middle of a meeting displayed on the client apparatus concerning embodiment of this invention. It is a figure which shows an example of a structure of the completion | finish condition table concerning embodiment of this invention. It is an image figure which shows an example of the audio | voice buffer concerning embodiment of this invention. It is an image figure which shows an example of the variable which manages the authority concerning embodiment of this invention, and the state of a variable. It is an example of the flowchart which shows the process of meeting room use concerning embodiment of this invention. It is an example of the flowchart which shows the process in which the process corresponding to each client apparatus concerning embodiment of this invention acquires the authority of speech by a voice or a character. It is an example of the flowchart which shows the process which carries out the audio | voice output of the process which acquired the authority of the speech utterance based on embodiment of this invention. It is an example of the flowchart which shows the process which the process concerning embodiment of this invention outputs a character. It is an example of the flowchart which shows the process which transmits the audio | voice input into each client apparatus concerning embodiment of this invention to a conference server. It is an example of the flowchart which shows the process which transmits the speech by the sound and character output by each process concerning embodiment of this invention to each client apparatus. It is an example of the screen in the middle of a meeting displayed on the client apparatus in connection with embodiment of this invention. It is an example of the flowchart which shows the process of the display screen generation of the character utterance concerning embodiment of this invention. It is an example of the flowchart which shows the display screen generation process of the character utterance in connection with the 2nd Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram illustrating an example of a system configuration according to an embodiment of the present invention.

  The client apparatus 102 of the present invention is, for example, a personal computer, has a network communication function, and can perform data communication with the conference server 101 (conference support server, video conference server) via the network 103 (LAN, WAN, etc.). is there.

  The conference server 101 is a server for realizing a conference between the client devices 102. By accessing the conference server 101, the client device 102 can acquire a conference screen (interface) and hold a conference.

  The client device 102 is a personal computer, a tablet terminal, a mobile terminal, or the like operated by a user who uses a conference system (a conference support system or a video conference system), a client-side application for accessing the conference server 101, and a dedicated device Module (application) is installed. When the conference system is a web conference system using the web, the client-side application may be configured to use a web browser. That is, the dedicated module may be an ActiveX (registered trademark) component downloaded from the conference server 101 via a web browser, for example. In the conference system, when a moving image is distributed from the client device 102 to another client device 102 (including itself) used by a conference participant through the conference server 101, an imaging device (not shown) is used. When transmitting audio, a microphone (not shown) is connected. Also, a speaker (not shown) is connected to view the other party's voice.

  Further, the conference server 101 and the client device 102 may be configured in the same casing so that one of the client devices 102 also serves as the conference server 101.

  FIG. 2 is a block diagram illustrating a hardware configuration example of an information processing apparatus applicable to the conference server 101 and the client apparatus 102.

  As shown in FIG. 2, the conference server 101 and the client device 102 are connected via a system bus 204 to a CPU (Central Processing Unit) 201, a RAM (Random Access Memory) 203, a ROM (Read Only Memory) 202, an input controller 205, A configuration is adopted in which a video controller 206, a memory controller 207, a communication I / F controller 208, and the like are connected.

  The CPU 201 comprehensively controls each device and controller connected to the system bus 204.

  Further, the ROM 202 or the external memory 211 will be described later, which is necessary for realizing the functions executed by each server or each PC, such as BIOS (Basic Input / Output System) and OS (Operating System) which are control programs of the CPU 201. Various programs are stored. Further, information necessary for carrying out the present invention is stored. The external memory may be a database.

  The RAM 203 functions as a main memory, work area, and the like for the CPU 201. The CPU 201 implements various operations by loading a program or the like necessary for executing the processing from the ROM 202 or the external memory 211 to the RAM 203 and executing the loaded program.

  The input controller 205 controls input from a keyboard (KB) 209 or a pointing device such as a mouse (not shown).

  The video controller 206 controls display on a display device such as the display 210. The display device may be a display device such as a liquid crystal display. These are used by the administrator as needed.

  The memory controller 207 is an external storage device (hard disk (HD)), flexible disk (FD), or PCMCIA (Personal Computer) that stores a boot program, various applications, font data, user files, editing files, various data, and the like. Controls access to an external memory 211 such as a Compact Flash (registered trademark) memory connected to a Memory Card International Association (Card Memory) card slot via an adapter.

  The communication I / F controller 208 connects and communicates with an external device via the network 103, and executes communication control processing in the network. For example, communication using TCP / IP (Transmission Control Protocol / Internet Protocol) is possible.

  The CPU 201 can display on the display 210 by executing an outline font rasterization process on a display information area in the RAM 203, for example. Further, the CPU 201 enables a user instruction using a mouse cursor (not shown) on the display 210.

  Various programs to be described later for realizing the present invention are recorded in the external memory 211 and executed by the CPU 201 by being loaded into the RAM 203 as necessary. Furthermore, definition files and various information tables used when executing the program are also stored in the external memory 211, and a detailed description thereof will be described later.

  FIG. 3 is a diagram showing components in the conference system according to the embodiment of the present invention. As shown in FIG. 3, all users and rooms (conference rooms) belong to a group (301, 303).

  A system administrator 302 is an administrator of the entire conference system, and registers, changes, and deletes groups and group administrators, and performs various settings related to the conference system.

  The group manager 304 is a manager in the conference system, and registers, changes, and deletes general users and hearing users, rooms, and tags in the group.

  The general user 305 is a user who actually performs a conference within the group, and can reserve, change, and delete the conference room, and can participate in the reserved conference and participate in the free space. Note that the group manager can reserve a conference room and participate in a conference in the same manner as a general user.

  In addition, for example, there are “guest users” who can participate only in meetings invited by e-mail, and “listening users” who listen to meetings held in groups. In addition, in order to hold a conference, it is necessary to register a room, and the room manager registers the room.

  The room is a room that participates by restricting the date and time and conference participants, and the conference room 306 that needs to be reserved before the conference, and the conference room 306 does not have to be reserved regardless of the date and time or the conference participants. There are two types of “free space” 307 where a conference can be held at any time. The user can use “conference room” and “free space” depending on the content of the conference.

  FIG. 4 is an example of a screen for registering a conference room displayed on the client device according to the embodiment of the present invention.

  A method for setting a virtual conference room (referred to as “room” in the present specification) in which a conference is performed according to an embodiment of the present invention will be described. FIG. 4 is a room registration screen for setting a conference room for conducting a conference in the present embodiment. The room registration screen in this figure is displayed by logging in to the system with the account of the administrator user (administrator) of the conference system who operates the client apparatus 102 and pressing the room management area on the left side of the screen. It appears on the display.

  The room name 401 receives a name input of an arbitrary room name. The room type 402 accepts selection of whether to set as “conference room” or “free space”. In the embodiment of the present invention, the “conference room” refers to a room that needs to be reserved by determining the participants of the conference and the start time. “Free space” refers to a room in which anyone can participate in the conference without a reservation for the conference and no particular setting of participants or the like.

  The capacity 403 receives the setting of the room capacity. The setting of the hearing function 404 accepts a setting as to whether or not to allow a hearing user of the conference, separately from the user invited to the conference. A user who listens to the conference can enter the room if there is a vacant capacity, but the authority to speak is limited.

  The setting of the recording function 405 accepts a setting as to whether or not to allow recording of a conference held in the room. In a room where recording is permitted, the conference can be recorded by receiving a press of a “recording start” button during the conference.

  The profile 406 receives selection of a profile to be applied to the room. Specifically, when the right call button is pressed, a list of profiles registered on the above-described profile registration screen in FIG. 3 is displayed, and selection of a profile to be applied to the room for which detailed settings are currently being made can be accepted. it can. When the pressing of the registration button 407 is accepted, the room and the room detailed settings are registered in the management table (not shown) of the conference server 101.

  FIG. 5 is an example of a screen for reserving a conference room displayed on the client apparatus 102 according to the embodiment of the present invention.

  A conference room can be reserved for a conference. This is the “meeting room” type that needs to be determined and reserved among the types of rooms assumed in the embodiment of the present invention, such as “meeting room” or “free space”, as described above. This is a reservation process required when using a room.

  The conference system reservation screen is displayed on the display device of the client device 102 by logging in to the system with the general user account of the conference system of the present embodiment and pressing the “conference room reservation 501” area on the left side of the screen. The

  The conference room name 502 accepts selection of the room (conference room or free space) registered on the room registration screen of FIG. 4 described above, and sets the conference name in the item of the conference title 503. The usage schedule of the corresponding room is set with the usage date 504 and the usage time 505. The participant 506 calls the user information registered in the conference system of the present embodiment and selects a conference participant (a person invited to the conference). If “Limit” is selected in 506, an invitation (participation) user selection screen is displayed, and user information is searched to select a user to invite. When the participants are not limited, the setting of the participants is not particularly accepted. In addition, the case where the participant to a meeting is not limited means the meeting like the opinion exchange meeting implemented in the room of a free space format, for example.

  The guest participant 507 invitation setting is set when an invitation notification (for example, a mail address linked to user information is transmitted as a destination) is transmitted to a user who wants to participate in the conference. When inviting a guest participant, a user selection screen (not shown) is displayed, and user information is searched and a user to be invited is selected.

  The conference recording setting 508 receives a setting as to whether or not to permit recording of a conference to be reserved. In the room registration screen of FIG. 4, if the recording function is set to “not permitted” in 405, each item of 508 is grayed out and cannot be selected.

  When the pressing of the reservation button 509 is accepted, the conference reservation is completed. When the conference reservation is completed, the conference server 101 stores the conference reservation information in the storage unit of the server and updates the conference reservation data table (not shown). In addition, when the pressing of the reservation button 509 is accepted, a conference reservation mail may be transmitted to the mail addresses of the user who made the reservation and the user invited to the conference.

  FIG. 6 is an example of a screen for participating in a conference displayed on the client device 102 according to the embodiment of the present invention. Through this screen, the user of the client device 102 can participate in the reserved conference. In the present embodiment, the conference participation screen is displayed on the display by logging in to the system with the general user account of the conference system of the present embodiment and pressing the “conference participation 601” area on the left side of the screen. Is done. The conference room displayed in FIG. 6 is a list of conferences in which the logged-in user account is registered as a conference participant. In FIG. 6, two conference rooms, room A and room B, are reserved, and it is possible to participate in the conference of the selected conference room by accepting pressing of the join button 602.

  FIG. 7 is an example of a screen during a conference displayed on the client device according to the embodiment of the present invention.

  The conference screen 700 is controlled to be displayed on the client device 102 and is divided into three participant screens 701. Here, the participant screen 701a displays the image of the participant A from the client device 102a, the participant screen 701b displays the image of the participant B from the client device 102b, and the participant screen 701c displays the image of the participant C from the client device 102c. .

  At the present time, the participant A has acquired the authority as a voice speaker. The speech content (voice data acquired from the client device 102a) of the voice speaker (participant A) is controlled to be played back as voice on the client device 102 (voice data playback control unit) and displayed as character information. Not. On the other hand, the participants B and C have no authority as a voice speaker, and have acquired the authority as a character speaker. The utterance contents (voice data acquired from the client devices 102b and 102c) of the character utterers (participants B and C) are converted into characters, and the respective utterance contents are displayed as characters on the participant screens 701b and 701c.

  In the embodiment of the present invention, the voice speaker is described as an example limited to one participant (voice data from one client device 102). However, a plurality of voices such as a character speaker described later are described. Speaking authority may be prepared.

  In FIG. 7, all of the participants A to C speak, and in this case, the content of the speech is reproduced or displayed in text by either voice or text. In the case where there is a possibility that the number of displays according to is likely to increase, the number of character speakers may be limited as will be described later. That is, not only a voice speaker but also a single character speaker may be used.

  FIG. 8 is a diagram showing an example of the configuration of the end condition table according to the embodiment of the present invention. In the embodiment of the present invention, the voice utterance authority is acquired from the voice data sent from the client apparatus 102 in the order in which the utterance is started (first come first served). When speech is stopped (when there is no speech data in the speech buffer), the speech utterance authority is canceled, and when there is another speaker (when there is speech data sent from another client device 102). ) Let other speakers acquire the authority to speak.

  However, the voice utterance authority is given to other speakers just by leaving a very short interval (ie, there is a section where there is very little voice data as voice data in the voice buffer), such as breathing during speech. If it is moved, the voice speaker and the character speaker are suddenly switched, which is inappropriate. Therefore, when a voice speaker interrupts his / her speech, it is judged how long the speech is considered to have been stopped, and if the speech is considered to be canceled, the right to speak is released. To control. As a result, it is possible to obtain an effect of preventing the voice speaker and the character speaker from being switched suddenly even though it is an inappropriate position to divide the sentence as a sentence.

  Similarly, as soon as the voice utterance authority of the voice utterer is canceled, it may not be appropriate that the speech of the participant who has been a character utterance suddenly switches to voice. For example, depending on the position where the content being spoken is divided into letters and voices, the front and back (the letters before, the voice after) may be meaningless (for example, “Give materials. ”Is separated into“ materials ”and“ claws ”. In that case, other participants who are listening to the remark must switch the timing of reading the voice from reading the character appropriately and understand it. In practice, since character utterance involves a process of converting voice data stored in the voice buffer into characters, a time difference occurs after the voice data is stored in the voice buffer. For this reason, the character utterance is dropped by the time difference, or the character utterance and the subsequent voice utterance are overlapped, and the other participants have to understand both at the same time.

  Therefore, as in the case where the voice utterance is switched to the character utterance, the character utterer interrupts the utterance, and after determining appropriately how long the voice utterance can be switched to the voice utterance, Control to get permission. As a result, it is possible to prevent the sudden change from the character utterance to the voice utterance at an inappropriate timing as a position where the utterance is divided into sentences.

  The setting of the end condition will be described with reference to FIG. The condition type 801 includes a voice output end condition 802 (voice output end condition storage unit) for setting an interval (elapsed time) from when the voice speaker interrupts the speech until the voice utterance authority is canceled, and the speech is interrupted. And a character output end condition 803 (character output end condition storage unit) for setting an interval (elapsed time) from when the character speaker attempts to acquire the voice utterance authority.

  In the terminal 804, terminal information for specifying the client apparatus 102 to be set is set. Also, the common 805 (default value) used when the terminal information for the terminal 804 is not set or when the terminal information is set but the voice output end condition 802 and the character output end condition 803 are not set is set. Is possible. In the example of FIG. 8, when “0.5 seconds” is set as the voice output end condition 802 and “0.3 seconds” is set as the character output end condition 803, and no end condition is set for each terminal, This set value is used.

  Reference numeral 806 describes termination conditions corresponding to each terminal. For example, “1.0 second” is set as the audio output end condition 802 for the client apparatus 102a. Since the character output end condition 803 is described as “-” (no setting), the setting value “0.3 second” of the common 805 is used.

  The setting values in the end condition table may be operated by displaying a setting screen on the client device 102 so that, for example, a user who acts as the “chairman” can be set every time a new participant is added. Further, when the participants are limited in advance by the participant 506 in FIG. 5, an end condition may be set at that time.

  Since the end condition setting value can be set individually for each terminal, for example, the end condition for the terminal used by the “chairperson” user or the “customer” can be made longer, making it difficult to cancel the voice utterance authority. The effect of enabling the corresponding operation is obtained.

  FIG. 9 is an image diagram showing an example of an audio buffer according to the embodiment of the present invention. 900 in FIG. 9 is an audio buffer managed by the system using the memory of the conference server 101. In the example of FIG. 9, audio data of the client apparatuses 102a to 102c (participants A to C) are separated into separate audio buffers. Store in (three audio buffers). The audio data is received by the conference server 101 from the client device 102 by a method described later with reference to FIG.

  900 has three audio buffers of client devices 102a-102c (participants A-C) as described above. A state in which the participants of the client apparatuses 102a to 102c are speaking (audio data is in the audio buffer) is represented by 901 (solid line). Further, a state where no speech is made (no audio data in the audio buffer) is represented by 902 (dotted line). Reference numerals 901 and 902 represent the progress of time from left to right, and 903 (arrow) represents the current time (that is, the beginning of the buffer). That is, as time progresses, 903 (arrow) advances from left to right. For example, in the audio buffer corresponding to the client device 102a of 900a, the speech is interrupted because 903 (arrow) is at the left end of 906a and is at the boundary between 901 (solid line) and 902 (dotted line). Point to. That is, the head of the audio buffer is not the left end of the line in FIG. 9, but represents the position that advances to the right as the time 903 (arrow) moves as time progresses. Actually, the size of the audio buffer is limited, but the participant leaves the conference room by repeatedly deleting the audio data that has been used and storing the audio data newly received from the client device. Used up to

  In 900a, the client apparatus 102a initially has the voice utterance authority, but there is no voice data at the left end of 906a. Thereafter, since the audio data appears again during the audio output end condition 802 of the client apparatus 102a in the end condition table (FIG. 8) (to the right end of 906a), it is considered that the speech has not been stopped, and the client apparatus 102a Voice utterance is continuously made without releasing the voice utterance authority. The voice buffers of the client apparatuses 102b and 102c also have voice data, but the voice utterance authority is not acquired.

  In 900b, the client apparatus 102a initially has the voice utterance authority, but no voice data is left at the left end of 906a. Thereafter, the audio data does not appear again during the audio output end condition 802 of the client apparatus 102a in the end condition table (FIG. 8) (up to the right end of 906a). In addition, after the voice utterance authority of the client apparatus 102a is canceled, the client apparatuses 102b and 102c once have no voice data, and are longer than the respective character output end conditions 803 in the end condition table (FIG. 8). Since the voice data is not resumed, the voice utterance authority can be acquired. Finally, since the voice data of the client apparatus 102c appears first, the client apparatus 102c acquires the voice utterance authority. When the client apparatus 102b resumes speaking, it becomes a character utterance again.

  In 900c, similarly to 900b, after the voice utterance authority of the client apparatus 102a is canceled, both of the client apparatuses 102b and 102c once have no audio data. The voice data of the client apparatus 102b is resumed before the voice data of the client apparatus 102c, but is resumed in a short time that does not satisfy the character output termination condition 803 in the termination condition table (FIG. 8). It cannot be acquired and resumed as a character utterance. On the other hand, since the voice data of the client apparatus 102c is resumed after satisfying the character output end condition 803, the voice utterance authority is acquired.

  FIG. 10 is an image diagram showing an example of a variable for managing authority and a state of the variable according to the embodiment of the present invention. In the following description, for the sake of convenience, it is described that the operating subject is the client device 102, but actually, in the conference server 101 generated corresponding to the client device 102 as described in the flowcharts of FIG. The process is a process executed by the CPU 201 of the conference server 101.

  First, acquisition and cancellation of voice utterance authority will be described. When none of the client apparatuses 102 has acquired the voice utterance authority, “voice lock variable 1001a = 1” is set as in 1000a. Further, it is understood that “voice output terminal 1002a = not specified”, and no client apparatus 102 has acquired the voice utterance authority.

  When any of the client devices 102 tries to acquire the voice utterance authority, the voice lock variable is locked. When the lock is applied, “speech lock variable 1001b = 0” as in 1000b, and the other client apparatuses 102 cannot acquire the voice utterance authority. If another client device 102 has already been locked, an error occurs in the lock processing. Even if the lock can be applied, the voice utterance authority cannot be acquired if other terminal information is already registered in the voice output terminal 1002. Since no terminal information has been registered in 1002b, the terminal information is registered.

  1000c is a state in which the client apparatus 102a is registered as terminal information in the audio output terminal 1002c. By releasing the lock, “speech lock variable 1001c = 1” is obtained, the processing for obtaining the voice utterance authority of the client apparatus 102a is completed, and voice utterance is started.

  When the voice utterance is completed, the client apparatus 102a locks the voice lock variable 1001 again (voice lock variable 1001d = 0), deletes the “client apparatus 102a” registered in the voice output terminal 1002d, and locks it. Is released. As a result, the state becomes 1000e.

  Also, as described above, the number of client devices that can acquire authority for character utterances may be limited. Images for acquiring the character utterance authority in this case are described in 1010f to 1010h. In this figure, the two client devices 102 can acquire the character utterance authority. The detailed description is the same as the processing at the time of voice utterance, except that there are a plurality of variables for registering the character output terminal, and therefore will be omitted.

  Note that there may be one or more voice utterance authorities and character utterance authorities. This number is not fixed and may be set by the user who registered the conference room according to FIG. 4 or the user who reserved the conference room according to FIG.

  The number of blocks at 1012 in FIG. 10 (two in the example) is the limit number, which can be realized by storing the limit number in a character output authority number storage unit (not shown). In addition, a plurality of voice utterance authorities can be recognized (a plurality of blocks 1002 are prepared), and the number can be stored in a voice output authority number storage unit (not shown).

  Next, details of the process of the conference system will be described with reference to the flowcharts of FIGS. 11 to 16.

  FIG. 11 is an example of a flowchart showing a process of using a conference room according to the embodiment of the present invention. Each step in FIG. 11 is executed by the CPU 201 of the conference server 101.

  In step S1101, use of the conference room is started. The start of use of the conference room (that is, the start of this flowchart) may be started automatically at the start time set by the use date 504 and the use time 505 when the conference room is reserved in FIG. It may be started upon receiving an instruction to start using the conference from the device 102.

  In step S1102, it is checked whether a new participant (client device 102) has entered the conference room.

  If it is determined in step S1103 that there is a new participant based on the new participant check in step S1102 ("YES"), the process proceeds to step S1104. If there is no new participant (“NO”), the process proceeds to step S1105.

  In step S1104, a process for processing a new participant (client device 102) processing is generated and started. Details of the process in the process will be described with reference to FIGS.

  In step S1105, it is determined whether an exit operation from the participant (client device 102) has been accepted in any of the processes generated in step S1104. If there is a process that has accepted the exit operation (“YES”), the process proceeds to step S1106. If there is no process that has accepted the exit operation (“NO”), the process proceeds to step S1107.

  In step S1106, the process that has accepted the exit operation in step S1105 is terminated and deleted.

  In step S1107, it is determined whether the conference has ended. Specifically, this is a case where a conference end operation is received from the client device 102 by a user operation. Moreover, you may implement so that it may determine that the meeting was complete | finished when all the participants left. If it is determined that the conference has ended ("YES"), the process proceeds to step S1108. If it is determined that the process has not been completed (“NO”), the process proceeds to step S1102.

In step S1108, a conference room use end process is performed.
Above, description of the flowchart of FIG. 11 is completed.

  FIG. 12 is an example of a flowchart showing a process in which a process corresponding to each client apparatus according to the embodiment of the present invention acquires the right to speak by voice or text. The process is a process generated in the flowchart of FIG. Each step in FIG. 12 is executed by the CPU 201 of the conference server 101.

  In step S <b> 1201, it is determined based on an operation in the client device 102 whether an exit event from the conference room has been received. When the exit event is accepted (“YES”), this flowchart ends. When the process ends, the process described with reference to FIG. 11 is notified that the process has ended by accepting an exit event. If an exit event has not been received (“NO”), the process proceeds to step S1202.

  In step S1202, it is determined whether there is audio data in the audio buffer corresponding to this process. Specifically, it is determined whether or not audio data shown as a solid line portion is stored in the audio buffer described in FIG. Note that the audio data is received from the client device 102 and stored in the audio buffer (audio data storage unit) (audio data registration unit), as described with reference to the flowchart of FIG. If there is audio data (“YES”), the process proceeds to step S1203. If not (“NO”), the process proceeds to step S1201.

  In step S1203, the voice lock variable is locked (1001 in FIG. 10).

  In step S1204, it is determined whether or not the process for locking in step S1203 has succeeded. If successful (“YES”), the process advances to step S1205. If not successful (“NO”), the process proceeds to step S1214.

  In step S1205, it is determined whether or not an audio output authority can be acquired (audio output authority grant determination unit). Specifically, it is determined whether there is a vacancy in the terminal registration 1002 in FIG. 10 and registration is possible. If the voice output authority has been acquired (“YES”), the process proceeds to step S1206. If it cannot be acquired (“NO”), the process proceeds to step S1213.

  In step S1206, an audio output authority is acquired for this process (audio output authority assigning unit). Specifically, terminal information of the client apparatus 102 corresponding to this process is registered in 1002 in FIG.

  In step S1207, since the sound output authority acquisition process is completed, the lock applied to the sound output lock variable is released.

  In step S1208, since this process has acquired the voice utterance authority, a process of outputting the voice data in the voice buffer as voice is executed. Details will be described with reference to the flowchart of FIG.

  FIG. 13 is an example of a flowchart illustrating a process of outputting a voice by a process that has acquired the right to speak by voice according to the embodiment of the present invention. Each step of FIG. 13 is executed by the CPU 201 of the conference server 101.

  In step S1301, it is determined whether there is audio data at the head of the audio buffer. Specifically, in the image diagram of the audio buffer in FIG. 9, is there audio data immediately after 903 (arrow) as the position of the audio buffer corresponding to the current time (is the solid line in the figure)? Judge by whether or not. If there is audio data (“YES”), the process proceeds to step S1302. If there is no audio data (“NO”), the process proceeds to step S1304.

  In step S1302, the audio data in the audio buffer is output as audio. This is because this process has a voice utterance authority and voice data exists.

  In step S1303, “0” is substituted (initialization) to indicate that there is no audio data in the audio buffer and “audio non-output start time” corresponding to the time when the audio utterance was interrupted is not set. Furthermore, returning to step S1301, if there is new audio data in the audio buffer, the voice utterance is repeated.

  In step S1304, it is determined whether or not “voice non-output start time” is “0”. The case of “0” means a state in which the processing after step S1304 has not been executed so far and there is no audio data for the first time (a state immediately after the speech in the client apparatus 102 is interrupted). If the “voice non-output start time” is “0” (“YES”), the process proceeds to step S1305. If the “voice non-output start time” is not 0 (“NO”), the process proceeds to step S1306 (because it is not necessary to set the current time as the voice non-output start time).

  In step S1305, the current time is set as the “voice non-output start time”. That is, the time when the speech of the participant in the client apparatus 102 is interrupted and no audio data is stored in the audio buffer corresponding to this process is set as the “audio non-output start time”.

  In step S1306, it is determined whether or not “current time−speech non-output start time” is a value greater than (or may include a case of equal to) the “speech output end condition”. That is, it is determined whether or not the value is larger than the value of the audio output end condition 802 stored in FIG. If the value is greater than the value of “sound output end condition” (may be equal) (“YES”), the process proceeds to step S1307 and “end condition” (return value for the flowchart of FIG. 12) is set to “ “true” is set, “audio non-output start time” is set to “0” (initialization) in step S1308, and this flowchart ends. If not, “false” is set in the “end condition” and this flowchart is ended.

  Thus, the description of the flowchart in FIG. 13 is completed. The description returns to the flowchart of FIG. At this point, step S1208 is complete.

  In step S1209, it is determined whether the return value from the processing of the flowchart of FIG. 13 is “false” or “true” (audio output authority release determination unit). In the case of “false”, the process returns to step S1208 (that is, the process of the flowchart of FIG. 13). If “true”, the process advances to step S1210. In the case of “false”, the voice data in the voice buffer is interrupted in the processing of FIG. 13, but the interruption time is shorter than the value of the voice output end condition 802 in FIG. This is the case. Therefore, the process returns to the process of the flowchart of FIG. 13 again, and if there is new voice data in the voice buffer, voice utterance is resumed.

  On the other hand, if the process returns to the process of the flowchart of FIG. 13 but there is no new audio data in the audio buffer, the determination in step S1306 is repeated. At this time, the “current time” advances from the previous determination, so that the “end condition” eventually becomes “true” when repeated many times (step S1307). That is, in any case, the time during which speech is interrupted becomes longer than the value of the audio output end condition 802.

  In step S1210, the voice lock variable 1001 in FIG. 10 is locked to release the voice utterance authority.

  In step S1211, the terminal information corresponding to this process registered in the voice output terminal of FIG. 10 is deleted (voice output authority release unit).

  In step S1212, the lock of the sound lock variable 1001 is released.

  In step S1213, the lock of the sound lock variable 1001 is released. This step is a release when the sound output authority cannot be acquired although the lock is applied.

  In step S1214, the process executes a process of outputting the audio data in the audio buffer as characters. Details will be described with reference to the flowchart of FIG.

  FIG. 14 is an example of a flowchart showing processing for outputting characters by the process according to the embodiment of the present invention. Each step in FIG. 14 is executed by the CPU 201 of the conference server 101.

  In step S1401, it is determined whether there is audio data at the head of the audio buffer. Specifically, in the image diagram of the audio buffer in FIG. 9, is there audio data immediately after 903 (arrow) as the position of the audio buffer corresponding to the current time (is the solid line in the figure)? Judge by whether or not. If there is audio data (“YES”), the process proceeds to step S1402. If there is no audio data (“NO”), the process proceeds to step S1404.

  In step S1402, the voice data in the voice buffer is converted into characters and output as characters (voice data character conversion unit).

  In step S1403, “0” is substituted to indicate that there is no voice data in the voice buffer and “character non-output start time” corresponding to the time when the character utterance was interrupted is set (initialization). Further, returning to step S1401, if there is new voice data in the voice buffer, the character utterance is repeated.

  In step S1404, it is determined whether or not “character non-output start time” is “0”. The case of “0” means a state in which the processes after step S1404 have not been executed so far and there is no audio data for the first time (a state immediately after the speech in the client apparatus 102 is interrupted). If the “character non-output start time” is “0” (“YES”), the process proceeds to step S1405. If the “character non-output start time” is not 0 (“NO”), the process proceeds to step S1406 (since it is not necessary to set the current time as the character non-output start time).

  In step S1405, the current time is set in “character non-output start time”. That is, the time when the participant's speech in the client apparatus 102 is interrupted and no voice data is stored in the voice buffer corresponding to this process is set as the “character non-output start time”.

  In step S1406, it is determined whether or not “current time−character non-output start time” is a value larger than the value of “character output end condition” (may be equal). That is, it is determined whether or not the value is larger than the value of the character output end condition 803 stored in FIG. If the value is greater than the value of “character output end condition” (may be equal) (“YES”), the process proceeds to step S1407 and “end condition” (return value for the flowchart of FIG. 12) is set to “ "true" is set, "character non-output start time" is set to "0" (initialization) in step S1408, and this flowchart ends. If not, “false” is set in “end condition” in step S1409, and this flowchart is ended.

  Thus, the description of the flowchart in FIG. 14 is completed. The description returns to the flowchart of FIG. At this point, step S1214 has been completed.

  In step S1215, it is determined whether the return value from the process of the flowchart of FIG. 14 is “false” or “true” (character output authority release determination unit). In the case of “false”, the process returns to step S1214 (that is, the process of the flowchart of FIG. 14). If “true”, the process advances to step S1201. In the case of “false”, the voice data in the voice buffer is interrupted in the processing of FIG. 14, but the interruption time is shorter than the value of the character output end condition 803 in FIG. is there. Therefore, returning to the process of the flowchart of FIG. 14 again, if there is new voice data in the voice buffer, the character utterance is resumed.

  On the other hand, if the process returns to the process of the flowchart of FIG. 14 but there is no new audio data in the audio buffer, the determination in step S1406 is repeated. At this time, the “current time” advances from the previous determination, so that the “end condition” will eventually become “true” when repeated many times (step S1407). That is, in any case, the time during which speech is interrupted becomes longer than the value of the character output end condition 803.

In FIG. 12, authority is not acquired when a character is uttered (character output in step S1214). Here, as with the voice utterance authority, the character utterance authority may be requested (1000f to 1000h in FIG. 10). This makes it possible to limit the number of client devices 102 (participants) who perform character utterances and make it easier to understand the utterances. The flowchart is the same as the acquisition and cancellation of the voice utterance authority, and thus the description thereof is omitted. However, it is possible to implement a character output authority assignment determination unit, a character output authority assignment unit, and a character output authority release unit.
This is the end of the description of the flowchart of FIG.

  In the end condition table of FIG. 8, the voice output end condition 802 and the character output end condition 803 are set according to time (time when the voice data in the voice buffer is interrupted), and in the description of the flowcharts of FIGS. The voice output or the character output is terminated when the above-described temporal condition is satisfied.

  However, the conditions may be specified by other methods. For example, “keyword” may be set as a condition. In that case, in step S1302 in FIG. 13 and step S1402 in FIG. 14, the processing ends when the utterance matches a keyword pattern such as “End now.” Or “End. Please”. The condition may be “true” and the process may return to the flowchart of FIG.

  Further, it may be a combination of the temporal termination condition of FIG. 8, the keyword termination condition, and other termination conditions.

  FIG. 15 is an example of a flowchart illustrating processing for transmitting audio input to each client device according to the embodiment of the present invention to the conference server. Steps S1501 to S1504 in FIG. 15 are executed by the CPU 201 of the client apparatus 102, and steps S1511 to S1514 are executed by the CPU 201 of the conference server 101.

  In step S1501, the client apparatus 102 accepts input of audio data from a user participating in the conference by connecting a microphone.

  In step S1502, the client apparatus 102 transmits the audio data received from the user to the conference server 101 (audio data transmission unit).

  In step S1511, the conference server 101 receives audio data from the client device 102 (audio reception unit).

  In step S1512, the conference server 101 stores the received audio data in the audio buffer (FIG. 9). If the user does not speak and there is no audio data, silence data for the same time may be stored in the audio buffer. Alternatively, when the audio data is extracted from the audio buffer so that the time is associated with the audio data stored in the audio buffer, the application on the use side analyzes and processes the silence interval between the audio data. You may be able to do it.

  Further, steps S1503 to S1504 described later in the client apparatus 102 are executed asynchronously with steps S1501 to S1502. Similarly, steps S1513 to S1514 described later in the conference server 101 are executed asynchronously with steps S1511 to S1512.

  In step S <b> 1503, the client apparatus 102 inputs and accepts, as a moving image, the state of the user who uses the client apparatus 102 by connecting the imaging apparatus.

  In step S1504, the client apparatus 102 transmits the moving image as image data to the conference server 101.

  In step S1513, the conference server 101 receives image data from the client device 102 (image reception unit).

In step S1514, the conference server 101 stores the received image data in an image buffer (image data storage unit) (image data registration unit).
Thus, the description of the flowchart of FIG. 15 is completed.

  FIG. 16 is an example of a flowchart showing a process of transmitting speech and text utterances output in each process according to the embodiment of the present invention to each client device. Steps S1601 to S1604 in FIG. 16 are executed by the CPU 201 of the client apparatus 102, and steps S1611 to S1615 are executed by the CPU 201 of the conference server 101.

  In step S1611, the conference server 101 acquires audio data (audio acquisition unit). Specifically, the audio data output in step S1302 of FIG. 13 is acquired.

  In step S <b> 1612, the conference server 101 transmits audio data to the client device 102 (audio transmission unit).

  In step S1601, the client apparatus 102 receives audio data from the conference server 101 (audio data receiving unit).

  In step S1602, the client apparatus 102 controls output of audio data. Specifically, audio data is output from a speaker or the like connected to the client device 102.

  Further, steps S1603 to S1604 described later in the client apparatus 102 are executed asynchronously with steps S1601 to S1602. Similarly, steps S1613 to S1615 described later in the conference server 101 are executed asynchronously with steps S1611 to S1612.

  In step S1613, the conference server 101 acquires image data. Specifically, the image data of the client apparatus 102 recorded in the image buffer in step S1514 in FIG. 15 is acquired.

  In step S1614, the conference server 101 synthesizes character information with image data (image synthesizer). Specifically, in step S1402 in FIG. 14, the voice data is converted into characters and output character information is acquired and synthesized with the voice data and the image transmitted from the corresponding client device 102 (701b in FIG. 7). , 701c, etc.). Further, when there are a plurality of client apparatuses 102 and there are a plurality of synthesized images (701a to 701c in FIG. 7, where 701a is not synthesized with character information), one more whole image data is obtained. A format to be displayed on the client apparatus 102 is generated (700 in FIG. 7).

  The character information may be transmitted to the client apparatus 102 separately from the entire image data, and the character information may be transmitted to the client apparatus 102 if the client apparatus 102 can synthesize the character information and the entire image data. .

  In step S1615, the conference server 101 transmits the entire image data combined in step S1614 to the client device 102 (entire image data transmission unit).

  In step S <b> 1603, the client apparatus 102 receives the entire image data from the conference server 101.

In step S1604, the client apparatus 102 controls display of the received entire image data. Specifically, it is displayed on a display device connected to the client device 102.
This completes the description of the flowchart of FIG.

  As described above, a method has been described in which utterances are presented to the user as voice utterances and character utterances, and voice utterances and character utterances are easily switched for the user at the end of each utterance.

  Next, using FIG. 17 to FIG. 18 (and FIG. 7 again), the character to be displayed at the time of “character utterance” is displayed on the screen (client side application of the client device 102, for example, a Web browser). It will be explained whether the information is displayed in an easy-to-understand position.

  In FIG. 7, in three client apparatuses (102a to 102c), three persons (A, B, C) speak and B and C utterances are character utterances. That is, since the participant screen corresponding to the terminal that is uttering characters is displayed on the client, the characters are also displayed on the corresponding participant screen. This screen structure makes it easy to see who is speaking.

  FIG. 17 is an example of a screen during a conference displayed on the client device according to the embodiment of the present invention. On the screen 1701, only the participant screen from one terminal is displayed. In this case, unlike FIG. 7, since the participant screen of B and C does not exist, it displays on the area | regions other than the participant screen corresponding to each speaker. This area may be combined with an image of one displayed participant screen. Further, when there is no participant screen corresponding to the speaker, a frame dedicated to character utterance (character utterance frame) may be prepared and displayed therein (screen 1702). Here, the characters are illustrated so as to flow from left to right in chronological order according to the timing of utterances, but this is merely an example, and may be in the vertical direction, other directions, and orientations.

  Although not shown, all utterances may be displayed in the character utterance frame regardless of whether the participant screen corresponding to the character utterance is displayed.

  Whether there is a participant screen corresponding to the utterance is displayed in the participant screen or whether the participant screen is displayed in the text utterance frame is set in the “utterance frame display flag”. If the utterance frame display flag is “true”, characters are displayed in the character utterance frame even if there is a participant screen corresponding to the utterance. If “false”, if there is a participant screen corresponding to the utterance, Display in the participant screen.

  If “utterance frame display flag = false”, the speech may not be displayed unless the participant screen is displayed in the speech. The display state of the participant screen is determined based on whether or not the conference server 101 controls to display a frame (display frame) on the screen corresponding to the participant (actually the client device 102). Determine by part. Specifically, when the “participant utterance display flag” is “true”, the character utterance frame is used, that is, when the participant screen corresponding to the character utterance is not displayed, the character utterance is changed to the character utterance. Display in the utterance frame. In the case of “false”, when the participant screen corresponding to the character utterance is not displayed, the character utterance frame may not be used and the content of the character utterance may not be displayed on the screen.

The utterance frame display flag and the participant utterance display flag are stored in a “spoken character display setting storage unit” (not shown).
This completes the description of FIG.

  FIG. 18 is an example of a flowchart showing processing for generating a display screen of character utterances according to the embodiment of the present invention. Each step in FIG. 18 is executed by the CPU 201 of the conference server 101. Specifically, step S1613, which is a processing step on the conference server side in FIG. 16, will be described in detail.

  In the loop from step S1801 to step S1808, processing is performed on each piece of audio data corresponding to the client apparatus 102 having the character utterance authority.

  In step S1802, the voice data is actually recognized in the voice buffer, and attention is paid to the voice data of one speaker (actually one terminal) who is going to speak a character from now on. Hereinafter, for the sake of convenience, it is referred to as “the utterance under attention”. For example, it expresses that “the utterance under consideration is not displayed as a character utterance”.

  In step S1803, it is determined whether or not to display the utterance under attention as a character utterance in the character utterance frame (character utterance frame display determination unit). Specifically, the determination is made based on whether the “speech frame display flag” is true or false. In the case of true, all utterances are displayed in the character utterance frame, and the process proceeds to step S1807. If false, the process advances to step S1804.

  In step S1804, it is determined whether or not a participant screen corresponding to the utterance being noticed (that is, corresponding to the client device 102 corresponding to the utterance being noticed) is displayed (image data display determination unit). If it is displayed (in the case of YES), the process proceeds to step S1805. If not displayed (NO), the process proceeds to step S1806.

  In step S1805, the utterance under attention is displayed as a character utterance on the corresponding participant screen.

  In step S1806, it is determined whether or not the utterance under consideration is displayed as a character utterance in the character utterance frame (participant utterance display determination unit). Specifically, the determination is made based on whether the “participant utterance display flag” is true or false. In the case of true, it is set to display the utterance under attention as a character utterance in the character utterance frame, and the process proceeds to step S1807. If false, the process proceeds to step S1808.

  In step S1808, the utterance under consideration is regarded as a character utterance and characters are synthesized in the character utterance frame. If there is already another character's character, it will be added so that it will not be erased.

Note that the character utterances made later in time may be shifted in time series, as shown by the deviation of the horizontal positions of the character utterances D and E in 1704 in FIG. In the conference server, as shown in the process on the conference server 101 side in FIG. 16, step S1614 (that is, the process described in the flowchart in FIG. 18) is asynchronously repeated. At that time, it is possible to realize by synthesizing the screen so that the display positions of the plurality of character utterances are shifted in time series according to the utterance time.
Above, description of the flowchart of FIG. 18 is completed.

  Next, a second embodiment will be described using the flowchart of FIG. Unlike the processing described with reference to the flowchart of FIG. 18, it is determined how to display the character output without using various preset flags.

  FIG. 19 is an example of a flowchart showing processing for generating a display screen for character utterances according to the second embodiment of the present invention. Each step in FIG. 19 is executed by the CPU 201 of the conference server 101.

  The loop from step S1901 to step S1906 is an iterative process for all character outputs.

  In step S1902, attention is focused on one character output.

  In step S1903, it is determined whether or not a participant screen corresponding to the focused character output is included in the entire image data. If included, the process proceeds to step S1904. If not included, the process proceeds to step S1905.

  In step S1904, the character output data under attention is synthesized with the participant screen corresponding to the character output under attention.

In step S1905, the character output under attention is displayed in the character utterance frame. If there is no character utterance frame, it is generated.
This completes the description of the flowchart in FIG.

  In the flowcharts of FIGS. 18 and 19, if there is a participant screen corresponding to the character output, it has been described that the participant screen and the output character are combined. Actually, it may be combined as image data in the image data area of the participant screen, or displayed in an adjacent area (for example, horizontally written below the image data, vertically written to the right of the image data) You may do it. In any method, it is only necessary to clearly understand that the character output corresponds to the participant screen.

  It should be noted that the configuration and contents of the various data described above are not limited to this, and it goes without saying that the various data and configurations are configured according to the application and purpose.

  Although one embodiment has been described above, the present invention can take an embodiment as, for example, a system, apparatus, method, program, or recording medium, and specifically includes a plurality of devices. The present invention may be applied to a system including a single device.

  Further, the program in the present invention is a program that allows a computer to execute the processing methods of the flowcharts shown in FIGS. 11 to 16 and 18, and the storage medium of the present invention uses the processing methods in FIGS. 11 to 16 and 18. A computer executable program is stored. Note that the program in the present invention may be a program for each processing method of each device in FIGS. 11 to 16 and 18.

  As described above, a recording medium that records a program that implements the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus stores the program stored in the recording medium. It goes without saying that the object of the present invention can also be achieved by executing the reading.

  In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium storing the program constitutes the present invention.

  As a recording medium for supplying the program, for example, a flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, DVD-ROM, magnetic tape, nonvolatile memory card, ROM, EEPROM, silicon A disk, solid state drive, or the like can be used.

  Further, by executing the program read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on an instruction of the program is actually It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the processing and the processing is included.

  Furthermore, after the program read from the recording medium is written to the memory provided in the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function expansion board is based on the instructions of the program code. It goes without saying that the case where the CPU or the like provided in the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  Further, the present invention may be applied to a system composed of a plurality of devices or an apparatus composed of a single device. Needless to say, the present invention can be applied to a case where the present invention is achieved by supplying a program to a system or apparatus. In this case, by reading a recording medium storing a program for achieving the present invention into the system or apparatus, the system or apparatus can enjoy the effects of the present invention. Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or apparatus can enjoy the effects of the present invention.

  In addition, all the structures which combined each embodiment mentioned above and its modification are also included in this invention.

101 conference server 102 client device 103 network 301 conference system 302 system administrator 303 group 304 group administrator 305 general user 306 conference room 307 free space

Claims (16)

A conference server capable of communicating with a client device via a network, capable of converting voice data from the client device into characters and displaying the characters,
Voice receiving means for receiving voice data transmitted from the client device;
Voice data registration means for registering the voice data received by the voice reception means in the voice data storage means in association with the client device;
Voice acquisition means for acquiring voice data corresponding to the client device from the voice data storage means;
Image receiving means for receiving image data transmitted from the client device;
Image data registration means for registering image data received by the image reception means in an image data storage means in association with the client device;
Voice data acquired by the voice acquisition means, voice data character conversion means for converting into characters,
When the image data corresponding to the converted character is set to be displayed on the client device, the image data is combined with the image data as a character output, and the image data is displayed on the client device. If it is not set to be displayed, image data based on a character utterance frame for displaying the character output is generated, and the image data combined with the character output is received from the client device. When the image data and the character utterance frame are generated, image combining means for combining all the image data based on the generated character utterance frame as whole image data;
Whole image data transmitting means for transmitting the whole image data synthesized by the image synthesizing means to the client device;
A conference server. Whether or not to display the converted character as a character output in a character utterance frame that is an area for displaying the character output, as a utterance character display setting storage means,
Character utterance frame display determination means for determining whether or not to display in the character utterance frame which is an area for displaying the character output as the character output based on the utterance frame display flag;
Have
Image composition means
When the character utterance frame display determination means determines that the character output is displayed in the character utterance frame, the image data based on the character utterance frame is generated, and the image data received from the client device or generated The conference server according to claim 1, wherein all image data based on the character utterance frame is synthesized as whole image data. The spoken character display setting storage means further includes:
Whether or not to display the character in the corresponding image data as the character output is stored as a value of a participant utterance display flag,
Participant utterance display determination means for determining whether to display the character output in the image data based on the participant utterance display flag;
Image data display determination means for determining whether or not the image data is included in the entire image data transmitted from the conference server to the client device;
The image composition means further includes:
The character utterance frame display determination means determines that the character output is not displayed in the character utterance frame,
When the image data display determining means determines that the image data is included in the entire image data, the character output is combined with the image data,
When it is determined that the image data is not included in the entire image data, and the character output is determined to be displayed on the image data by the participant utterance display determination means, the character The image data based on an utterance frame is generated, and all of the image data received from the client device or the generated image data based on the character utterance frame is synthesized as whole image data. Conference server. A voice output authority granting judgment unit for judging whether or not to output voice data corresponding to the client device based on the order of times when the voice data transmitted from the client device is registered in the voice data storage unit;
An audio output authority granting unit for granting an audio output authority to the client device when it is determined by the audio output authority granting determination unit that voice output;
Audio output end condition storage means for storing an audio output end condition for releasing the audio output authority of the client device to which the audio output authority is granted;
Audio output authority release determination means for determining whether or not to release the audio output authority given to the client device based on the audio output end condition stored in the audio output end condition storage means;
Audio output authority release means for releasing the audio output authority from the client device when it is determined by the audio output authority release determination means to release the audio output authority;
Voice data transmitting means for acquiring voice data of the client device to which the voice output authority is given by the voice acquisition means and transmitting the voice data to the client device;
The conference server according to any one of claims 1 to 3, wherein the conference server is provided. Character output authority for determining whether or not to convert voice data corresponding to the client device into characters and to output characters based on the order of times when the voice data transmitted from the client device is registered in the voice data storage means Grant determination means;
A character output authority granting unit for granting a character output authority to the client device when it is determined by the character output authority granting determination unit that character output;
Voice data corresponding to the client device to which the character output authority has been granted by the character output authority granting means is acquired from the voice data storage means and converted into characters;
Character output end condition storage means for storing a character output end condition for releasing the character output authority of the client device to which the character output authority is granted;
Character output authority release determination means for determining whether or not to release the character output authority given to the client device based on the character output end condition stored in the character output end condition storage means;
Character output authority release means for releasing character output authority from the client device when it is determined to release the character output authority by the character output authority release determination means,
The image composition means includes
When the character output authority is granted, the image data based on the character utterance frame is generated, and the image data received from the client device or all of the image data based on the generated character utterance frame is displayed as an entire image. 5. The conference server according to claim 4, wherein the conference server is synthesized as data. The audio output end condition is specified by the time that elapses without resumption of audio data after the audio data corresponding to the client device registered in the audio data storage means is interrupted,
The voice output authority release determination means is
The voice output authority given to the client device is canceled based on the interruption time of the voice data corresponding to the client device registered in the voice data storage means and the voice output end condition specified by the time. 6. The conference server according to claim 4, wherein the conference server determines whether or not. The character output end condition is specified by a time that elapses without restarting the voice data after the voice data corresponding to the client device registered in the voice data storage unit is interrupted,
The character output authority release determination means includes
The character output authority assigned to the client device is canceled based on the interruption time of the audio data corresponding to the client device registered in the audio data storage means and the character output end condition specified by the time. The conference server according to claim 5, wherein the conference server determines whether or not. Voice output authority number storage means for storing the number of the client devices to which the voice output authority is granted;
Further comprising
The voice output authority grant determining means further includes:
8. The method according to claim 4, wherein it is determined to output voice data to the client device corresponding to the number stored in the voice output authority number storage unit. Conference server. The character output authority grant determining means further includes:
9. The character output authority is given to the client device that is determined not to grant the voice output authority by the voice output authority determination unit. The conference server described in the section. Character output authority number storage means for storing the number of the client devices to which the character output authority is granted;
Further comprising
The character output authority grant determining means further includes:
10. The apparatus according to claim 5, wherein voice data is determined to be output to the client device corresponding to the number stored in the character output authority number storage unit. Conference server. The voice output end condition can specify a keyword that means the end of the speech among the speeches of the participants participating in the conference in the client device,
The voice output authority release determination means further includes:
The determination as to whether or not to release the audio output authority assigned to the client device based on a keyword that is the audio output end condition in audio data corresponding to the client device. The conference server according to any one of claims 4 to 10. The character output end condition can specify a keyword that means the end of the speech among the speeches of the participants participating in the conference in the client device,
The character output authority release determination means further includes:
The determination as to whether or not to release the character output authority assigned to the client device based on a keyword that is the character output end condition in the audio data corresponding to the client device. The conference server according to any one of claims 5 to 11. A client system and a conference server are communication systems that can communicate via a network,
The client device is
Voice data transmitting means for transmitting the speech of the participant participating in the conference input to the client device to the conference server as voice data;
Audio data receiving means for receiving audio data transmitted from the conference server;
Audio data reproduction control means for reproducing the audio data;
Have
Voice receiving means for receiving voice data transmitted from the client device;
Voice data registration means for registering the voice data received by the voice reception means in the voice data storage means in association with the client device;
Voice acquisition means for acquiring voice data corresponding to the client device from the voice data storage means;
Image receiving means for receiving image data transmitted from the client device;
Image data registration means for registering image data received by the image reception means in an image data storage means in association with the client device;
Voice data acquired by the voice acquisition means, voice data character conversion means for converting into characters,
When the image data corresponding to the converted character is set to be displayed on the client device, the image data is combined with the image data as a character output, and the image data is displayed on the client device. If it is not set to be displayed, image data based on a character utterance frame for displaying the character output is generated, and the image data combined with the character output is received from the client device. When the image data and the character utterance frame are generated, image combining means for combining all the image data based on the generated character utterance frame as whole image data;
Whole image data transmitting means for transmitting the whole image data synthesized by the image synthesizing means to the client device;
A conference system. A control method for a conference server that is communicable with a client device via a network and capable of converting voice data from the client device into characters and displaying the characters,
A voice receiving step for receiving voice data transmitted from the client device;
A voice data registration step in which the voice data registration means registers the voice data received in the voice reception step in the voice data storage means in association with the client device;
A voice acquisition step in which voice acquisition means acquires voice data corresponding to the client device from the voice data storage means;
An image receiving step for receiving image data transmitted from the client device;
Image data registration means for registering the image data received by the image receiving step in the image data storage means in association with the client device;
A voice data character conversion means for converting the voice data acquired by the voice acquisition step into a character;
When the image composition unit is set to display the image data corresponding to the converted character on the client device, the image composition unit composes the image data as character output, and the image data If it is not set to be displayed on the client device, it generates image data based on a character utterance frame for displaying the character output, and further, the image data combined with the character output; An image synthesis step of synthesizing all image data based on the generated character utterance frame as whole image data when the image data received from the client device and the character utterance frame are generated;
A whole image data transmitting means for sending the whole image data combined in the image combining step to the client device;
A method for controlling a conference server, comprising: A program for causing a computer to communicate with a client device via a network, functioning as a conference server capable of converting voice data from the client device into characters and displaying the characters,
The computer,
Voice receiving means for receiving voice data transmitted from the client device;
Voice data registration means for registering voice data received by the voice reception means in voice data storage means in association with the client device;
Voice acquisition means for acquiring voice data corresponding to the client device from the voice data storage means;
Image receiving means for receiving image data transmitted from the client device;
Image data registration means for registering image data received by the image receiving means in an image data storage means in association with the client device;
Voice data character conversion means for converting voice data acquired by the voice acquisition means into characters;
When the image data corresponding to the converted character is set to be displayed on the client device, the image data is combined with the image data as a character output, and the image data is displayed on the client device. If it is not set to be displayed, image data based on a character utterance frame for displaying the character output is generated, and the image data combined with the character output is received from the client device. When the image data and the character utterance frame are generated, image combining means for combining all the image data based on the generated character utterance frame as whole image data,
Whole image data transmitting means for transmitting the whole image data synthesized by the image synthesizing means to the client device;
A program for causing a server to function as a conference server. A computer-readable recording medium on which the program according to claim 15 is recorded.

Images