JP2020036225A - Information processing apparatus, information processing method, and information processing program - Google Patents

Abstract

To detect a speech request from a terminal and perform speech control without being affected by a connection method or a terminal type in a conference system that can support a plurality of connection methods and can be connected to a plurality of terminal types.SOLUTION: An information processing apparatus according to the present invention that decodes media signals received by a plurality of terminals, synthesizes one or more media signals selected from the plurality of media signals, and transmits an encoded synthesized media signal includes: speech request detecting means for analyzing each of the decoded media signals and detecting a predetermined speech request state on the basis of each of the media signals; and notification means for notifying a speech permission authority terminal of speech request information indicating that speech has been requested from the transmission source of the media signal from which the speech request state is detected when the speech request state is detected by the speech request detecting means.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, an information processing method, and an information processing program, and can be applied to, for example, a conference system.

  For example, a video conference system has a statement control function in which a participant requests a statement from a chair terminal and allows the participant to speak according to the chairman's judgment (see Patent Document 1).

  2. Description of the Related Art Conventionally, a speech control method of a video conference system uses an event notification function that can be used in a connection method (communication protocol) adopted by the video conference system. For example, a conference terminal requesting a statement request notifies the chair terminal of an event, and the chair determines whether or not to permit speech. When the chair permits speech, the chair terminal notifies the conference server of a speech permission event.

JP 2006-033657 A

  There are various types of connection systems (communication protocols) that can be used as recent video conference systems, and there are many types of conference terminals. For example, there is a method in which a conference terminal connects to a conference server of a different connection method (communication protocol) via a gateway. Further, the conference terminal may be a mobile terminal such as a smartphone or a tablet terminal instead of the dedicated terminal. Therefore, the video conference system is becoming capable of accommodating a plurality of connection methods and various types of conference terminals.

  However, when different connection methods or various types of conference terminals are connected to the video conference system, notification of a request from a conference terminal of a different connection type (communication protocol) or a request for a speech from a different type of conference terminal is sent to the chair terminal. There is a problem that the task cannot be performed.

  Therefore, in a conference system that can support a plurality of connection methods and can connect to a plurality of terminal types, it is necessary to detect a speech request from a terminal and perform a speech control without being affected by the connection method or the terminal type. There is a demand for an information processing apparatus, an information processing method, and an information processing program that can perform the processing.

  In order to solve such a problem, an information processing apparatus according to a first aspect of the present invention decodes a media signal received from each of a plurality of terminals and converts one or a plurality of media signals selected from the plurality of media signals. In an information processing apparatus that transmits a combined and encoded composite media signal to a plurality of terminals, (1) a statement that analyzes each decoded media signal and detects a predetermined statement request state based on each media signal; When the request request means and (2) the statement request state is detected by the statement request detecting means, statement request information indicating that the request has been made from the source of the media signal that has detected the statement request state is set in advance. Notification means for notifying the speech permission authority terminal.

  The information processing method according to the second invention decodes a media signal received from each of a plurality of terminals, combines one or a plurality of media signals selected from the plurality of media signals, and encodes the combined media. In the information processing method for transmitting a signal to a plurality of terminals, (1) the speech request detecting unit analyzes each of the decoded media signals, detects a predetermined speech request state based on each of the media signals, and (2) When the statement request detecting means detects the statement request state, the notifying means detects the statement request state, and transmits statement request information indicating that a request has been made from the transmission source of the media signal to a preset statement request. It is characterized in that the permission authority terminal is notified.

  An information processing program according to a third aspect of the present invention is a computer-readable storage medium that decodes a media signal received from each of a plurality of terminals, combines one or more media signals selected from the plurality of media signals, and encodes the combined media. In an information processing program for transmitting a signal to a plurality of terminals, the computer includes: (1) a message request detecting unit that analyzes each decoded media signal and detects a predetermined message request state based on each media signal; (2) When the utterance request detecting means detects the utterance request state, the utterance request information indicating that the utterance is requested from the transmission source of the media signal that has detected the utterance request state is transmitted to a predetermined utterance permission authority terminal. It is made to function as a notifying means for notifying to.

  According to the present invention, it is possible to detect a speech request from a terminal and perform speech control without being affected by a connection method or a terminal type.

FIG. 3 is a block diagram illustrating an internal configuration of a statement request detection unit and a statement control operation unit according to the embodiment, and a relationship between the statement request detection unit and the statement control operation unit. 1 is an overall configuration diagram illustrating an overall configuration of a conference system according to an embodiment. FIG. 2 is an internal configuration diagram illustrating an internal configuration of an MCU according to the embodiment. FIG. 3 is an explanatory diagram illustrating an overall operation of the audio signal processing unit according to the embodiment. 6 is a flowchart illustrating a statement request detection process in the MCU according to the embodiment. It is a block diagram showing an internal configuration of a synthesis unit of an image signal processing unit according to a modified embodiment.

(A) Main Embodiment Hereinafter, an embodiment of an information processing apparatus, an information processing method, and an information processing program according to the present invention will be described in detail with reference to the drawings.

  In this embodiment, a case where a video conference system is constructed using the present invention will be exemplified.

(A-1) Configuration of Embodiment (A-1-1) Overall Configuration FIG. 2 is an overall configuration diagram illustrating an overall configuration of a conference system according to the embodiment.

  2, a conference system 7 according to the embodiment includes a conference server 1 and a plurality (three in FIG. 2) of conference terminals 5 (5-1 to 5-3).

  The conference server 1 synthesizes a conference by synthesizing a gatekeeper (GK) 2 having functions such as a connection permission of the conference terminal 5 and an address conversion process, and audio, video, and data obtained from the conference terminals 5 at a plurality of bases. And a multipoint control unit (MUC: Multipoint Control Unit, hereinafter referred to as “MUC”) 3 having a function of converting the data into data for use.

  The gatekeeper (GK) may not be installed due to the system configuration in the following cases.

  Here, the following case means that in the conference server 1, In the case where the H.323 (described later) is not used, the correspondence between the IP address and the telephone number is stored and managed, and there is no need for mediation in connection to the other party's conference terminal.

  For example, in the case of three or more participants (conference terminal 5), a device that provides a conference server function called MCU3 is used for the conference system 7. The MCU 3 receives audio / video from each of the conference terminals 5 (5-1 to 5-3), synthesizes a voice to be uttered, and distributes the synthesized voice to each of the conference terminals 5 (5-1 to 5-3). . Further, the MCU 3 switches or synthesizes a video of a required format and distributes the video to each conference terminal 5 (5-1 to 5-3).

  The conference server 1 connects a plurality of conference terminals 5-1 to 5-3 to set a virtual conference. The conference server 1 receives the audio, video, and data of the participant through a plurality of conference terminals 5-1 to 5-3 connected to the conference, and synthesizes the video data or the audio according to the conference settings. Then, it transmits to each of the conference terminals 5-1 to 5-3.

  The conference server 1 can use a plurality of connection methods. The connection method (communication protocol) of the conference system includes, for example, SIP (Session Initiation Protocol) and the ITU-T recommended communication protocol H.264. H.323, a Web conference service using the Internet (Internet telephone service), and the like.

  Here, the Web conference service can be interconnected with a conference terminal by using WebRTC (Web Real-Time Communication) and using protocols such as HTML, HTTP, TCP / IP, and UDP / IP. is there.

  The conference server 1 can support a plurality of connection methods and can connect to various conference terminals 5.

  In this embodiment, for example, the conference terminal 5-1 and the conference terminal 5-2 are connected to the H.264 connection system (communication protocol) of the conference system. It is assumed that the terminal uses H.323 (or SIP) and participates in the conference of the conference server 1.

  In this embodiment, for example, the conference terminal 5-3 is a terminal that uses the connection method (communication protocol) of the conference system used in the Web conference service and participates in the conference of the conference server 1. I do. That is, it is assumed that the conference terminal 5-3 is connected to the conference server 1 via the gateway (GW) 6.

  Each of the conference terminals 5-1 to 5-3 is used by a participant who participates in a conference at each base, and is, for example, a device including a microphone, a speaker, a camera, a display, a control device, and the like. As each of the conference terminals 5-1 to 5-3, for example, a dedicated conference terminal, a personal computer, a smartphone, a tablet terminal, a wearable terminal, a mobile terminal, and the like can be applied.

  Each of the conference terminals 5 (5-1 to 5-3) is, for example, a terminal using software that provides various operation functions on a personal computer, or a terminal in which a hardware maker implements a unique operation method. Applicable.

  In this embodiment, a case where three conference terminals 5-1 to 5-3 hold one conference is illustrated, but the number of conference terminals 5 that hold one conference is not particularly limited. Absent.

[Speech control operation unit 4]
The utterance control operation unit 4 can specify that one or more participants are allowed to make a utterance. The speech control operation unit 4 is one of the functions of the conference system provided by the conference server 1.

  More specifically, from among a plurality of participants, a person (speaker permitted to decide) who can give permission to the participant is determined, and in the conference terminal 5 of the person permitted to speak, the speech control operation unit 4 is determined. Works effectively.

  In general, in many cases, the chairperson of the conference is the speaker who has determined permission to speak. Therefore, here, the conference terminal 5 of the person who has determined permission to speak is also referred to as a “chairperson terminal”. In FIG. 1, the conference terminal 5-1 is the chair terminal. The chair terminal is also referred to as a "speaking permission authority terminal".

  Note that, in this embodiment, a case is described in which the speech control operation unit 4 functions effectively by being turned on in the conference terminal 5 (5-1 to 5-3) of the speech permission decider, but is not limited thereto. Not done.

  The chair terminal 5-1 can receive a speech request from the conference terminal 5 of the participant who wants to speak. At the chair terminal 5-1 that has received the statement request, the statement permission determiner determines whether or not to permit the participant to speak, based on the progress of the conference and the like.

  When the speech permission decider permits the participant to speak, the speech permission decider specifies the participant in the chair terminal 5-1 by a predetermined designation method. Then, the chair terminal 5-1 sends information (speech designation information) designating the conference terminal 5 of the participant to the MCU 3.

  The MCU 3 that has received the utterance designation information from the chair terminal 5-1 transmits the audio (may include video) from the designated conference terminal 5 to all the conference terminals 5 (5-1 to 5-3). To be synthesized. That is, the voice of the participant whose speech has not been permitted is included in the synthesized speech for the conference, and the other participants can hear the voice of the participant whose speech has been permitted. In the following, the authority to speak is also called the right to speak.

  By the way, conventionally, when a participant wants to speak, the conventional conference terminal is equipped with a statement request processing unit for transmitting a statement request to the chair terminal. The request for speech can be made either directly from the requesting terminal to the chair terminal or indirectly via the MCU, and both are included.

  However, it is possible to support multiple connection methods, and when holding a conference with a variety of conference terminals, it is difficult to unify the request method because the connection method (communication protocol) is different. May not have a statement request processing unit.

  Therefore, in this embodiment, the MCU 3 of the conference server 1 determines whether the participant wants to speak or the participant speaks based on the audio signal received from each of the conference terminals 5 (5-1 to 5-3). The state of having made the message is detected, and the statement request operation can be notified to the statement control operation unit 4 of the chair terminal 5-1.

(A-1-2) Internal Configuration of MCU 3 FIG. 3 is an internal configuration diagram showing an internal configuration of the MCU 3 according to the embodiment.

  3, the MUC 3 according to the embodiment roughly includes an audio signal processing unit 10, a video signal processing unit 20, and a control unit 30.

  In addition, although the case where the MCU 3 includes the audio signal processing unit 10 and the video signal processing unit 20 is illustrated, the MCU 3 may include a data signal processing unit and the like. The MCU 3 is constructed by installing a processing program (for example, the information processing program according to the present embodiment) on an information processing apparatus such as a personal computer. Note that the present invention is not limited to one information processing device, and the processing units in FIG. 3 may be distributed and arranged in a plurality of information processing devices. Note that some or all of the components of the audio signal processing unit 10 and the video signal processing unit 20 may be configured by hardware.

[Control unit 30]
The control unit 30 controls the functions of the audio signal processing unit 10 and the video signal processing unit 20 of the MCU 3, and controls conference holding, encoding processing, and the like. It is a processing unit or device that performs various functions. The control unit 30 also controls the bandwidth of a communication line between the conference server 1 and each of the conference terminals 5 (5-1 to 5-3).

[Audio signal processing unit 10]
The audio signal processing unit 10 decodes audio (encoded audio signal) received from each of the plurality of conference terminals 5-1 to 5-3, and sets the audio to be synthesized based on an instruction from the control unit 30. The voice is synthesized, the synthesized signal is encoded, and transmitted to the plurality of conference terminals 5-1 to 5-3.

  The audio signal processing unit 10 includes an audio receiving unit 11 (11-1 to 11-3), a decoding unit 12 (12-1 to 12-3), a combining unit 17, an encoding unit 15, and an audio transmitting unit 16 (16- 1 to 16-3).

  Here, in order to hold one or more meetings, the MCU 3 virtually forms one or more meeting rooms on the meeting system in accordance with the number of held meetings.

  Also, here, the audio receiving unit 11, the decoding unit 12, and the audio transmitting unit 16 are created by the number of conference terminals 5 connected to one conference room held by the MCU 3.

  The audio receiving unit 11 (11-1 to 11-3) receives an audio signal from the corresponding conference terminal 5, and supplies the received audio signal to the decoding unit 12 (12-1 to 12-3). .

  The decoding unit 12 (12-1 to 12-3) decodes the audio signal received from the audio receiving unit 11 (11-1 to 11-3) and supplies the audio signal to the synthesizing unit 17. Between the MCU 3 and each of the conference terminals 5 (5-1 to 5-3), an audio signal is compression-encoded and transmitted / received in order to reduce the amount of information and secure a band. The compression coding method of the audio signal is not particularly limited. For example, G. 711 (ITU-T standardization technology); 722, MPEG-4 AAC-LD, etc., and more specifically, PCM (pulse code modulation), ADPCM (adaptive differential PCM), or the like can be used.

  The decoding unit 12 (12-1 to 12-3) constantly decodes the audio signal received from the conference terminal 5 and supplies the audio signal to the synthesizing unit 17. That is, in addition to the audio signal from the conference terminal 5 to which the floor is given (to speak), the voice received from the conference terminal 5 to which the floor is not given (to speak is not allowed). The decoding unit 12 also decodes the signal.

  Upon receiving the decoded audio signal (PCM signal) from the decoding unit 12 (12-1 to 12-3), the synthesizing unit 17 selects one or more audio signals to be synthesized and selects the selected audio signal. Are combined and given to the encoding unit 15. Further, the synthesizing unit 17 includes a speech request detecting unit 13 and a synthesis target voice selecting unit 14.

  Receiving the speech signal (PCM signal) decoded from the decoding unit 12 (12-1 to 12-3), the speech request detection unit 13 allows the participant to request a speech based on the speech signal from each conference terminal 5. It is to detect whether or not it is performed. A detailed description of the statement request detection unit 13 will be described later.

  The synthesis target audio selection unit 14 selects an audio signal to be synthesized from the audio signals (PCM signals) decoded from the decoding units 12 (12-1 to 12-3), and selects the audio signal selected as the synthesis target. To the encoding unit 15.

  The encoding unit 15 receives the audio signal synthesized by the synthesis target audio selecting unit 14 of the synthesizing unit 17, encodes the audio signal, and outputs the encoded audio signal to the audio transmitting unit 16 (16-1 to 16-3). Give to).

  An arbitrary number of encoding units 15 are created for one conference. This is because the conference server 1 of this embodiment can support a plurality of connection methods (communication protocols), and thus the communication methods are different, so that the encoding methods may be different. Therefore, according to the instruction of the control unit 30, the number of the encoding units 15 corresponding to the number of encoding systems to be supported may be created.

  The encoding method applied to the audio signal has been described in the detailed description of the decoding unit 12, and the description is omitted here.

  The audio transmission unit 16 (16-1 to 16-3) transmits the audio signal encoded by the encoding unit 15 to the corresponding conference terminal 5 (5-1 to 5-3).

[Video signal processing unit 20]
The video signal processing unit 20 decodes the video (encoded video signal) received from each of the plurality of conference terminals 5-1 to 5-3, and changes the video format based on an instruction from the control unit 30. The data is converted and synthesized, and transmitted to the plurality of conference terminals 5-1 to 5-3.

  The video signal processing unit 20 includes a video receiving unit 21 (21-1 to 21-3), a decoding unit 22 (22-1 to 22-3), a combining unit 27, an encoding unit 25, and a video transmitting unit 26 (26- 1-26-3).

  Here, in order to hold one or more meetings, the MCU 3 virtually forms one or more meeting rooms on the meeting system in accordance with the number of held meetings.

  Also, here, the video receiving unit 21, the decoding unit 22, and the video transmitting unit 26 are created for one conference held by the MCU 3 by the number of conference terminals 5 to be connected.

  The video receivers 21 (21-1 to 21-3) receive video signals from the corresponding conference terminals 5, and provide the received video signals to the decoders 22 (22-1 to 22-3). . Between the MCU 3 and each of the conference terminals 5 (5-1 to 5-3), in order to reduce the amount of information and secure a band, a video signal is compression-encoded and transmitted / received. The compression encoding method of the video signal is not particularly limited. For example, H. 261, H .; H.263, MPEG-2, MPEG-4, H.264. 264, H .; 265 or the like can be used.

  The decoding unit 22 (22-1 to 22-3) decodes the video signal received from the video receiving unit 21 (21-1 to 21-3) and supplies the video signal to the synthesizing unit 23.

  The synthesizing unit 23 converts the video signal into a required video format based on the video signal from the decoding unit 22 (22-1 to 22-3), synthesizes the video, and provides the video to the encoding unit 25. Usually, one synthesizing unit 23 is created for one conference. However, a plurality of combining units 23 may be created for one conference.

  The encoding unit 25 encodes the video signal from the synthesizing unit 23 and supplies the encoded video signal to the video transmission unit 26 (26-1 to 26-3).

  An arbitrary number of encoding units 25 are created for one conference. This is because the conference server 1 of this embodiment can support a plurality of connection methods (communication protocols), and thus the communication methods are different, so that the encoding methods may be different. Therefore, in accordance with an instruction from the control unit 30, the number of encoding units 25 corresponding to the number of encoding systems to be supported may be created.

  The encoding method applied to the video signal has been described in detail in the decoding unit 22, and the description is omitted here.

  The video transmission unit 26 (26-1 to 26-3) transmits the video signal encoded by the encoding unit 25 to the corresponding conference terminal 5 (5-1 to 5-3).

(A-1-3) Internal Configuration of the Statement Request Detector 13 FIG. 1 shows the internal configuration of the statement request detector 13 and the statement control operation unit 4 according to the embodiment, and the statement request detector 13 and the statement control operation unit 4. FIG. 4 is a block diagram showing the relationship of FIG.

  In FIG. 1, the comment request detection unit 13 includes a comment request target selection unit 131, a level detection unit 132, a DTMF detection unit 133, a voice recognition detection unit 134, a keyword storage unit 135, and a comment request detection unit 136.

  Here, the “speaking request state” refers to a state in which a participant is speaking or the participant wants to speak by analyzing media data (sound signal in this embodiment) for which speaking is not permitted. It means that you are doing.

  Based on the speech permission information from the control unit 30 of the MCU 3, the speech request target selection unit 131 selects a speech request state detection target and a speech from the speech signals from the decoding units 12 (12-1 to 12-3). Select a signal. That is, the statement request target selection unit 131 selects the voice signal of the participant whose speech is not permitted.

  More specifically, the speaking permission information includes identification information of the conference terminal 5 to which speaking is currently permitted. Therefore, the statement request target selection unit 131 selects the audio signal of the conference terminal 5 other than the conference terminal 5 to which the statement is permitted, based on the statement permission information. In this way, the voice of the participant other than the participant to whom the floor has been granted is selected during the conference, and from the selected voice, it is determined whether the participant is speaking or the participant wants to speak. Can be detected.

  The level detection unit 132 detects a speech request state based on the audio level of the audio signal selected by the speech request target selection unit 131. That is, when the participant speaks, the sound level may increase. Therefore, the level detection unit 132 monitors the audio level of the audio signal, detects a state in which the audio level has increased as a speech request state, and detects the speech request state. More specifically, the level detection unit 132 monitors the time-series change of the audio level of each conference terminal 5 and sets the speech request state when the gradient of the time-series change of the audio level sharply increases beyond the threshold. Good. Further, the voice request state may be set when the audio level of each conference terminal 5 exceeds one or more threshold values.

  The DTMF detection unit 133 detects a speech request state when detecting a synthesized signal tone (DTMF signal) in a frequency band based on the audio signal selected by the speech request target selection unit 131. For example, when the conference terminal 5 is equipped with a DTMF transmitter, when requesting a speech, it can be realized by arranging among the participants to transmit any DTMF signal. As a result, the participant requesting a speech operates the DTMF transmitter to transmit a voice signal including the DTMF signal to the MCU 3, and the DTMF detection unit 133 detects the DTMF signal, thereby detecting the speech request state.

  The voice recognition detection unit 134 performs a voice recognition process based on the voice signal selected by the voice request target selection unit 131, and detects a voice request state based on the voice recognition result. The keyword storage unit 135 is a corpus (dictionary) that stores a preset message. The keyword storage unit 135 stores at least a keyword indicating that a participant requests a statement. The voice recognition detection unit 134 converts a voice signal into a sound wave by a voice recognition algorithm, specifies a phoneme, and performs voice recognition by pattern matching. The speech recognition detection unit 134 refers to the keyword storage unit 135 and matches the obtained word with the keyword to determine whether or not a request is made to make a statement. Note that the speech recognition algorithm is not particularly limited.

  Here, any one of the detection methods can be designated for the level detection unit 132, the DTMF detection unit 133, and the voice recognition detection unit 134. One detection method may be specified, or a plurality of detection methods may be specified. Various methods can be applied as a selection method of the detection method. For example, a detection method may be set in advance. Further, the conference terminal 5 may specify the detection method by an operation by the participant or the chair.

  In addition, the level detecting unit 132, the DTMF detecting unit 133, or the voice recognition detecting unit 134 monitors a voice signal (decoded signal) for which the voice is not permitted to detect, and detects a voice request state from the voice signal, thereby providing the following advantages. There is.

  As described above, the decoding unit 12 (12-1 to 12-3) constantly decodes the audio signal received from the conference terminal 5 (5-1 to 5-3). That is, the decoding unit 12 always decodes not only the voice of the participant (conference terminal 5) to which the speech is permitted, but also the voice of the participant (the conference terminal 5) to which the speech is not permitted. Conventionally, the decoded voice of a participant whose speech is not permitted is discarded. This is because the decoded voice of the participant whose speech is not permitted is not a voice signal to be synthesized.

  However, in this embodiment, the level detection unit 132, the DTMF detection unit 133, and the speech recognition detection unit 134 monitor the decoded voice of the participant whose speech is not permitted, which has conventionally been discarded because it is not targeted for synthesis. Then, the participant's comment request state is detected.

  From another point of view, conventionally, the decoding unit 12 always decodes even the voice not used when synthesizing the voice, so that the processing load is applied and the decoded voice is not effectively used. However, according to this embodiment, the decoded voice that is always decoded can be effectively used, and the processing load of the decoding unit 12 does not increase by implementing this method.

  When the speech request detection unit 136 detects the speech request state by the level detection unit 132, the DTMF detection unit 133, or the voice recognition detection unit 34, the speech request detection unit 136 notifies the speech request reception unit 41 of the speech control operation unit 4 of the speech request signal. . Thereby, even when the connection method, the type of the conference terminal 5 and the like are different, it is possible to notify the speech control operation unit 4 of the speech request signal.

  As illustrated in FIG. 1, the utterance control operation unit 4 according to the embodiment includes an utterance request reception unit 41, an utterance request display unit 42, and an utterance designation unit 43.

  The comment request receiving unit 41 receives a comment request signal from the MCU 3. As described above, when the speech request detection unit 13 of the MCU 3 detects the speech request state, the speech request signal is received from the speech request detection unit 13.

  The statement request display section 42 displays on the display (display section) of the terminal 5 (that is, the chair terminal) on which the statement control operation section 4 is mounted, that the statement request has been made. Thus, the chair can be notified that another participant wants to speak.

  When the chair permits the participant who is requesting the utterance to speak, the utterance designation unit 43 notifies the MCU 3 of utterance designation information indicating that the participant is permitted to speak. The speech designation information includes, for example, identification information of the conference terminal 5 of the participant. Accordingly, the synthesis target voice selection unit 14 can set the voice signal of the participant whose speech is not permitted to be synthesized based on the utterance designation information from the control unit 30 of the MCU 30.

(A-2) Operation of Embodiment [Overall Operation of Conference Server (Audio Signal Processing Unit)]
Hereinafter, first, the operation of the audio signal processing in the conference server 1 will be described with reference to the drawings.

  FIG. 4 is an explanatory diagram showing the overall operation of the audio signal processing unit 10 according to this embodiment.

  When a call is established between the conference server 1 and the conference terminals 5-1 to 5-3, the MCU 3 notifies the conference terminals 5-1 to 5-3 of the transmission destination information of the audio signal (S11). The destination information of the audio signal includes information on server addresses and communication port numbers of the audio receiving units 11-1 to 11-3 of the audio signal processing unit 10. The information assigned to the audio signal processing unit 10 by the control unit 30 when the call is established is used as the information of the server address and the communication port number. Then, after the call is established, each of the conference terminals 5-1 to 5-3 transmits an audio signal to the audio signal processing unit 10.

  In the audio signal processing unit 10, the audio receiving units 11-1 to 11-3, the decoding units 12-1 to 12-3, and the audio transmitting units 16-1 to 16-3 are provided for each of the conference terminals 5-1 to 5-3. Is created (S12). Here, since the conference server 1 establishes a call with the three conference terminals 5-1 to 5-3, the audio receiving units 11-1 to 11- corresponding to the respective conference terminals 5-1 to 5-3. 3. The decoding units 12-1 to 12-3 and the audio transmission units 16-1 to 16-3 are created.

  Audio signals from the conference terminals 5-1 to 5-3 are received by the audio receiving units 11-1 to 11-3, and the received audio signals are decoded by the decoding units 12-1 to 12-3. Obtained (S13).

  The decoded audio signal is sent to the synthesizing unit 17, where the audio signal is synthesized (S14). At this time, the control unit 30 instructs the synthesis unit 17 about information on the synthesis target, and the synthesis unit 17 synthesizes the audio signal of the synthesis target specified by the control unit 30. That is, the speech permission information including the identification information of the conference terminal permitted to speak is notified from the control unit 30 to the synthesizing unit 17, and the synthesizing unit 17 synthesizes the audio signal from the conference terminal 5 permitted to speak.

  Then, the synthesized audio signal is provided to the encoding unit 15, and the audio signal is encoded by the encoding unit 15 (S15).

  The audio signal encoded by the encoding unit 15 is transmitted to the corresponding audio transmission units 16-1 to 16-3 according to the instruction of the control unit 30, and transmitted to the conference terminals 5-1 to 5-3. (S16). The operation is performed again from the above-described processing in step S13 until the end of the conference.

[Speech request detection process]
Next, a speech request detection process in the MCU 3 according to the embodiment will be described in detail with reference to the drawings.

  FIG. 5 is a flowchart illustrating a statement request detection process in the MCU 3 according to the embodiment.

  As described in S14 of FIG. 4, the synthesizing unit 17 is notified of the speech permission information from the control unit 30, and the synthesis target voice selection unit 14 of the synthesis unit 17 converts the speech signal to be synthesized based on the speech permission information. Combine.

  Along with such speech synthesis processing by the synthesis target speech selection unit 14, the speech request detection unit 13 monitors the speech signal of a participant whose speech is not permitted.

  As the method of detecting the speech request state, any one of a voice level method, a DTMF detection method, and a voice recognition method or a combination thereof is designated.

  In FIG. 5, the statement request target selection unit 131 selects a voice signal of a participant (conference terminal 5) not permitted to speak, based on the statement permission information (S101).

[Audio level method]
The operation when the voice level method is designated as the method of detecting the speech request state will be described.

  The level detector 132 monitors the audio signal selected by the synthesis target audio selector 14. The level detection unit 132 compares the audio level value of the input audio signal with the threshold, and determines whether the audio level value is larger than the threshold (S102).

  Then, when the voice level value is larger than the threshold, the level detection unit 132 detects the speech request state, and proceeds to S106. If the audio level value is equal to or less than the threshold, the process returns to S101, and the level detection unit 132 continues monitoring the audio signal.

  For example, when a participant who is not permitted to speak makes a certain speech while the conference is in progress, or when the participant makes a statement such as “Chair, please let me speak”, an audio signal from the conference terminal 5 at the base concerned is given. Will increase. Since the voice of this participant is not a synthesis target, it is not included in the synthesized voice at this time, and other participants cannot hear it. However, even in such a case, since the voice level value becomes larger than the threshold, the voice request state is detected, and the voice control operation unit 4 is notified of the voice request.

  Here, the case where the audio level value of the audio signal is compared with the threshold has been described as an example, but the time series change of the audio level is monitored, and the inclination of the audio level time series change is compared with the threshold. Is larger than the threshold value, the statement request state may be detected. Alternatively, the voice request state may be detected when the voice of a certain level in the time series of the voice level exceeds the average voice level (threshold) for a certain time.

[DTMF detection method]
The operation when the DTMF method is specified as the method of detecting the speech request state will be described.

  The DTMF detection unit 133 monitors the audio signal selected by the speech request target selection unit 131. The DTMF detection unit 133 determines whether or not the input audio signal includes a synthesized signal sound (DTMF signal) in a frequency band (S103).

  Then, when the DTMF signal is included, the DTMF detection unit 133 detects the speech request state, and proceeds to S106. When the DTMF signal is not included, the process returns to S101, and the DTMF detection unit 133 continues monitoring the audio signal.

[Speech recognition method]
The operation when the voice recognition method is specified as the method of detecting the speech request state will be described.

  The speech recognition detection unit 134 monitors the speech signal selected by the speech request target selection unit 131. The voice recognition detection unit 134 converts a voice signal into a sound wave by a voice recognition algorithm, specifies a phoneme, and performs voice recognition by pattern matching (S104). The speech recognition detection unit 134 matches the obtained word with the keyword by referring to the keyword storage unit 135 (S105).

  Then, when the keyword is detected, the voice recognition detection unit 134 detects the state of the utterance request, and proceeds to S106. When no keyword is detected, the process returns to S101, and the voice recognition detection unit 134 continues monitoring the voice signal.

  For example, in the keyword storage unit 135, keywords such as “chair” and “please let me speak” are stored. Then, when a participant who is not permitted to speak emits a message such as "Chairman, let me speak", it is detected as a speech request state by voice recognition. Therefore, the speech control operation unit 4 is notified of the speech request.

  Next, the speech request detection unit 136 is notified of a detection result from the level detection unit 132, the DTMF detection unit 133, or the speech recognition detection unit 34.

  When the speech request state is detected, the speech request detection unit 136 notifies the speech control operation unit 4 of a speech request signal including the identification information of the conference terminal 5 that has detected the speech request state (S106).

  In the statement control operation unit 4, the chair determines whether or not the participant is permitted to speak. Then, when the chair allows the participant to speak, the speech designation information including the identification information of the conference terminal 5 of the participant is transmitted to the MCU 3. When the speech designation information is received by the synthesis target voice selection unit 14 via the control unit 30 of the MCU 3 (S107), the synthesis target speech selection unit 14 converts the speech signal of the conference terminal 5 specified by the speech designation information into the synthesis target speech signal. (S108). Thereby, a speech synthesis including the speech of the participant is created.

(A-3) Effects of Embodiment As described above, according to this embodiment, a decoded audio signal is monitored in a conference room using speech control without using a special speech request mechanism. This makes it possible to detect that the participant has made a statement or desires to make a statement, and allows the participant to determine whether or not to permit speech.

  Further, in this embodiment, since the speech request state is detected from the decoded audio signal, regardless of the difference in the communication protocol (connection method) with the conference server, the connection using the Internet, and the difference in the type of the conference terminal, Utterance control can be performed. As a result, a conference system can be realized by mutually using various terminals.

  Furthermore, in this embodiment, conventionally, the voice from each site is always decoded and the voice not used for the voice synthesis is discarded, but the voice signal not used for the voice synthesis can be effectively used. Also, there is no increase in processing due to voice decoding.

(B) Other Embodiments Although various modified embodiments have been described in the above-described embodiments, the present invention can be applied to the following modified embodiments.

  (B-1) In the above-described embodiment, the case where it is detected that a participant has made a speech or that the participant wants to make a speech is detected based on a voice signal of a participant (conference terminal) not permitted to speak. Illustrated. However, the detection may be performed based on image data of a participant (conference terminal) not permitted to speak, which is limited to this.

  For example, as illustrated in FIG. 6, the synthesizing unit 24 of the video signal processing unit 20 includes an utterance request detection unit 13-1 instead of the utterance request detection unit 13 in the above-described embodiment. -1 may analyze the decoded image data to detect the speech request state. At this time, as illustrated in FIG. 6, the synthesis target speech selection unit 14-1 gives the audio signal selected as the synthesis target to the encoding unit 15 instead of the synthesis target speech selection unit 14.

  Here, when the speech request detection unit 13-1 receives the decoded video signal from the decoding unit 22 (22-1 to 22-3), the participant makes a speech based on the video signal from each conference terminal 5. This is to detect whether or not the request has been made. A detailed description of the statement request detection unit 13-1 will be described later. Note that the statement request detection unit 13-1 can be used in combination with the statement request detection unit 13 of the above-described embodiment.

  Here, the synthesis target voice selection unit 14-1 is decoded by the decoding unit 12 (12-1 to 12-3) based on the detection result of the speech request detection unit 13-1 received via the control unit 30. An audio signal to be synthesized is selected from the audio signals thus obtained, and the audio signal selected as the synthesis target is provided to the encoding unit 15. Note that the synthesis target voice selection unit 14-1 can be used in combination with the synthesis target voice selection unit 14 of the above-described embodiment.

  Here, for example, a case is described in which the statement request detection unit 13-1 detects that the participant is appealing for a statement by waving his hand toward the camera.

  In such a case, the speech request detection unit 13-1 may detect the motion of a participant whose speech is not permitted by using a motion detection (or moving object detection) algorithm for the decoded video stream. Note that various algorithms can be applied to the motion detection algorithm. For example, the utterance request detection unit 13-1 divides the video stream into frames, detects a detectable object such as a "hand" in advance, detects the target, and detects the moving amount and moving speed of the object. For example, a method of detecting a speech request state based on a comparison result between the command and a preset threshold may be applied. Of course, the present invention is not limited to this, and various methods can be applied as long as the movement (movement) of the participant who is requesting the speech can be detected from the video stream.

7 conference system 7 conference server, 2 gatekeeper (GK), 3 MCU, 10 audio signal processing unit, 20 video signal processing unit, 30 control unit,
11 (11-1 to 11-3): voice receiving unit, 12 (12-1 to 12-3): decoding unit, 17: combining unit, 15: coding unit, 16 (16-1 to 16-3) ... Speech transmission unit, 13 ... Speaking request detection unit, 14 ... Synthesis target selection unit,
131: utterance request target selection unit 132: level detection unit 133: DTMF detection unit 134: voice recognition detection unit 134: keyword storage unit 136: utterance request detection unit
5 (5-1 to 5-3): conference terminal, 4: speech control operation unit, 41: speech request receiving unit, 42: speech request display unit, 43: speech designation unit.

Claims (9)

An information processing apparatus that decodes a media signal received from each of a plurality of terminals, combines one or a plurality of media signals selected from the plurality of media signals, and transmits an encoded combined media signal to the plurality of terminals. At
Analyzing each of the decoded media signals, speech request detection means for detecting a predetermined speech request state based on each of the media signals,
When the utterance request detecting means detects the utterance request state, the utterance request information indicating that the utterance is requested from the transmission source of the media signal that has detected the utterance request state is transmitted to the utterance permission authority set in advance. An information processing apparatus comprising: a notification unit that notifies a terminal.   2. The information processing apparatus according to claim 1, wherein the speech request detection unit monitors the decoded media signal other than the synthesis target based on synthesis target information related to the generation of the synthesis media signal. The media signal is an audio signal,
3. The speech request detection unit according to claim 1, wherein the speech request detection unit detects the speech request state based on a time-series change of each audio signal not used for the synthesized speech among the decoded audio signals. An information processing apparatus according to claim 1.   4. The information processing apparatus according to claim 3, wherein the statement request detecting unit detects the statement request state based on a change in a speech level value of each of the speech signals not used for the synthesized speech. .   4. The speech request detecting means performs speech recognition using each of the speech signals not used for the synthesized speech, and detects the speech request state based on the speech recognition result. An information processing apparatus according to claim 1.   4. The information according to claim 3, wherein the speech request detection unit detects the speech request state by detecting a frequency synthesis signal included in each of the speech signals that is not used in the synthesized speech. Processing equipment. The media signal is a video signal,
3. The speech request detection unit according to claim 1, wherein the speech request detection unit detects the speech request state based on a time-series change of each video signal not used for the composite video among the decoded video signals. An information processing apparatus according to claim 1. An information processing method for decoding a media signal received from each of a plurality of terminals, combining one or a plurality of media signals selected from the plurality of media signals, and transmitting an encoded combined media signal to the plurality of terminals At
The speech request detection means analyzes each of the decoded media signals and detects a predetermined speech request state based on each of the media signals.
When the notifying means detects the utterance request state by the utterance request detecting means, utterance request information indicating that an utterance has been requested from the transmission source of the media signal that has detected the utterance request state is set in advance. An information processing method for notifying an authorized speech permission terminal. An information processing program for decoding a media signal received from each of a plurality of terminals, combining one or a plurality of media signals selected from the plurality of media signals, and transmitting an encoded combined media signal to the plurality of terminals At
Computer
Analyzing each of the decoded media signals, speech request detection means for detecting a predetermined speech request state based on each of the media signals,
When the utterance request detecting means detects the utterance request state, utterance request information indicating that the source of the media signal that has detected the utterance request state has requested utterance is set to a predetermined utterance permission authority. An information processing program for functioning as a notifying means for notifying a terminal.

Images