JP2015070515A - Information processing apparatus, information processing system, control method of information processing apparatus, control method of information processing system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for smoothly recovering delay of voice when after transmitting inputs of voice data in a plurality of client terminals to a server and mixing the voice data in the server, the voice data are transmitted to other client terminals as mixed voice data.SOLUTION: When voice data in a queue for storing voice data received from respective client terminals exceeds predetermined capacity in a server for mixing voice data received from a plurality of client terminals and distributing the mixed voice data to the respective client terminals, delay of voice is generated. In this case, delay of distribution of voice data is recovered by skipping silent data determined as data not including voice out of voice data to be mixed without using the silent data for mixing.

Description

  The present invention relates to a technique for recovering audio delay due to network delay or the like in an application that transmits and receives audio over a network.

  In an application that transmits and receives audio over a network, for example, a queue for accumulating uplink audio from each client may be prepared on the server in consideration of network fluctuations. However, having a queue also causes a situation in which delay increases as data accumulates in the queue.

  It is possible to recover the delay by discarding silence from the voice of each client received by the client. However, when mixing by the server, the downlink voice received by the client is one voice data mixed by the server. It becomes. In that case, if any one of the clients is speaking, the mixed voice is not silent and is not discarded, and the delay cannot be recovered.

  For this reason, when downlink audio is mixed and transmitted to a client by a server, when the downlink communication between the client and the server fluctuates, the audio that has already been transmitted to other clients is transmitted to the client. Therefore, a state in which all the audio received thereafter is delayed is maintained.

  As described above, when the delay increases, specifically, the following problems occur. If a remote conference is performed at multiple locations, audio delays accumulate over time, which may hinder the progress of the conference. For example, when the other party thinks that he / she is not speaking and there is a possibility that the voice is duplicated and cannot be heard. In addition, there is a possibility that the conversation does not hold, for example, the question is asked again because the answer to the question is not received.

  The communication system in Patent Document 1 has a system configuration in which audio data is transmitted from a transmission device to a reception device, and the transmission side device generates silence state information based on the volume level of the audio data, and the transmission device A jitter absorption buffer that accumulates data and adjusts the delay time, and when the amount of audio data stored in the jitter absorption buffer exceeds the set allowable storage amount, the allowable storage amount is increased and the jitter absorption buffer A jitter buffer adjustment unit that discards audio data indicating a silent state and restores the permissible accumulated amount to the default value when the accumulated audio data amount is within the set allowable accumulated amount for a certain period of time. It is.

JP2012-124689

  However, the technique described in Patent Document 1 is limited to a case where the transmission terminal and the reception terminal are in a one-to-one relationship. For example, the relationship is a one-to-many relationship as in a conference system. Audio data is not directly sent from one client to another client. For example, it is not possible to transmit audio data to a client terminal after mixing audio data of another client terminal in a conference server.

  In view of the above problems, an object of the present invention is when audio data input at a plurality of client terminals is transmitted to a server, mixed at the server, and then transmitted as audio data that has been mixed to another client terminal. In other words, a technology for smoothly recovering the delay of voice is provided.

  The present invention provides a client terminal that transmits input voice data to an information processing apparatus in a predetermined unit, the information processing apparatus connectable via a network, the voice data received from the client terminal, All transmitted / received audio data to / from the information processing apparatus with the transmitted terminal information indicating whether or not audio data has already been transmitted to any other client terminal as a queue corresponding to the client terminal Voice data in a queue corresponding to the client terminal other than the client terminal that transmits voice data among the queues stored in the mixing voice storage means, Mixes generated to be sent to the client terminal as mixing audio data And transmitted terminal information recording means for describing in the transmitted terminal information that the voice data in the queue is used to generate mixed voice data for transmitting to the client terminal in the mixing means. And a downlink delay recovery processing determination unit that determines whether or not to perform downlink audio delay recovery processing for the queue in the mixing audio storage unit when mixing is performed in the mixing unit based on a predetermined condition; Silence data determination means for determining whether or not the voice data is silence data that is considered to contain no voice based on a condition, and the mixing means transmits the mixing voice to the client terminal When generating data, the downstream delay recovery processing determination means uses the mixing sound. If it is determined to perform the downstream audio delay recovery process for the queue in the storage means, the silent data is sent to the untransmitted audio data extracted from the queue corresponding to the other client terminal in order to generate mixing audio data. If it is determined by the determination means that the data is silence data, the process is not performed on the voice data but the process is performed on the next voice data.

  According to the present invention, input of audio data in a plurality of client terminals is transmitted to a server, mixed in the server, and then transmitted as audio data already mixed to other client terminals. It became possible to recover.

It is an example of the figure which shows the system configuration | structure which concerns on embodiment of this invention. It is an example of the figure which shows the structure of the hardware which concerns on embodiment of this invention. It is an example of the figure which shows the function structure of the software which concerns on embodiment of this invention. It is a figure which shows an example of the image which shows generation | occurrence | production of a reproduction delay and recovery processing in the client terminal which concerns on embodiment of this invention. It is a figure which shows an example of the image which shows the condition where the audio | voice information is stored in a queue in the conference server which concerns on embodiment of this invention. It is a figure which shows an example of the image which shows generation | occurrence | production of a delay state based on the reception state from a client terminal, and a recovery process in the conference server which concerns on embodiment of this invention. It is a figure which shows an example of the image which shows generation | occurrence | production of a delay state based on the transmission state to a client terminal, and a recovery process in the conference server which concerns on embodiment of this invention. It is an example of the flowchart of the reproduction | regeneration delay recovery process in the client terminal which concerns on embodiment of this invention. It is an example of the flowchart of the delay recovery process based on the reception state from the client terminal in the conference server which concerns on embodiment of this invention. It is an example of the flowchart of the delay recovery process based on the transmission state to the client terminal which concerns on embodiment of this invention. FIG. 8 is an example of a diagram illustrating a delay recovery method when a delay occurs in FIG. 7 using a queue image.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is an example of a diagram showing a system configuration according to an embodiment of the present invention. The client terminals 101 (multiple) and the conference server 102 can be connected via a network.

  In the conference system (FIG. 1), one of the users participating in the conference (for convenience, the organizer) accesses the conference server 102 from the client terminal 101 and reserves the conference room. The conference room is a possible conference space, and only “invited participants” described later can enter the room. In addition, as a free space, an unspecified user can participate or cannot speak, but only viewing is possible.

  The organizer determines the conference ID (or conference room ID, room number, etc.) for identifying the conference, the time to use the conference room, etc., and invites a specific participant to be registered in the conference server. Participant's notification destination (for example, e-mail address) may be used to call for participation. In order to allow only a specific user to participate, a password for entering the conference room can be described in the notification.

  Here, audio data from a microphone (not shown) of a user (actual conference participant) of each client terminal 101a to 101c or imaging data from a camera (not shown) is shown as an image in FIGS. Thus, it is once transmitted to the conference server 102. For example, audio is transmitted from each client terminal 101 to the conference server 102 as uplink audio for the number of client terminals 101.

  The above uplink voice is mixed by the conference server 102 with the uplink voice received from the plurality of client terminals 101 and distributed to the other client terminals 101 (downlink voice data). However, it is not necessary to mix the voice of the terminal itself with the downlink voice data sent back to the client terminal 101 that transmitted a certain uplink voice during mixing.

  Here, although the client terminal 101 and the conference server 102 are described as separate cases, one client terminal 101 may be configured to have the functions of the conference server 102 in the same case.

  In the embodiment of the present invention, the description will be given by taking the conference system as an example. However, the present invention is not necessarily limited to the conference system. For example, voice exchange is performed via a network, and packets (data including voice data are ) Provided as a usable technique when a delay occurs.

  FIG. 2 is an example of a diagram illustrating a hardware configuration according to the embodiment of the present invention. As shown in FIG. 2, the client terminal 101 and the conference server 102 are connected via a system bus 204 to a CPU (Central Processing Unit) 201, a RAM (Random Access Memory) 202, a ROM (Read Only Memory) 203, an input controller 205, A configuration is adopted in which a video controller 206, a memory controller 207, a communication I / F controller 208, and the like are connected. The CPU 201 comprehensively controls each device and controller connected to the system bus 204.

  Further, the ROM 203 or the external memory 211 will be described later, which is necessary for realizing the functions executed by each server or each PC, such as BIOS (Basic Input / Output System) and OS (Operating System) which are control programs of the CPU 201. Various programs are stored. Further, information necessary for carrying out the present invention is stored. The external memory may be a database.

  The RAM 202 functions as a main memory, work area, and the like for the CPU 201. The CPU 201 implements various operations by loading a program or the like necessary for executing the processing from the ROM 203 or the external memory 211 to the RAM 202 and executing the loaded program.

  The input controller 205 controls input from a keyboard (KB) 209 or a pointing device such as a mouse (not shown).

  The video controller 206 controls display on a display device such as the display 210. The display device may be a display device such as a liquid crystal display. These are used by the administrator as needed.

  The memory controller 207 is an external storage device (hard disk (HD)), flexible disk (FD), or PCMCIA (Personal Computer) that stores a boot program, various applications, font data, user files, editing files, various data, and the like. Controls access to an external memory 211 such as a Compact Flash (registered trademark) memory connected to a Memory Card International Association (Card Memory) card slot via an adapter.

  The communication I / F controller 208 connects and communicates with an external device via a network, and executes communication control processing on the network. For example, communication using TCP / IP (Transmission Control Protocol / Internet Protocol) is possible.

  Note that the CPU 201 can display on the display 210 by executing an outline font rasterization process on a display information area in the RAM 202, for example. Further, the CPU 201 enables a user instruction using a mouse cursor (not shown) on the display 210.

  Various programs to be described later for realizing the present invention are recorded in the external memory 211 and executed by the CPU 201 by being loaded into the RAM 202 as necessary. Furthermore, definition files and various information tables used when executing the program are also stored in the external memory 211, and a detailed description thereof will be described later.

  FIG. 3 is an example of a diagram showing a functional configuration of software according to the embodiment of the present invention. The software components of the client terminal 101 and the conference server 102, each storage unit, and data exchange between them (excluding the dotted arrows associated with the audio data 328 and the like from the terminal 3) are shown in the figure.

  First, an outline of the flow of audio data will be described. When the client terminal 101 receives voice input from the user, it is converted into voice data and transmitted to the conference server 102. The audio data received by the conference server from the plurality of client terminals 101 is stored in a queue prepared for each client terminal 101. Audio data transmitted to a certain client terminal 101 is obtained by mixing audio data one by one from a queue corresponding to another client terminal 101 except for the audio data of the client terminal 101 itself (corresponding to the queue). To the client terminal 101. The client terminal 101 that has received the mixed voice data reproduces the mixed voice data of the other client terminal 101 (the voice data of the client terminal 101 itself is not mixed), so that the user listens to the voice. I can do it.

  In the audio input unit 311, the client terminal 101 receives input of audio data through a connected device such as a microphone used by the user. The audio data that has received the input is transmitted from the audio transmission unit 312 to the conference server 102. Here, the voice data is divided into a certain size (for example, voice inputted for 10 milliseconds with respect to time). In the following description, the term “voice data” refers to input voice data divided into a certain size. This is merely an example, and it is assumed that the conference system standard such as the number of data bits is followed.

  The audio data (upstream audio data) transmitted from the client terminal 101 is received by the audio receiving unit 321 of the conference server 102. The received audio data is stored in the mixing audio storage unit 326. The mixing audio storage unit 326 prepares a queue corresponding to each client terminal 101 participating in the conference, and stores a certain number of audio data. In FIG. 3, in the mixing audio storage unit 326, each element of the queue corresponding to each client terminal 101 is arranged in a vertical direction (seven rectangles in the example) corresponding to the terminals 1 to 3. Show.

  At the time of audio reception, if the uplink check unit 322 determines that processing delay has occurred due to reception (acquisition) of uplink audio data, delay recovery processing is performed. Specifically, this processing is performed for each received client terminal 101, and the number of audio data stored in the corresponding queue exceeds a predetermined number based on the upstream delay threshold of the server threshold storage unit 327. If it is determined that there is, the silence data is deleted.

  Here, one unit of audio data stored in each queue corresponds to one rectangle, and more specifically, a pair of audio data 328 and transmitted terminal information 329 is stored. The audio data 328 is audio data received from the client terminal 101 described above. On the other hand, the received audio data is mixed with audio data received from other terminals and transmitted from the conference server 102 to each client terminal 101 (mixed audio transmission unit 323). At that time, a transmitted client terminal 101 and an untransmitted client terminal 101 are generated due to some state (for example, a difference in network line speed). Therefore, information on whether the client terminal 101 has already been transmitted or the client terminal 101 that has not been transmitted is described in “transmitted terminal information 329” so that a delay recovery process described later can be performed for each client terminal 101. Any data structure may be used as long as the transmitted / untransmitted client terminal 101 can be identified.

  When mixing audio data to be transmitted to a certain client terminal 101, the audio data originally received from the client terminal 101 is omitted from the object to be mixed. That is, the audio data sent from the client terminal 101 other than itself is mixed.

  The downlink check unit 324 performs a delay recovery process on the client terminal 101 to be mixed (that is, the client terminal 101 other than the client terminal 101 to be transmitted) when the mixing audio transmission unit 323 transmits the audio data to the client terminal 101. . Specifically, this processing is performed when mixing the silence data in each queue when the downstream delay flag (existing for each client terminal 101) of the server threshold value storage unit 327 is “ON”. Skip and mix the next audio data.

  After completion of the delay recovery process by the downlink check unit 324 (including a case where it is determined that there is no need for delay recovery), the mixed data is transmitted to a predetermined client terminal 101. In the process of the conference server 102, the audio reception unit 321 and the mixing audio transmission unit 323 do not need to be synchronized processes.

  In the mixing voice receiving unit 313 of the client terminal 101, the conference server 102 receives data obtained by mixing voice data other than that of the client terminal 101 and stores it in the received voice storage unit 316 (queue). At that time, if it is determined that the number exceeds a predetermined number based on the reproduction delay threshold value of the terminal threshold value storage unit 317, the silence data is deleted.

  The audio reproduction unit 315 takes out the mixed audio from the received audio storage unit 316 (queue) and reproduces it. In the processing of the client terminal 101, the mixing audio reception unit 313 and the audio reproduction unit 315 do not have to be synchronized processing.

  The audio delay recovery process is executed by both the client terminal 101 and the conference server 102. An outline of each delay recovery process will be described with reference to FIG. 4 (client terminal 101 side) and FIGS. 5 to 7 (conference server 102 side) using an image of a queue storing audio data.

  FIG. 4 is a diagram showing an example of an image showing the occurrence of reproduction delay and recovery processing in the client terminal according to the embodiment of the present invention. In FIG. 4, 1) a state where no delay has occurred, 2) a state where a delay has occurred, and 3) a delay recovery method will be described. Note that all the queues according to the embodiment of the present invention described with reference to FIGS. 4 to 7 are described as FIFOs. First, “a state in which no delay occurs” will be described with reference to 1).

(1) First, the client terminal 101 receives audio data (mixed audio data) from the conference server 102.
(2) The audio data is stored in the received audio storage unit 316 as data in a waiting state for reproduction.

  (3) In the example of FIG. 4, only the received “A1” is stored. However, it may be regarded as a delay if the number is within a desired range based on the reproduction delay threshold. The queue is a FIFO, and the previously stored audio data is first extracted and reproduced.

  (4) The voice data at the head of the queue is taken out and played back by the voice playback unit 315, so that the user of the client terminal 101 can hear it.

  In the following description of FIG. 4, the flow of (1) to (4) is the same although there is a difference in whether or not a delay occurs.

  Next, the “state in which delay occurs” will be described with reference to 2). For example, it is assumed that a plurality of packets (represented by the same symbols as the voice data for convenience. In the example, seven packets A1 to A7) are received at the client terminal 101 almost simultaneously due to network delay or the like. Seven audio data are accumulated in the queue (received audio storage unit 316) of the client terminal 101, and reproduction of the audio data is delayed.

  In the subsequent processing, since the audio data that is deleted from the queue for playback and the audio data that is received and stored in the queue are in principle the same timing, the delay speed is constant (naturally there is a delay). It will never recover).

  Therefore, if another delay occurs for some reason, the total number of the delayed audio data is gradually accumulated, and finally reaches a level at which the user feels uncomfortable with the conference call. That is, when images captured by the camera are transmitted / received separately, it is conceivable that the speech image and sound are significantly different from each other, and the context of speech content with other client terminals 101 is incorrect.

  Therefore, as described above, based on the reproduction delay threshold value, if the number is within a desired range, it is regarded as a delay. However, when the predetermined range is exceeded, it is necessary to perform a delay recovery process during reproduction.

  The “delay recovery method” will be described with reference to 3) at the end of FIG. That is, based on the reproduction delay threshold value, delay recovery processing is performed assuming that the number of desired ranges has been exceeded. The audio data in the queue stores the audio data of A1 to A7. Of these, the audio data to which “()” is given (A2, A4, A5, A6 are described as (A2), for example) ) Is “silent data”. Here, the silent data is “voice data that is regarded as not including voice” in the system. The determination as to whether or not the audio data is silent data is a well-known technique disclosed in “Japanese Patent Laid-Open No. 2000-31223” and will not be described.

  For audio data reproduction, audio data is acquired one by one from the head of the queue and reproduced. The number of audio data stored in the queue is counted at the time of acquisition, and this number is set as a reproduction delay threshold. Based on this, when the number of desired ranges is exceeded, delay recovery processing is started. In that case, all audio data stored in the queue is confirmed, and the silence data is deleted. The delay recovery process is performed at a time without being interrupted by other processes.

FIG. 5 is a diagram showing an example of an image showing a situation in which audio information is stored in a queue in the conference server according to the embodiment of the present invention. FIG. 5 shows a state when no delay occurs.
(1) First, the conference server 102 receives audio data for each client terminal 101.

  (2) Next, the received audio data is stored in a queue prepared for each client terminal 101 in the mixing audio storage unit 326. The information to be stored is not only the audio data 328 as described with reference to FIG. 3, but also a transmission that stores for each other client terminal 101 whether or not the other client terminal 101 has already mixed and transmitted the audio data. Stored terminal information 329 is also stored. Both (328, 329) are collectively referred to as audio information for convenience. In the initial state immediately after being placed in the queue, it is not transmitted to any other client terminal 101. The information indicating whether or not the transmission has been completed may be a flag prepared with an array of all other client terminals 101, or a list of terminals that have already been transmitted or a list of terminals that have not been transmitted. Any data structure may be used.

(3) Next, the audio data of each queue (audio data included in the audio information at the head of the queue) is extracted and mixed. As described above, the audio data of other queues are mixed except for the queue corresponding to the client terminal 101 to which the audio data is to be transmitted.
(4) The mixed audio data is transmitted to the client terminal 101.

  FIG. 6 is a diagram showing an example of an image showing the generation and recovery processing of the delay state based on the reception state from the client terminal in the conference server according to the embodiment of the present invention. In FIG. 6, first, “1) a situation in which delay occurs due to acquisition of uplink voice data (acquisition of voice data transmitted from the client terminal 101 to the conference server 102)” will be described.

  (1) First, it is assumed that audio data is received from the client terminal 101A without delay. On the other hand, seven audio data (B1 to B7) are received from the client terminal 101B due to a network problem or the like.

  (2) Accordingly, one audio information is stored in the queue corresponding to the client terminal 101A, but seven audio information is stored in the corresponding queue in the client terminal 101B.

  (3) Mixing for transmission to the client terminal 101C uses audio data extracted from the queues for the client terminal 101A and B. From the queue for the client terminal 101A, the voice data “A7” (voice having the same time as “B7”) is acquired and used, but from the queue for the client terminal 101B, the voice data “B1” (from “B7”) is used. (Sound of 6 unit time ago) remains as delay data. Accordingly, mixing audio data of “A7 + B1” with different time is generated.

  As in the case of reproduction in the client terminal 101 described above, the audio information that is deleted from the queue for mixing and the audio information that is received and stored in the queue are in principle the same number, so the delay speed is constant. (The delay will not recover naturally). Accordingly, when a delay occurs only for the audio data received from the client terminal 101B for some reason, the total number of the delayed audio data is gradually accumulated.

  (4) The audio data that is finally transmitted (mixed) to the client terminal 101 reaches a level that is determined to be a problem by the user. That is, the audio data of the client terminals 101 other than the client terminal 101 that has received the mixed audio data is being mixed, but there is a possibility that the conversation may not be established due to a time lag.

  Next, “2) Delay recovery method” for the above-described delay will be described. As an example, it is assumed that audio information exceeding a predetermined number of ranges is accumulated in the queue received from the client terminal 101B as in 1) (determination based on the uplink delay threshold). When it is determined that audio information exceeding a predetermined number of ranges has been accumulated, delay recovery processing is started. In the queue on the left side of the figure “2)”, there are four pieces of voice information (those assigned “()”) corresponding to silence data among B1 to B7, and these are deleted. Whether the data is silent data is determined by the client terminal 101 and added to the audio data to be transmitted. The conference server 102 does not analyze the inside of the actual voice data, but determines whether or not the data is silence based on the additional information of the voice data transmitted from the client terminal 101. However, the conference server 102 may determine whether the data is silence data. Note that the conference server may determine whether or not the audio data is silent even in the audio data downlink processing (audio data transmission to the client terminal 101). The rest is stored as three (B1, B3, B7) audio information as shown on the right queue. These are executed for every queue (prepared for each client terminal 101). The delay recovery process is performed at a time without being interrupted by other processes.

FIG. 7 is a diagram showing an example of an image showing the generation and recovery processing of the delay state based on the transmission state to the client terminal in the conference server according to the embodiment of the present invention. In FIG. 7, first, “3) a situation in which delay occurs due to downlink audio data acquisition (mixed audio data transmission from the conference server 102 to the client terminal 101)” will be described.
(1) The conference server 102 receives audio data from both the client terminals 101A and 101B.

  (2) Voice information (information in which voice data and transmitted terminal information are associated) is stored in each corresponding queue. In 3) of FIG. 7, seven pieces of audio information are stored.

  (3) Extract audio data from each queue for each client terminal 101. In order to mix the audio data to be sent to each client terminal 101, all audio data of the other client terminals 101 other than itself are extracted for each client terminal 101.

(4) Next, the mixed audio data is transmitted to the client terminal 101C. At that time, a delay may occur depending on the state of the network. On the other hand, it is assumed that the voice transmitted to the client terminal 101B (voice data obtained by mixing the voice data of the client terminal 101A and the client terminal 101C) can be transmitted without delay. In this case, the voice of the client terminal 101A that has been transmitted to the client terminal 101B but not transmitted to the client terminal 101C exists in the queue. If there is at least one untransmitted client terminal 101 in the transmitted terminal information 329 of the voice information, the voice information stored in the queue cannot be deleted. Even if the data is not deleted, the data is ignored in the mixing / transmission in the client terminal 101B, but cannot be ignored in the client terminal 101C. Therefore, the audio reproduced at the client terminal 101A and the audio reproduced at the client terminal 101C It is also assumed that time intervals have accumulated and the temporal context of the conversation gradually becomes unnatural.
Next, "4) Delay recovery method" for the above-described delay will be described with reference to FIG.

  FIG. 11 is an example of a diagram illustrating a delay recovery method in the case where a delay occurs in FIG. 7 using a queue image.

As an example, when a delay occurs in the voice transmitted to the client terminal 101C (determined by the downlink delay flag), the silence data is skipped when the voices of the client terminal 101A and the client terminal 101B are taken out from the queue and mixed. Delay is recovered by mixing non-silent audio data. Note that if there is no untransmitted terminal in the skipped silence data, it is deleted from the queue. In the queue of “4)”, there are four pieces of audio information (with “()” added) corresponding to silent data among B1 to B7, and these are skipped. These are executed for every queue (prepared for each client terminal 101). The delay recovery process is performed at a time without being interrupted by other processes.

  The flowcharts of FIGS. 8 to 10 explain the delay recovery processing for the delay in the client terminal 101 or the conference server 102.

FIG. 8 is an example of a flowchart of the reproduction delay recovery process in the client terminal according to the embodiment of the present invention. Each step (S801 to S813) in the flowchart of FIG. 8 is executed by the CPU 201 of the client terminal 101, and steps S814 to S815 are executed by the CPU 201 of the conference server 102. The flowchart of FIG. 8 corresponds to the image of the queue of FIG.
In step S801, mixed audio data transmitted from the conference server is received.

In S802, the received mixed voice data is stored in the queue of the received voice storage unit 316 of the client terminal 101 (enqueue).
In step S803, the number of audio data stored in the queue is counted.

  In step S804, it is determined whether audio data exceeding a predetermined number of ranges is accumulated in the queue (determination based on a reproduction delay threshold).

  In S805, the process branches based on the determination as to whether or not a predetermined number of out-of-range audio data has been accumulated. When the predetermined number is within the range, the process proceeds to NO and proceeds to S810. If the predetermined number is out of the range, the process proceeds to YES and proceeds to S806 in order to execute the <silent discard> routine.

The processes in S806 to S809 are performed at a time without being interrupted by other processes. Here, all of the audio data in the queue of the client terminal 101 is checked.
In S806, one audio data at the head of the queue is dequeued (taken out).
In S807, it is checked whether or not the extracted audio data is silence data.

  If it is determined in S808 that the checked audio data is silent, the process returns to S806 to check the next audio data as it is (without returning the audio data to the queue). That is, audio data determined to be silent is discarded. If it is determined that the checked audio data is not silent, the audio data (taken from the queue in S806) is enqueued (stored) again in S809.

  As described above, the queue is a FIFO, and the processing of S806 to S809 is performed once for all audio data without being interrupted by other processing, so that all the silent data is deleted. Even after a change from the left cue described in 4-3) to the right cue, the temporal order of the remaining audio data is ensured.

  As described above, when a delay occurs in the queue of the client terminal 101 that has received the mixed audio data in the received audio storage unit 316, the delay recovery process is performed.

  Next, in S810, it is counted again whether audio data exceeding a predetermined number of ranges is accumulated in the queue.

  In S811, it is determined whether audio data exceeding a predetermined number of ranges has been accumulated (may be the same value as the reproduction delay threshold value or a different value). That is, it is determined whether or not a sufficient effect has been obtained as a result of completing the delay recovery process.

  In S812, if audio data exceeding a predetermined number of ranges is accumulated (in the case of YES), the process proceeds to S813. If not (NO), the process returns to S801 (sound reception). Note that the audio reproduction is performed by an asynchronous process and is not related to the delay recovery process, and thus is not shown in the flowchart.

  In step S813, the conference server 102 is notified of delay recovery processing. That is, the method of deleting silent data cannot be further handled by the client terminal 101. Therefore, in this case, the conference server side is requested to support for delay recovery. Further, after notifying the conference server 102 of the delay recovery process, the process returns to S801. On the other hand, in S814 of the conference server 102, the notification from S813 of the client terminal 101 is received.

In step S815, the “downlink delay flag” is turned on to indicate that there is a request for support for recovery support processing from the client terminal 101. As a result, the “downbound voice delay recovery” process operating in the conference server 102 may be activated. However, “downlink voice delay recovery” may be activated by a determination in an original routine regardless of whether there is a request from the client terminal 101.
This completes the description of the flowchart relating to the delay recovery processing in the client terminal 101.

  FIG. 9 is an example of a flowchart of delay recovery processing based on the reception state from the client terminal in the conference server according to the embodiment of the present invention. Each step of the flowchart of FIG. 9 is executed by the CPU 201 of the conference server 102. The following processing is performed for each individual queue prepared for each client terminal 101 in the conference server 102. Further, since the processing described with reference to the flowchart of FIG. 9 is audio data received by the conference server 102 from the client terminal 101, the audio delay recovery processing relating to uplink audio data is described. The flowchart in FIG. 9 corresponds to the images in FIGS. 5 and 6.

  In step S <b> 901, voice data (more precisely, a communication packet including voice data) is received from the client terminal 101. Here, only one flowchart is described, but specifically, even if one receiving unit accepts the data and determines which of the plurality of client terminals 101 transmits the data. In the first place, when the connection between the conference server 102 and the client terminal 101 is established, a receiving unit may be generated in multiple threads for each “client terminal 101” and may be used as a receiving unit for a specific client terminal 101. However, in S902 and after, it is assumed that the client terminal 101 is identified by the receiving unit, and the processing corresponds to one identified client terminal 101.

  In step S902, the voice data received in step S901 is already stored in the queue corresponding to the identified client terminal 101 in the mixing voice storage unit 326 in the voice information (the voice data in the packet and the transmitted terminal described above). Check how many pieces of information 329) are stored.

  In S903, it is determined whether or not the number of the audio information exceeds a predetermined number according to the upstream delay threshold value of the server threshold value storage unit 327.

  In S904, when it is determined that the number of audio information in the queue exceeds the predetermined number according to the uplink delay threshold (in the case of YES), the process proceeds to S905. If it is determined that it has not exceeded (NO), the process proceeds to S909.

  In S909, it is determined that the number of audio information stored in the queue does not exceed the predetermined number according to the uplink delay threshold, that is, it is determined that the audio information can still be stored (without performing delay recovery processing). (Enqueue) and return to S901 for the next voice reception.

  When the processing proceeds to S905, the processing from S905 to S908 is repeated for all the audio information stored in the queue. In the meantime, the loop from S905 to S908 does not come off.

  If it is determined that the predetermined number is exceeded according to the upstream delay threshold, one piece of audio information is extracted (dequeued, taken out from the queue) from the queue in S905.

  In S906, it is determined whether or not the audio data included in the audio information extracted in S905 is “silence data”. In S907, when the audio data included in the audio information extracted in S905 is “silent data” (in the case of YES), the process returns to S905. That is, since the dequeued audio information is silent data, it is not returned (deleted) to the queue.

  In S908, since the dequeued audio information is not silence data, it is enqueued (returned to the end of the queue).

  By performing the processing from S905 to S908, the audio information is reduced by the number of silence data in the queue, so that an effect of reducing the audio delay can be obtained. This completes the description of the flowchart in FIG. 9.

  FIG. 10 is an example of a flowchart of delay recovery processing based on the transmission state to the client terminal according to the embodiment of the present invention. Each step of the flowchart of FIG. 10 is executed by the CPU 201 of the conference server 102. The flowchart of FIG. 10 is a process of mixing the audio information (audio data included) in the queues of the plurality of client terminals 101 in the mixing audio storage unit 326 of the conference server 102 and transmitting the audio information to the client terminal 101. Since it is transmitted from the conference server 102 to the client terminal 101, it is uplink voice data. This corresponds to the image of the queue in FIG.

The voice data of the terminal itself is not mixed with the mixed voice data transmitted to a certain client terminal 101. Accordingly, the queues of the mixing audio storage unit 326 processed in the flowchart of FIG. 10 are processed for each queue corresponding to a queue other than the client terminal 101 to be transmitted.
In S1001, one piece of audio information is extracted from the queue (dequeue).
In S1002, the downlink delay flag that is turned “ON” in S815 of FIG. 8 is checked.

  In S1003, when the downlink delay flag is “ON” (in the case of YES), the process proceeds to S1004. In other words, this is because a delay recovery process on the conference server 102 side is requested from the client terminal 101 side. If it is “OFF” (NO), the process proceeds to S1007.

  In S1002, the threshold may be used in the same manner as the check at the time of uplink (uplink delay threshold in the server threshold value storage unit 327) instead of checking the downlink delay flag. This threshold value is set as a downstream delay threshold value, and when the number of audio information in the queue reaches a predetermined number based on the downstream audio threshold value, the process proceeds to S1004 (YES direction) in S1003, and if not, the process proceeds to S1007. .

  In S1004, it is checked whether or not the audio data included in the audio information dequeued in S1001 is silence data.

  In S1005, when the audio data checked in S1004 is “silent data” (in the case of YES), the process proceeds to S1006, the next audio information is taken out from the queue, and the process returns to S1004 to repeat the process. By repeating this process, the process of ignoring “silent data” proceeds.

  On the other hand, if the sound data checked in S1004 is not “silent data” (NO), the process proceeds to S1007, and is transmitted to the client terminal 101 as mixed sound in S1008 as described above. In S1007, the audio data used for mixing is not returned to the queue as unnecessary data. However, in actuality, since it may be necessary to use it in another client terminal 101, “all client terminals (except for the client terminal that has transmitted this audio data) have not yet transmitted this audio data. In this case, it is necessary to enqueue again. Above, description of the flowchart of FIG. 10 is completed.

  It should be noted that the configuration and contents of the various data described above are not limited to this, and it goes without saying that the various data and configurations are configured according to the application and purpose.

  Although one embodiment has been described above, the present invention can take an embodiment as, for example, a system, apparatus, method, program, or recording medium, and specifically includes a plurality of devices. The present invention may be applied to a system including a single device.

  Further, the program according to the present invention is a program capable of executing the processing method of the flowcharts of FIGS. 8 to 10, and the storage medium of the present invention can execute the processing method of the flowcharts of FIGS. 8 to 10. The program is stored. The program in the present invention may be a program for each processing method of each device in the flowcharts of FIGS.

  As described above, a recording medium that records a program that implements the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus stores the program stored in the recording medium. It goes without saying that the object of the present invention can also be achieved by executing the reading.

  In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium storing the program constitutes the present invention.

  As a recording medium for supplying a computer program, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a DVD-ROM, a magnetic tape, a nonvolatile memory card, a ROM, an EEPROM, Silicon disks, solid state drives, etc. can be used.

  Further, by executing the program read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on an instruction of the program is actually It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the processing and the processing is included.

  Furthermore, after the program read from the recording medium is written to the memory provided in the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function expansion board is based on the instructions of the program code. It goes without saying that the case where the CPU or the like provided in the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

Further, the present invention may be applied to a system composed of a plurality of devices or an apparatus composed of a single device. Needless to say, the present invention can be applied to a case where the present invention is achieved by supplying a program to a system or apparatus. In this case, by reading a recording medium storing a program for achieving the present invention into the system or apparatus, the system or apparatus can enjoy the effects of the present invention. Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or apparatus can enjoy the effects of the present invention.
In addition, all the structures which combined each embodiment mentioned above and its modification are also included in this invention.

101 Client terminal 102 Conference server 103 Network 311 Audio input unit 312 Audio transmission unit 313 Mixing audio reception unit 314 Reception audio check unit 315 Audio reproduction unit 316 Reception audio storage unit 317 Terminal threshold storage unit 321 Audio reception unit 322 Uplink check unit 323 Mixing Audio transmission unit 324 Down check unit 325 Mixing unit 326 Mixing audio storage unit 327 Server threshold storage unit 328 Audio data 329 Transmitted terminal information

Claims (10)

A client terminal that transmits input audio data to the information processing apparatus in a predetermined unit; and the information processing apparatus connectable via a network,
Audio data received from the client terminal, and transmitted terminal information indicating that the audio data has already been transmitted to any other client terminal as a queue corresponding to the client terminal, Audio storage means for mixing for storing queues of all client terminals that transmit and receive audio data;
Mixing means for generating, in the queue stored in the mixing voice storage means, voice data in a queue corresponding to the client terminal other than the client terminal that sends voice data to be sent to the client terminal as mixing voice data When,
In the mixing means, the transmitted terminal information recording means for describing in the transmitted terminal information that the voice data in the queue is used to generate mixing voice data for transmitting to the client terminal;
A downlink delay recovery processing determination unit that determines whether or not to perform a downlink audio delay recovery process for a queue in the mixing audio storage unit when performing mixing in the mixing unit based on a predetermined condition;
Silence data determination means for determining whether or not the sound data is silence data that is regarded as not including sound, based on a predetermined condition;
With
The mixing means, when generating the mixing voice data to be transmitted to the client terminal, when the downlink delay recovery processing determination means determines that the downlink voice delay recovery processing for the queue in the mixing voice storage means is performed, If the unsent audio data taken out from the queue corresponding to the other client terminal to generate mixing audio data is determined to be silence data by the silence data determination means, An information processing apparatus characterized by proceeding to the next audio data without processing.   The predetermined condition in the downlink delay recovery processing determination means is a downlink delay indicating whether or not a notification is received from the client terminal to perform a downlink voice delay recovery process for a queue in the mixing voice storage means in the information processing apparatus. The information processing apparatus according to claim 1, wherein the flag is set to a value indicating that downlink voice delay recovery processing for the queue in the mixing voice storage unit is to be performed.   The predetermined condition in the downlink delay recovery processing determination means is that the mixing is performed when the number of audio data in a queue used for generating the mixed audio data exceeds a predetermined range based on a downlink delay threshold. 2. The information processing apparatus according to claim 1, wherein the information processing apparatus determines that the downstream audio delay recovery process for the queue in the audio storage means is to be performed. A queue corresponding to the client terminal in the mixing voice storage means, wherein the voice data received from the client terminal and the transmitted terminal information indicating that the voice data has been transmitted to any other client terminal are identifiable. Voice registration means for mixing to be stored in
When it is determined that the number of audio data stored in the queue corresponding to the client terminal exceeds a predetermined range based on the uplink delay threshold stored in the server threshold storage unit, the silence data in the queue , Server silent data deleting means for deleting transmitted terminal information corresponding to the silent data from the queue;
The information processing apparatus according to claim 1, further comprising:   The predetermined condition in the silence data determination means is a result of determining whether or not the voice data is silence data in the client terminal as additional information of the voice data, and the silence data determination means The information processing apparatus according to any one of claims 1 to 4, wherein the information processing unit determines whether or not the sound data is silent based on the additional information. An information processing system in which a client terminal that transmits input audio data to an information processing device in a predetermined unit and an information processing device can be connected via a network,
The information processing apparatus includes:
Audio data received from the client terminal, and transmitted terminal information indicating that the audio data has already been transmitted to any other client terminal as a queue corresponding to the client terminal, Audio storage means for mixing for storing queues of all client terminals that transmit and receive audio data;
Mixing means for generating, in the queue stored in the mixing voice storage means, voice data in a queue corresponding to the client terminal other than the client terminal that sends voice data to be sent to the client terminal as mixing voice data When,
In the mixing means, the transmitted terminal information recording means for describing in the transmitted terminal information that the voice data in the queue is used to generate mixing voice data for transmitting to the client terminal;
Mixing audio data transmitting means for transmitting the mixing audio data to the client terminal;
A downlink delay recovery processing determination unit that determines whether or not to perform a downlink audio delay recovery process for a queue in the mixing audio storage unit when performing mixing in the mixing unit based on a predetermined condition;
Silence data determination means for determining whether or not the sound data is sound data silence data that is regarded as not including sound based on a predetermined condition;
With
The client terminal is
Mixing audio data receiving means for receiving the mixing audio data transmitted by the mixing audio data transmitting means;
Received voice registration means for storing the mixed voice data received by the mixed voice data receiving means in a queue of the received voice storage means as reproduced voice data;
When the number of the reproduced audio data stored in the queue of the received audio storage means exceeds a predetermined range based on the first reproduction delay threshold stored in the terminal threshold storage means, the received audio storage A reproduction delay recovery processing means for determining whether or not the reproduced audio data stored in the cue of the means is silent data, and deleting from the queue if the data is silent data;
As a result of the silence data being deleted from the cue by the reproduction delay recovery processing means, the number of the reproduced audio data is further determined based on a second reproduction delay threshold value stored in the terminal threshold value storage means. A downstream voice delay recovery processing requesting means for notifying a request to perform a downstream voice delay recovery process for the queue in the mixing voice storage means in the information processing apparatus,
With
The mixing means, when generating the mixing voice data to be transmitted to the client terminal, when the downlink delay recovery processing determination means determines that the downlink voice delay recovery processing for the queue in the mixing voice storage means is performed, If the sound data taken out from the queue corresponding to the other client terminal to generate mixing sound data is determined to be soundless data by the soundless data determining means, the sound data is processed. An information processing system characterized by proceeding with processing for the next audio data. A method of controlling the information processing apparatus connectable via a network with a client terminal that transmits input audio data to the information processing apparatus in a predetermined unit,
The audio registration unit for mixing receives the audio data received from the client terminal, and transmitted terminal information indicating that the audio data has been transmitted to any other client terminal. A mixing voice registration step for registering voice data in a queue in the mixing voice storage means for storing queues of all client terminals that transmit and receive voice data as a queue corresponding to the terminal;
The mixing unit is to transmit the audio data of the queue corresponding to the client terminal other than the client terminal that transmits the audio data among the queues stored in the mixing audio storage unit to the client terminal as mixing audio data. A mixing step to generate,
The transmitted terminal information recording means describes in the transmitted terminal information that in the mixing step, the voice data in the queue is used to generate mixing voice data for transmitting to the client terminal. A terminal information recording step;
Downlink delay recovery processing determination means for determining whether or not to perform downlink audio delay recovery processing for the queue in the mixing audio storage means when performing the mixing in the mixing step based on a predetermined condition A determination step;
A silent data determining means, based on a predetermined condition, including a silent data determining step for determining whether the voice data is silent data that is regarded as not including voice;
In the mixing step, when generating the mixing voice data to be transmitted to the client terminal, in the downlink delay recovery processing determination step, when it is determined to perform the downlink voice delay recovery processing for the queue in the mixing voice storage unit, If unsent audio data extracted from a queue corresponding to another client terminal to generate mixing audio data is determined to be silence data in the silence data determination step, A control method for an information processing apparatus, wherein the process is performed on the next audio data without performing the process. A program that can be executed by a client terminal that transmits input audio data to the information processing apparatus in a predetermined unit, and the information processing apparatus that can be connected via a network,
The information processing apparatus;
Audio data received from the client terminal, and transmitted terminal information indicating that the audio data has already been transmitted to any other client terminal as a queue corresponding to the client terminal, Mixing voice registration means for registering voice data in a queue in the mixing voice storage means for storing queues of all client terminals that transmit and receive voice data;
Mixing means for generating, in the queue stored in the mixing voice storage means, voice data in a queue corresponding to the client terminal other than the client terminal that sends voice data to be sent to the client terminal as mixing voice data ,
In the mixing means, the transmitted terminal information recording means for describing in the transmitted terminal information that the voice data in the queue is used to generate mixing voice data for transmitting to the client terminal,
A downlink delay recovery process determination unit that determines whether or not to perform a downlink audio delay recovery process for a queue in the mixing audio storage unit when performing the mixing in the mixing unit based on a predetermined condition;
Based on a predetermined condition, the audio data is made to function as silence data determination means for determining whether or not the audio data is silence data that is regarded as not including sound,
The mixing means, when generating the mixing voice data to be transmitted to the client terminal, when the downlink delay recovery processing determination means determines that the downlink voice delay recovery processing for the queue in the mixing voice storage means is performed, If the unsent audio data taken out from the queue corresponding to the other client terminal to generate mixing audio data is determined to be silence data by the silence data determination means, A program characterized by proceeding to the next audio data without processing. A control method of an information processing system connectable via a network with a client terminal that transmits input audio data to an information processing device in a predetermined unit,
The information processing apparatus includes:
The audio registration unit for mixing receives the audio data received from the client terminal, and transmitted terminal information indicating that the audio data has been transmitted to any other client terminal. A mixing voice registration step for registering voice data in a queue in the mixing voice storage means for storing queues of all client terminals that transmit and receive voice data as a queue corresponding to the terminal;
The mixing unit is to transmit the audio data of the queue corresponding to the client terminal other than the client terminal that transmits the audio data among the queues stored in the mixing audio storage unit to the client terminal as mixing audio data. A mixing step to generate,
The transmitted terminal information recording means describes in the transmitted terminal information that in the mixing step, the voice data in the queue is used to generate mixing voice data for transmitting to the client terminal. A terminal information recording step;
A mixing voice data transmitting means for transmitting the mixing voice data to the client terminal;
Downlink delay recovery processing determination means for determining whether or not to perform downlink audio delay recovery processing for the queue in the mixing audio storage means when performing the mixing in the mixing step based on a predetermined condition A determination step;
A silent data determining means, based on a predetermined condition, including a silent data determining step for determining whether the voice data is silent data that is regarded as not including voice;
The client terminal is
A mixing sound data receiving means for receiving the mixing sound data transmitted by the mixing sound data transmitting step;
A reception voice registration step in which the reception voice registration means stores the mixing voice data received in the mixing voice data reception step as reproduction voice data in a queue of the reception voice storage means;
In the reproduction delay recovery processing means, the number of the reproduced audio data stored in the queue of the received audio storage means exceeds a predetermined range based on the first reproduction delay threshold stored in the terminal threshold storage means. In this case, it is determined whether or not the reproduced audio data stored in the queue of the received audio storage means is silence data, and if it is silent data, a reproduction delay recovery processing step of deleting from the queue;
As a result of the deletion of silence data from the queue by the reproduction delay recovery processing step, the downlink audio delay recovery processing request means further stores the second number of the reproduced audio data stored in the terminal threshold storage means. A downlink audio delay recovery process request step for notifying a request to perform a downlink audio delay recovery process for a queue in the mixing audio storage means in the information processing apparatus when a predetermined range is exceeded based on a reproduction delay threshold Including
In the mixing step, when generating the mixing voice data to be transmitted to the client terminal, in the downlink delay recovery processing determination step, when it is determined to perform the downlink voice delay recovery processing for the queue in the mixing voice storage unit, If unsent audio data extracted from a queue corresponding to another client terminal to generate mixing audio data is determined to be silence data in the silence data determination step, A control method for an information processing system, characterized in that the processing is performed on the next audio data without processing. A program that can be executed in an information processing system in which a client terminal that transmits input voice data to an information processing device in a predetermined unit and the information processing device can be connected via a network,
The information processing apparatus;
Audio data received from the client terminal, and transmitted terminal information indicating that the audio data has already been transmitted to any other client terminal as a queue corresponding to the client terminal, Mixing voice registration means for registering voice data in a queue in the mixing voice storage means for storing queues of all client terminals that transmit and receive voice data;
Mixing means for generating, in the queue stored in the mixing voice storage means, voice data in a queue corresponding to the client terminal other than the client terminal that sends voice data to be sent to the client terminal as mixing voice data ,
In the mixing means, the transmitted terminal information recording means for describing in the transmitted terminal information that the voice data in the queue is used to generate mixing voice data for transmitting to the client terminal,
Mixing audio data transmitting means for transmitting the mixing audio data to the client terminal;
A downlink delay recovery process determination unit that determines whether or not to perform a downlink audio delay recovery process for a queue in the mixing audio storage unit when performing the mixing in the mixing unit based on a predetermined condition;
Based on a predetermined condition, the audio data is made to function as silence data determination means for determining whether or not the audio data is silence data that is regarded as not including sound,
The client terminal is
Mixing audio data receiving means for receiving the mixing audio data transmitted by the mixing audio data transmitting means;
Received voice registration means for storing the mixing voice data received by the mixing voice data receiving means in a queue of the received voice storage means as reproduced voice data;
When the number of the reproduced audio data stored in the queue of the received audio storage means exceeds a predetermined range based on the first reproduction delay threshold stored in the terminal threshold storage means, the received audio storage A reproduction delay recovery processing means for determining whether or not the reproduced audio data stored in the cue of the means is silent data, and deleting it from the cue if it is silent data;
As a result of the silence data being deleted from the cue by the reproduction delay recovery processing means, the number of the reproduced audio data is further determined based on a second reproduction delay threshold value stored in the terminal threshold value storage means. If it exceeds, the information processing apparatus functions as a downlink voice delay recovery process request means for notifying a request to perform a downlink voice delay recovery process for the queue in the mixing voice storage means,
The mixing means, when generating the mixing voice data to be transmitted to the client terminal, when the downlink delay recovery processing determination means determines that the downlink voice delay recovery processing for the queue in the mixing voice storage means is performed, If the sound data taken out from the queue corresponding to the other client terminal to generate mixing sound data is determined to be soundless data by the soundless data determining means, the sound data is processed. A program characterized by proceeding with processing for the next audio data.

Images