JP2016092679A - Voice processing unit, program and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a voice processing unit capable of mixing a larger number of voice channels by software at a low cost.SOLUTION: A voice processing unit includes a plurality of voice processing means, each having a plurality of tone signal reception means for receiving the reception sound data based on the reception sound of each time series, first synthesis means for generating first synthesized voice data by synthesizing the reception sound data received by the plurality of tone signal reception means, buffer means for holding the first synthesized voice data, second synthesis means for generating second synthesized voice data by synthesizing the first synthesized voice data, generated in other voice processing means, respectively, and third synthesis means for generating transmission sound data by synthesizing the voice data, obtained by removing the reception sound data related to the transmission destination from the reception sound data received by the plurality of tone signal reception means, for each reception destination.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an audio processing apparatus, a program, and a method. For example, audio mixing of a conference server (for example, an apparatus such as an MCU (Multipoint Control Unit)) that constitutes a conference system that provides a conference environment by connecting multiple points to a network. Applicable to processing.

  Conventionally, in a conference system that provides a conference environment by connecting multiple points to a network, dedicated hardware is usually used for the process of mixing audio between multiple sites.

  By the way, in an apparatus for performing audio mixing processing of a conventional conference system, ITU-TG When a high-compression encoding method such as 729 is used, there is a problem that the processing load on the decoder and encoder increases, and the number of channels that can be mixed is limited. Furthermore, in recent years, from the viewpoint of cost reduction and maintenance management related to network equipment, it is required to implement a mixing function in software on a general-purpose server without using dedicated hardware.

  As a conventional technique for solving such a problem, there is a technique described in Patent Document 1.

  Patent Document 1 describes that a plurality of communication blocks for mixing a plurality of input / output audio data and a central mixer are used to reduce the load of mixing calculation processing and realize multi-point audio mixing. Yes.

Japanese Patent Publication No. 2008-211291

  Generally, a general-purpose server is equipped with a plurality of CPUs, and each CPU is composed of a plurality of CPU cores. However, when the central mixer in the apparatus of Patent Document 1 is mounted on a general-purpose server by software, The processing capability of one CPU core becomes the performance limit of media processing thread processing. For example, in the case of a 12-core (2 CPU × 6 core) server, the number of channels that can be mixed is limited by the processing capacity of 1/12 of the entire CPU.

  Therefore, a voice processing apparatus, program, and method capable of performing software mixing processing on more voice channels at a low cost are desired.

  The sound processing apparatus of the first aspect of the present invention includes (1) a plurality of sound processing means, and (1-1) each of the sound processing means is (1-2) reception based on reception sound for each time series. A plurality of sound signal receiving means for receiving sound data; and (1-3) a first synthesizing means for generating first synthesized sound data by synthesizing the received sound data received by the plurality of sound signal receiving means. , (1-4) buffer means for holding the first synthesized voice data, and (1-5) second synthesized voice by synthesizing the first synthesized voice data generated by each of the other voice processing means. A second synthesizing unit that generates audio data; and (1-6) audio data obtained by excluding reception sound data relating to the transmission destination from reception sound data received by the plurality of sound signal reception units for each transmission destination; Generate transmission sound data synthesized with the second synthesized voice data. And having a third combining means.

  The audio processing program of the second aspect of the present invention is (1) makes a computer function as a plurality of audio processing means, (2) each of the audio processing means is based on (2-1) received sound for each time series. A plurality of sound signal receiving means for receiving the received sound data; and (2-2) a first synthesizing means for generating the first synthesized sound data by synthesizing the received sound data received by the plurality of sound signal receiving means. (2-3) buffer means for holding the first synthesized voice data and (2-4) first synthesized voice data generated by each of the other voice processing means to synthesize the second A second synthesizing unit that generates synthesized audio data; (2-5 audio data obtained by excluding reception sound data related to the transmission destination from reception sound data received by the plurality of sound signal reception units for each transmission destination; The second synthesized voice data is synthesized And having a third synthesizing means for generating a transmitted sound data.

  According to a third aspect of the present invention, there is provided a speech processing method executed by the speech processing apparatus, comprising: (1) a plurality of speech processing means; and (2) each speech processing means comprising a plurality of sound signal receiving means and first synthesis means. , Buffer means, second synthesizing means, and third synthesizing means, (3) each of the sound signal receiving means receives the received sound data based on the received sound for each time series, and (4) The first synthesizing unit synthesizes reception sound data received by the plurality of sound signal receiving units to generate first synthetic audio data. (5) The buffer unit converts the first synthetic audio data into the first synthetic audio data. (6) the second synthesizing unit synthesizes the first synthesized audio data generated by each of the other audio processing units to generate second synthesized audio data, and (7) the above-described The third synthesizing unit receives a plurality of the sound signals for each transmission destination. And generating a transmitted sound data stage obtained by synthesizing the audio data and the second synthesized speech data from the receiving sound data received, excluding the received sound data relating to the destination.

  According to the present invention, it is possible to provide a voice processing apparatus capable of performing software mixing processing on a larger number of voice channels at a low cost.

It is the block diagram shown about the structural example of the media processing thread | sled which operate | moves with the multipoint audio | voice mixing apparatus which concerns on 1st Embodiment. It is the block diagram shown about the hardware constitutions and connection structure of the multipoint audio | voice mixing apparatus which concerns on 1st Embodiment. It is the block diagram shown about the example of the structure of the terminal connected to the multipoint audio | voice mixing apparatus which concerns on 1st Embodiment. It is the block diagram shown about the example of the internal structure of the audio | voice reception process which concerns on 1st Embodiment. It is the block diagram shown about the example of the internal structure of the audio | voice transmission process which concerns on 1st Embodiment. It is explanatory drawing shown about the structural example of the circular buffer which concerns on 1st Embodiment. It is explanatory drawing shown about the example of the internal structure of the mixer in 1st Embodiment. It is the timing chart shown about the example of operation (operation of a media processing thread) of the multipoint audio mixing device concerning a 1st embodiment. It is the timing chart shown about the example of operation (operation of a media processing thread) of the multipoint audio mixing device concerning a 2nd embodiment.

(A) First Embodiment Hereinafter, a first embodiment of a sound processing apparatus, program, and method according to the present invention will be described in detail with reference to the drawings. Below, the example which applies the audio | voice processing apparatus and audio | voice processing program of this invention to a multipoint audio | voice mixing apparatus is demonstrated.

(A-1) Configuration of the First Embodiment FIG. 2 is a block diagram showing an example of the hardware configuration and connection configuration of the multipoint audio mixing device 10 of this embodiment.

  As shown in FIG. 2, the multipoint audio mixing apparatus 10 includes N × M terminals 20-1_1 to 20-M_N (terminals 20-1_1, 20-1_2,..., 20-1_N, 20-2_1, 20−). 2_2, ..., 20-2_N, ... 20-M_1, 20-M_2, ..., 20-M_N) and the network 40. N and M are arbitrary integers of 2 or more. The N × M terminals 20 are the maximum number of terminals 20 that can be connected to the multipoint audio mixing apparatus 10, and the number of terminals 20 connected to the multipoint audio mixing apparatus 10 is N × M. It may be the following.

  For example, an IP network can be applied as the network 40, but the network connection configuration between the multipoint audio mixing apparatus 10 and each terminal 20 is not limited. In this embodiment, it is assumed that real-time transmission / reception of audio (conference audio) data can be performed between the multipoint audio mixing apparatus 10 and each terminal 20 by IP communication.

  FIG. 3 is a block diagram showing an example of the internal configuration of the terminal 20.

  Each terminal 20 functions as a conference terminal (telephone terminal). The specific configuration of each terminal 20 is not limited to the configuration shown in FIG. 3, and for example, an IP telephone or a PC installed with a softphone application can be applied.

  In this embodiment, the following description will be made assuming that all the terminals 20 have the configuration shown in the block diagram of FIG. 3, but the specific configuration of each terminal 20 is not limited to the configuration of FIG. . For example, an IP telephone or a computer functioning as a soft phone (for example, a PC, a smartphone, a tablet PC or the like in which a soft phone application is installed) can be applied.

  The terminal 20 shown in FIG. 3 includes a call processing unit 21, a communication unit 22 as a network interface, a microphone 23 that captures a user's voice, and a speaker 24 that outputs a voice to the user. The microphone 23 and the speaker 24 function as a handset in the terminal 20, and for example, a telephone or speakerphone handset or headset can be applied.

  The call processing unit 21 performs processing related to a call such as processing of voice data / voice signals and call control processing. For example, when the terminal 20 is configured by a general-purpose computer such as a PC, the call processing unit 21 is a constituent element corresponding to a softphone application.

  The call processing unit 21 connects a call for telephone communication with the multipoint audio mixing device 10 and transmits and receives audio data in real time. Protocols for call control processing and voice communication between the terminal 20 and the multipoint voice mixing device 10 are not limited. For example, protocols such as SIP (Session Initiation Protocol) and RTP (Real-time Transport Protocol) are used. It is assumed that call control processing and voice communication are possible.

  The call processing unit 21 transmits audio data obtained by performing a predetermined encoding process (codec conversion) on the audio signal captured by the microphone 23 to the multipoint audio mixing apparatus 10. In addition, the call processing unit 21 performs a process of decoding the audio data received from the multipoint audio mixing device 10 into an audio signal and outputting the audio signal from the speaker 24. The sound signal encoding method (codec) used between the multipoint audio mixing apparatus 10 and the terminal 20 is not limited, but for example, ITU-T G. 711, G.G. An encoding scheme such as 729 can be applied.

  Next, the hardware configuration of the multipoint audio mixing apparatus 10 will be described.

  The multipoint audio mixing apparatus 10 includes a data processing unit 11 and a communication unit 12.

  The multipoint audio mixing apparatus 10 can be configured, for example, by installing the audio processing program of the embodiment on a general-purpose computer (server apparatus) such as a PC or a workstation.

  The communication unit 12 is a communication interface for connecting to the network 40.

  The data processing unit 11 functions as data processing means (computer) that performs voice processing (for example, transmission / reception of voice data, voice mixing processing, etc.) related to telephone communication (conference communication) with the terminal 20 in software. is there.

  The data processing unit 11 includes a processor 111 and a memory 112 for functioning as data processing means (computer).

  In FIG. 2, the processor 111 is illustrated as one block, but may be physically configured by a plurality of processors. The processor 111 may be configured using a processor (multi-core processor) having a plurality of cores.

  Further, in FIG. 2, the memory 112 is illustrated as one block, but the specific configuration is not limited. For example, the volatile memory (eg, SRAM, DRAM, etc.) that operates at high speed, and the nonvolatile You may make it comprise combining the memory of multiple media, such as memory (for example, flash memory, HDD, etc.). In this embodiment, a description will be given assuming that the audio processing program of the embodiment is installed in a computer configured by the data processing unit 11.

  Next, a functional configuration (configuration of a thread) when the voice processing program of the embodiment is installed in the data processing unit 11 will be described with reference to FIG.

  As shown in FIG. 1, the data processing unit 11 (speech processing program of the embodiment) of the multipoint speech mixing apparatus 10 includes media processing threads 30 (30-1 to 30-M) as M speech processing means. By using this, the audio mixing processing of the NXM channel (channel for transmitting and receiving audio data with N × M terminals 20) is realized.

  The media processing thread 30 is activated periodically, for example, every 10 ms, and performs processing of audio data in the period processing (in this example, 10 ms) as one processing unit. That is, each media processing thread 30 is capable of processing N channels (processing a stream of audio data transmitted and received by N terminals 20). In the data processing unit 11, means for generating and managing each media processing thread 30 (30-1 to 30-M) (for example, an environment such as middleware or programming language for generating and managing threads) is not limited. Therefore, various configurations can be applied. Further, the interval between periodic activations and the time for processing audio data in the media processing thread 30 are not limited to 10 ms.

  Each media processing thread 30 has N audio reception processes 31. In FIG. 1, the media processing threads 30-1 to 30-M have audio reception processes 31-1_1 to 311-1_N, 31-2_1 to 31-2_N,..., 31-M_1 to 31-M_N, respectively. . In addition, the media processing threads 30-1 to 30-M have audio transmission processes 36-1_1 to 36-1_N, 36-2_1 to 36-2_N,..., 36-M_1 to 36-M_N, respectively. The voice reception processes 31-1_1 to 31-M_N receive and process voice data (encoded voice data) supplied from the terminals 20-1_1 to 20-M_N, respectively. The voice transmission processes 36-1_1 to 36-M_N transmit the voice data (encoded voice data) subjected to the mixing process to the terminals 20-1_1 to 20-M_N, respectively.

  FIG. 4 is a block diagram showing the internal configuration of each audio reception process 31.

  As shown in FIG. 4, each audio reception process 31 includes an RTP reception 311, a jitter buffer 312, and a decoder 313. The RTP reception 311 executes reception processing of RTP packets that have arrived before the periodic processing, and inputs an RTP payload (voice encoded data) into the jitter buffer 312. The jitter buffer 312 takes out encoded audio data for 10 ms (a predetermined processing unit time), demodulates (demodulates into linear audio data) by the decoder 313, and audio data for 10 ms (linear audio data). Is output.

  FIG. 5 is a block diagram showing the internal configuration of each audio transmission process 36.

  As shown in FIG. 5, the voice transmission processing 36 includes an encoder 361 and an RTP transmission 362. The encoder supplies the encoded voice data for 10 ms (a predetermined processing unit time) to the RTP transmission 362. The RTP transmission 362 generates an RTP packet when the encoded voice data for the RTP packetization period can be accumulated, and transmits the RTP packet. In this embodiment, as an example, it is assumed that the RTP packetization period is 20 ms. The RTP packetization period is not limited to 20 ms.

  Each media processing thread 30 (30-1 to 30-M) includes an addition process 32 (32-1 to 32-M), a circular buffer 33 (33-1 to 33-M), and an addition process 34 (34-). 1 to 34-M) and a mixer 35 (35-1 to 35-M). For example, the media processing thread 30-1 includes an addition process 32-1, a circular buffer 33-1, an addition process 34-1 and a mixer 35-1.

  Hereinafter, the media processing threads 30-1 to 30-M are referred to as the 1st to Mth media processing threads 30, respectively. In the following, an arbitrary m-th (m is an arbitrary integer from 1 to M) media processing thread 30 is represented as a media processing thread 30-m. Hereinafter, the (m−1) th media processing thread 30 is referred to as a media processing thread 30- (m−1), and the (m + 1) th media processing thread 30 is referred to as a media processing thread 30- (m + 1). Shall. Therefore, the media processing thread 30-m includes the media processing thread 30-m, the audio reception processes 31-m_1 to 31-m_N, the addition process 32-m, the circular buffer 33-m, the addition process 34-m, and the mixer 35-m. And voice transmission processing 36-m_1 to 36-m_N.

  The order of the media processing threads 30-1 to 30-M will be described as being cyclically managed. For example, it is assumed that the media processing threads 30 whose order is adjacent to the media processing thread 30-1 are the media processing thread 30-2 and the media processing thread 30-M. Therefore, for example, when the media processing threads 30- (m + 1) to 30- (m + M-1) are used, the media processing threads 30-1 to 30-M other than the media processing thread 30-m are represented. For example, when m = 3, media processing threads 30- (m + 1) to 30- (m + M-1) represent media processing threads 30-1 to 30-2 and media processing threads 30-4 to 30-M. It will be.

  In the following, the internal configuration of each media processing thread 30 will be described with an example centering on the mth media processing thread 30-m.

  The addition process 32-m adds (synthesizes) unit time audio data (for example, linear audio data for 10 ms) output from each of the audio reception processes 31-m_1 to 31-m_N, and adds 10 ms of audio. Data is written to the circular buffer 33-m. Note that various audio data addition techniques (synthetic techniques) are applied to the process in which the addition process 32 adds (synthesizes) sounds related to a plurality of audio data and generates the audio data after the addition (after synthesis). Detailed explanation is omitted.

  FIG. 6 is a block diagram showing the internal configuration of the circular buffer 33-m.

  The circular buffer 33 includes Z buffer surfaces 331-1 to 331-Z, a reading position pointer group 332, a writing surface selection unit 333, a reading surface selection unit 334, and a reading position pointer selection unit 335. The circular buffer 33-m holds a write position pointer WP (m). Each buffer surface 331 is a buffer capable of holding one unit (in this embodiment, 10 ms of audio data (linear audio data)) when the audio data is processed by the media processing thread 30. Hereinafter, pointer values (address values) corresponding to the buffer surfaces 331-1 to 331 -Z are 1 to Z, respectively.

  The write position pointer WP (m) manages a pointer value for writing audio data (a pointer value indicating one of the buffer surfaces 331-1 to 331 -Z). The writing surface selection means 333 updates (increments) the value of the writing position pointer WP (m) and then writes linear audio data for 10 ms into the buffer surface 331 corresponding to the writing position pointer WP (m). . The pointer value of the write position pointer WP (m) is incremented by 1 at the time of writing, and becomes 1 at the next writing time when reaching Z. That is, audio data is cyclically written into the buffer surfaces 331-1 to 331 -Z by the writing surface selection means 333.

  The read position pointer group 332 includes read position pointers RP (that is, M-1 pieces) corresponding to each of the media processing threads 30- (m + 1) to 30- (m + M-1) other than the self (media processing thread 30-m). Are read position pointers RP). For example, in FIG. 6, the read position pointer group 332 of the media processing thread 30-m includes read position pointers RP (1), RP (2),..., RP (m−1), RP (m + 1),. RP (M) is arranged. The read position pointer RP holds a pointer value of audio data to be read by the corresponding media processing thread 30 (a pointer value indicating one of the buffer surfaces 331-1 to 331 -Z).

  The read position pointer selection means 335 selects a read position pointer RP corresponding to any one of the media processing threads 30, updates (increments) the pointer value of the read position pointer RP, and then selects the selected read position pointer RP. The pointer value is supplied to the reading surface selection means 334. The reading surface selection unit 334 performs processing of reading and outputting audio data from the buffer surface 331 corresponding to the supplied pointer value (outputting to the media processing thread 30 selected by the reading position pointer selection unit 335).

  Similarly to the writing surface selection unit 333, the reading position pointer selection unit 335 increments the pointer value of the reading pointer value RP by 1 for each reading trigger, and when it reaches Z, 1 is read at the next reading trigger. Performs a cyclic operation.

  For example, in the media processing thread 30- (m−1), after updating the reading position pointer RP (m−1), one unit (10 ms worth) from the buffer surface corresponding to the reading position pointer RP (m−1). ) Audio data (linear audio data). The reading position pointer RP (m−1) is incremented by 1 at the time of reading, and when reaching to Z, a cyclic operation of setting to 1 at the next reading time is performed. The same read processing is performed by the media processing threads 30-1 to 30- (m-2) and 30- (m + 1) to 30-M.

  The addition process 34 requests other media processing threads 30 (circular buffer 33) to output audio data (audio data held in the buffer surface 331 corresponding to the pointer value of the reading position pointer RP), Audio data obtained by adding (synthesizing) them is generated and supplied to the mixer 35. That is, the addition process 34-m is performed for 10 ms from the circular buffers 33 (circular buffers 33-1 to (m-1) and circular buffers 33- (m + 1) to 33-M) other than the media processing thread 30-m. The linear audio data is read and added (synthesized), and 10 ms of linear audio data is transferred to the mixer 35-m. Note that various audio data addition techniques (synthesis techniques) are applied to the process in which the addition process 34 adds (synthesizes) sounds related to a plurality of sound data and generates the sound data after the addition (after synthesis). Detailed explanation is omitted.

  Next, the internal configuration of the mixer 35 will be described.

  FIG. 7 is an explanatory diagram showing the internal configuration of the mixer 35-m that constitutes an arbitrary media processing thread 30-m.

  The mixer 35-m includes mixer units 351-1 to 351-N that synthesize audio data supplied to each of the audio transmission processes 36-m_1 to 36-m_N.

  The mixer unit 351-n synthesizes (mixes) the audio data supplied from the audio reception processes 31-m_ (n + 1) to 31-m_ (n + N-1) and the audio data supplied from the addition process 34-m. The generated voice data is generated and supplied to the corresponding voice transmission processing 36. Specifically, for example, as illustrated in FIG. 7, the mixer unit 351-1 supplies the synthesized audio data to the audio transmission process 36-m_1. Therefore, as shown in FIG. 7, the mixer unit 351-1 synthesizes (mixes) the audio data supplied from the audio transmission processing 36-m_2 to 36-m_N and the audio data supplied from the addition processing 34-m. ). Note that various audio data synthesizing techniques can be applied to the process in which the mixer unit 351 synthesizes audio related to a plurality of audio data and generates synthesized audio data, and thus detailed description thereof is omitted.

  As a result, the terminal 20-1_1 that is the destination of the voice transmission process 36-1_1 receives data other than the voice data transmitted from the own apparatus (terminal 20-1_1) (voice data supplied to the voice reception process 31-1_1). Audio data obtained by synthesizing (mixing) audio data of all the terminals 20-1-2 to 20-M_N is transmitted. The same applies to all the other terminals 20-1-2 to 20-M_N. As described above, the multipoint audio mixing apparatus 10 combines (mixing) a combination of audio data excluding the destination terminals 20 with respect to all M × N terminals 20-1_1 to 20-M_N. ) Can be sent.

(A-2) Operation of the First Embodiment Next, the operation (audio processing method of the embodiment) of the multipoint audio mixing apparatus 10 of the first embodiment having the above configuration will be described.

  FIG. 8 is a timing chart showing an example of write / read timing of the circular buffer 33-m in this embodiment.

  In the example illustrated in FIG. 8, the number of buffer surfaces 331 configuring the media processing thread 30 is illustrated as eight (that is, Z = 8).

  FIG. 8A shows the periodic processing timing of the media processing thread 30-m, the writing timing of the circular buffer 33-m, the changing timing of the writing position pointer WP (m), and the pointer value. In (b), the periodic processing timing of the media processing thread 30- (m-1), the reading timing of the circular buffer 33- (m-1), the changing timing of the reading position pointer RP (m-1), and the pointer value are illustrated. Has been. Further, FIG. 8C illustrates the periodic processing timing of the media processing thread 30- (m + 1), the reading timing of the circular buffer 33- (m + 1), the changing timing of the reading position pointer RP (m + 1), and the pointer value. Yes. Note that the write position pointer WP shown in FIG. 8A is a pointer belonging to the media processing thread 30-m. Also, the read position pointer RP (m−1) shown in FIG. 8B and the read position pointer RP (m + 1) shown in FIG. 8C are both stored in the read position pointer group 332 of the media processing thread 30-m. It belongs to.

  In FIG. 8, data is transferred from the media processing thread 30- (m) to the media processing thread 30- (m-1), and data is transferred from the media processing thread 30- (m) to the media processing thread 30- (m + 1). Although the transfer has been described, the same applies to the data transfer from the media processing thread 30- (m) to the media processing threads 30-1 to 30- (m-2) and 30- (m + 2) to 30-M. Therefore, detailed explanation is omitted.

  Here, it is assumed that the media processing thread 30- (m) is periodically activated every 10 ms, for example, and the activation cycle is not disturbed. Then, in the media processing thread 30- (m), a write trigger to the circular buffer (m) occurs every 10 ms. At the first writing opportunity, the writing position pointer WP (m) is set to 1 and linear audio data for 10 ms is written to the buffer surface 331-1. At the writing opportunity after the next time, the writing position pointer WP (m) is incremented, and linear audio data for 10 ms is written to the corresponding buffer surface 331. In the media processing thread 30- (m), the write position pointer WP (m) is set to 1 at the next write trigger when the write position pointer WP (m) becomes 8, which is the same as the number of buffer planes. 10 ms worth of linear audio data is written.

  Here, it is assumed that the media processing thread 30- (m−1) is periodically activated every 10 ms as in the case of the media processing thread 30- (m), and the activation cycle is not disturbed. Then, in the media processing thread 30- (m−1), a read trigger from the circular buffer 33- (m) occurs every 10 ms.

  For example, the media processing thread 30- (m−1) starts the reading process after data is written to the buffer surface 331-1 of the media processing thread 30- (m).

  After the media processing thread 30- (m) writes linear audio data for 10 ms to the buffer surface 331-1, when a read trigger of the media processing thread 30- (m-1) occurs, this is the media processing thread 30-. This is the first read trigger in (m-1). Then, the read position pointer RP (m−1) is set to 1 in the media processing thread 30- (m). Then, the audio data (linear audio data for 10 ms) held on the buffer surface 331-1 of the media processing thread 30- (m) is read into the media processing thread 30- (m-1). In the media processing thread 30- (m), the reading position pointer RP (m-1) is incremented when the media processing thread 30- (m-1) reads from the next time onward, and the audio data ( 10 ms linear audio data) is supplied. The reading position pointer RP (m−1) is set to 1 at the next reading opportunity when the number (maximum number) of the buffer surfaces 331 becomes 8 (= Z).

  Here, it is assumed that the media processing thread 30- (m + 1) periodically starts every 10 ms, similarly to the media processing threads 30-30- (m) and 30- (m-1). However, here, as shown in FIG. 8, the media processing thread 30- (m + 1) reads the audio data in the buffer surface 331-5 of the media processing thread 30- (m), and the cycle processing takes time. It is assumed that it took. Further, here, as shown in FIG. 8, in the media processing thread 30- (m + 1), the activation cycle is disturbed at the cycle of reading the buffer surfaces 331-5, 331-6, 331-7, 331-8. Suppose that it occurred. In the processing by the software (thread) in the data processing unit 11, for example, due to the occurrence of waiting for data writing to the hard disk, the periodic processing time may be longer than usual, and the activation cycle may be disturbed.

  In this case, in the media processing thread 30- (m + 1), the RTP transmission processing is delayed and fluctuated in accordance with the disturbance of the startup cycle in the cycle of reading the buffer surfaces 331-5, 331-6, 331-7, 331-8. However, the desired mixing process can be continued.

(A-3) Effects of First Embodiment According to the first embodiment, the following effects can be achieved.

  As described above, the multipoint audio mixing apparatus 10 synthesizes (mixes) audio data excluding audio data related to each destination terminal 20 to all M × N terminals 20-1_1 to 20-M_N. ) Can be sent.

  In the data processing unit 11, for example, each media processing thread 30 can be operated in parallel by different CPU cores. Therefore, in the past, N × M channels were the performance limit of N channel mixing processing in one CPU core. The mixing process can be executed. Therefore, the multipoint audio mixing apparatus 10 of the present invention can process N × M audio data in software on the general-purpose OS. In particular, in the multipoint audio mixing device 10, each media processing thread 30 includes an addition process 32, a circular buffer 33, an addition process 34, and a mixer 35, thereby collecting audio data synthesized by other media processing threads 30. And step by step. As a result, the multipoint audio mixing apparatus 10 does not require a configuration for centrally processing the audio data of the plurality of media processing threads 30, and only N × M channel of audio data processing distributed to the plurality of media processing threads 30 is required. Mixing processing is realized.

(B) Second Embodiment Hereinafter, a second embodiment of the speech processing apparatus and the mixing program according to the present invention will be described in detail with reference to the drawings. Below, the example which applies the audio | voice processing apparatus and mixing program of this invention to a multipoint audio | voice mixing apparatus is demonstrated.

(B-1) Configuration of Second Embodiment The configuration of the multipoint audio mixing apparatus 10 of the second embodiment can also be shown using the above-described FIGS. Below, only the difference with 1st Embodiment is demonstrated about the multipoint audio | voice mixing apparatus 10 of 2nd Embodiment.

  Among the media processing threads 30, the operation timing may be shifted between the media processing threads 30 due to disturbance of the activation cycle or the like. In that case, in some media processing threads 30, the position of any read position pointer RP may pass the position of the write position pointer WP, and incorrect data may be processed in time series. Therefore, in the second embodiment, a process for absorbing a difference in operation timing between the media processing threads 30 due to the start cycle disturbance is performed.

  Specifically, in the second embodiment, for example, a predetermined number of media processing threads 30- (m) for media processing threads 30 other than one or more media processing threads 30- (m) serving as an arbitrary reference. The reading process is started after writing to the buffer surface 331 (in this embodiment, for example, after writing data to the buffer surface 331-2 in this embodiment). Furthermore, in the second embodiment, the read position pointer RP of the media processing thread 30 is written when the audio data is read by each media processing thread 30 (reading from the circular buffer 33 of the other media processing thread 30). A processing configuration that prevents the position pointer WP (m) from being overtaken (for example, no increment is performed when overtaking) is added.

(B-2) Operation | movement of 2nd Embodiment Next, operation | movement (voice processing method of embodiment) of the multipoint audio | voice mixing apparatus 10 of 2nd Embodiment which has the above structures is demonstrated.

  FIG. 9 is a timing chart showing an example of write / read timing of the circular buffer 33-m according to the second embodiment.

  In the example illustrated in FIG. 9, the number of buffer surfaces 331 configuring the media processing thread 30 is illustrated as eight (that is, Z = 8).

  FIG. 9A illustrates the periodic processing timing of the media processing thread 30- (m), the writing timing of the circular buffer 33-m, the change timing of the writing position pointer WP (m), and the pointer value. In FIG. 9B, the periodic processing timing of the media processing thread 30- (m-1), the reading timing of the circular buffer 33- (m-1), the changing timing of the reading position pointer RP (m-1), and the pointer value. Is shown. Further, FIG. 9C illustrates the periodic processing timing of the media processing thread 30- (m + 1), the reading timing of the circular buffer 33- (m + 1), the changing timing of the reading position pointer RP (m + 1) and the pointer value. Yes. Note that the write position pointer WP shown in FIG. 9A is a pointer belonging to the media processing thread 30- (m). Also, the read position pointer RP (m−1) shown in FIG. 9B and the read position pointer RP (m + 1) shown in FIG. 9C are both read position pointer groups of the media processing thread 30- (m). It belongs to 332.

  In FIG. 9, data is transferred from the media processing thread 30- (m) to the media processing thread 30- (m-1), and data is transferred from the media processing thread 30- (m) to the media processing thread 30- (m + 1). Although the transfer has been described, the same applies to the data transfer from the media processing thread 30- (m) to the media processing threads 30-1 to 30- (m-2) and 30- (m + 2) to 30-M. Therefore, detailed explanation is omitted.

  For example, the media processing thread 30-m is periodically started every 10 ms, but it takes time to perform periodic processing at a cycle in which the buffer surface 331-4 is written, and the buffer surfaces 331-5, 331-6, 331-7, Assume that a disturbance in the start cycle occurs in the cycle in which 331-8 and 331-1 are written. In processing by software, for example, due to the occurrence of waiting for data writing to the hard disk, the cycle processing time becomes longer than usual, and the startup cycle may be disturbed.

  Similarly to the media processing thread 30-m, the media processing thread 30- (m-1) is periodically activated every 10 ms, and the activation cycle is not disturbed. Then, in the media processing thread 30- (m−1), a read trigger from the circular buffer 33-m occurs every 10 ms.

  In this embodiment, media processing threads 30 other than the media processing thread 30-m start reading processing after data is written to the buffer surface 331-2 of the media processing thread 30-m. Therefore, the media processing thread 30- (m−1) starts the reading process after data is written to the buffer surface 331-2 of the media processing thread 30-m. Therefore, as shown in FIG. 9, after the media processing thread 30-m writes 10 ms worth of audio data (linear audio data) to the buffer surface 331-2, the media processing thread 30- (m-1) is read. When an opportunity occurs, this is the first reading opportunity. At this time, the media processing thread 30-m sets the reading position pointer RP (m-1) to 1, and reads audio data (linear audio data) for 10 ms from the buffer surface 331-1. The media processing thread 30-m increments the reading position pointer RP (m−1) and reads audio data (linear audio data) for 10 ms from the corresponding buffer surface 331 at the next reading opportunity. Become.

  As shown in FIG. 9, at the first read trigger (read trigger of the media processing thread 30-m) when the read position pointer RP (m-1) is 5, the write position pointer WP ( The value of m) is 5. Therefore, at this time, the latest audio data is written in the buffer surface 331-5 in the media processing thread 30-m, and the audio data is not yet written in the buffer surface 331-6. Therefore, the media processing thread 30- (m-1) reads the audio data of the buffer surface 331-5 with the reading position pointer RP (m-1) held as the previous value (without incrementing) at this reading opportunity. Make sure. In this case, the media processing thread 30- (m-1) reads the audio data of the buffer surface 331-5 twice in succession, and at this moment, the continuity of the audio is lost for a moment. Can be maintained.

  The media processing thread 30- (m-1) executes a process similar to the packet loss compensation in the decoder, instead of reading the voice data twice in succession, so that the artificial voice (dummy voice) is obtained from the past voice. ) May be generated. Then, the continuity of the voice can be maintained even at this opportunity. Then, at the next read trigger when the read position pointer RP (m−1) is equal to 8 as the number of buffer planes, the read position pointer RP (m−1) is set to 1, and the buffer of the media processing thread 30-m The voice data (linear voice data) for 10 ms is read from the surface 331-1.

(B-3) Effects of Second Embodiment According to the second embodiment, the following effects can be obtained in addition to the effects of the first embodiment.

  In the multipoint audio mixing apparatus 10 of the second embodiment, after the data is written to the plurality of buffer surfaces 331 by the media processing thread 30 as an arbitrary reference, the reading process of the other media processing thread 30 is started. ing. In the multipoint audio mixing apparatus 10 according to the second embodiment, the reading position pointer RP is changed to the writing position pointer WP (when the media processing thread 30 reads) (when the audio data is read from other media processing threads 30). Since a process for preventing overtaking m) is added, the continuity of the read data voice can be maintained.

(C) Other Embodiments The present invention is not limited to the above-described embodiments, and may include modified embodiments as exemplified below.

  (C-1) In the second embodiment, the process for preventing the read position pointer RP from overtaking the write position pointer WP (m) has been described. However, the buffer plane number difference between the write position pointer WP (m) and the read position pointer RP becomes a transmission delay of the buffer plane number difference × starting cycle, and it is not desirable that this transmission delay becomes too large. Therefore, each media processing thread 30 may perform delay recovery processing for transmission delay. For example, each media processing thread 30 increments the read position pointer RP by two when “write position pointer WP (m) −read position pointer RP ≧ X” (for example, X = 4), Data recovery may be performed to perform a delay recovery process for transmission delay. Further, each media processing thread 30 increments the reading position pointer RP by two only in a silent section where there is no audio data when “writing position pointer WP (m) −reading position pointer RP ≧ X” is satisfied. May be.

  (C-2) In each of the above embodiments, the example in which the audio processing device of the present invention is applied to a multipoint audio mixing device has been described. However, the audio processing device can be applied to various devices that receive and mix audio data. it can. In each of the above embodiments, an example in which a packet of encoded audio data is received and audio mixing processing has been described, but the encoding method and division format of audio data at the time of reception are not limited. Of course.

  DESCRIPTION OF SYMBOLS 10 ... Multipoint audio mixing apparatus, 11 ... Data processing part, 111 ... Processor, 112 ... Memory, 12 ... Communication part, 40 ... Network, 20, 20-1_1 to 20-M_N ... Terminal, 21 ... Call processing part, 22 Communication unit, 23, microphone, 24, speaker, 30, 30-1 to 30-M, media processing thread, 31, 31-1_1 to 31-M_N, voice reception processing, 311 ... RTP reception, 312 ... jitter buffer, 313: Decoder, 32, 32-1 to 32-M ... Addition processing, 33, 33-1 to 33-M ... Circular buffer, 34, 34-1 to 34-M ... Addition processing 34, 35, 35-1 35-M: Mixer, 36, 36-1_1-36-M_N: Audio transmission processing, 361: Encoder, 362: RTP transmission, 331, 331-1-331-Z: Buffer surface, 3 2 ... read position pointers, 333 ... writing surface selecting means, 334 ... read plane selecting means, 335 ... read position pointer selecting means, 351,351-1~351-N ... mixer.

Claims (9)

A plurality of voice processing means,
Each of the above voice processing means
A plurality of sound signal receiving means for receiving reception sound data based on reception sound for each time series;
First synthesizing means for synthesizing received sound data received by the plurality of sound signal receiving means to generate first synthesized sound data;
Buffer means for holding first synthesized speech data;
Second synthesis means for synthesizing the first synthesized voice data generated by each of the other voice processing means to generate second synthesized voice data;
For each transmission destination, transmission sound data is generated by synthesizing voice data obtained by excluding the reception sound data related to the transmission destination from the reception sound data received by the plurality of sound signal receiving means and the second synthesized voice data. And a third synthesizing unit.   The buffer means stores the first synthesized speech data in a time series order in a circular buffer in which a plurality of buffer units holding one time-series first synthesized speech data are circularly arranged. The speech processing apparatus according to claim 1.   3. The buffer means starts outputting the first synthesized voice data held in the circular buffer after holding a plurality of first synthesized voice data in the circular buffer. The voice processing apparatus according to 1.   The buffer means has a read pointer for managing the read position of the circular buffer and a write pointer for managing the write position of the circular buffer, so that the position of the read pointer does not pass the position of the write pointer. 4. The voice processing apparatus according to claim 2, wherein the position of the reading pointer is controlled.   The buffer means does not increment the position of the read pointer if the position of the read pointer and the position of the write pointer are the same when reading the first synthesized speech data corresponding to the read pointer. The speech processing apparatus according to claim 4.   The buffer means reads the first synthesized speech data corresponding to the read pointer, and if the difference between the position of the write pointer and the position of the read pointer is equal to or greater than a threshold value, 2 for the read position pointer. The speech processing apparatus according to claim 4, wherein the value is incremented.   7. The sound processing apparatus according to claim 1, wherein each of the sound processing means is constituted by a thread on a computer. Let the computer function as multiple audio processing means,
Each of the above voice processing means
A plurality of sound signal receiving means for receiving reception sound data based on reception sound for each time series;
First synthesizing means for synthesizing received sound data received by the plurality of sound signal receiving means to generate first synthesized sound data;
Buffer means for holding first synthesized speech data;
Second synthesis means for synthesizing the first synthesized voice data generated by each of the other voice processing means to generate second synthesized voice data;
For each transmission destination, transmission sound data is generated by synthesizing voice data obtained by excluding the reception sound data related to the transmission destination from the reception sound data received by the plurality of sound signal receiving means and the second synthesized voice data. A voice processing program comprising: a third synthesizing unit. In the speech processing method executed by the speech processing apparatus,
A plurality of voice processing means,
Each sound processing means comprises a plurality of sound signal receiving means, first synthesizing means, buffer means, second synthesizing means, and third synthesizing means,
Each of the sound signal receiving means receives reception sound data based on reception sound for each time series,
The first synthesizing unit synthesizes the received sound data received by the plurality of sound signal receiving units to generate first synthesized voice data,
The buffer means holds the first synthesized voice data,
The second synthesis means synthesizes the first synthesized voice data generated by each of the other voice processing means to generate second synthesized voice data,
The third synthesizing means, for each transmission destination, audio data obtained by excluding reception sound data related to the transmission destination from reception sound data received by the plurality of sound signal reception means, and the second synthetic audio data. A voice processing method characterized by generating synthesized transmission sound data.

Images