JP6476768B2 - Voice processing apparatus, program and method - Google Patents

Description

  The present invention relates to an audio processing device, program, and method, and, for example, audio mixing of a conference server (for example, a device such as an MCU (Multipoint Control Unit)) configuring a conference system that provides a conference environment by connecting multiple points via a network It can be applied to processing.

  Conventionally, in a conference system that provides a conference environment by connecting multiple points on a network, dedicated hardware is usually used for processing for mixing voices between multiple locations.

  By the way, in an apparatus for performing audio mixing processing of a conventional conference system, ITU-T G. When a high compression coding method such as 729 is used, the processing load on the decoder and encoder increases, and the number of channels that can be mixed is limited. Furthermore, in recent years, from the viewpoint of cost reduction and maintenance regarding the network equipment, it is required to realize the mixing function as software on a general-purpose server without using a dedicated hardware.

  As a prior art for solving such a subject, there is a description art of patent documents 1.

  Patent Document 1 describes that, by using a plurality of communication blocks for mixing a plurality of input / output audio data and a central mixer, the load of the mixing operation processing can be reduced to realize multipoint audio mixing. There is.

Patent publication 2008-211291 gazette

  Generally, a general-purpose server mounts a plurality of CPUs, and each CPU is configured of a plurality of CPU cores. When the central mixer in the device of Patent Document 1 is implemented as software on the general-purpose server, The processing capacity of one CPU core becomes the performance limit of media processing thread processing. For example, in the case of a server of 12 cores (2 CPU × 6 cores), the number of channels that can be mixed is limited by the processing capacity of 1/12 of the entire CPU.

  Therefore, there is a need for an audio processing apparatus, program and method that can perform software mixing with low cost and more audio channels.

The voice processing apparatus according to the first aspect of the present invention comprises (1) a plurality of voice processing means, and (1-1) each of the voice processing means comprises (1-2) reception based on received sound for each time series A plurality of sound signal reception means for receiving sound data; and (1-3) a first synthesis means for synthesizing the reception sound data received by the plurality of sound signal reception means to generate first synthesized speech data. , (1-4) buffer means for holding the first synthetic speech data, and (1-5) the first synthetic speech data generated by each of the other speech processing means are synthesized to produce a second synthesis (2-6) voice data obtained by excluding the received sound data relating to the transmission destination from the received sound data received by the plurality of sound signal receiving means for each transmission destination; Generating transmission sound data synthesized with the second synthetic speech data Have a third combining means (1-7) said buffer means, the circular buffer having a plurality of buffer portion is cyclically arranged to hold the first of the synthesized speech data of one time series, time series The first synthetic speech data is stored in order, and (1-8) the buffer means holds the predetermined number of first synthetic speech data in the circulation buffer, and then holds the first synthetic speech data in the circulation buffer. Starting to output the synthesized speech data .

The voice processing program according to the second aspect of the present invention makes (1) the computer function as a plurality of voice processing means, and (2) each of the above voice processing means is based on the received sound for each (2-1) time series. A plurality of sound signal receiving means for receiving the reception sound data; and (2-2) a first synthesis means for synthesizing the reception sound data received by the plurality of sound signal reception means to generate a first synthesized speech data And (2-3) buffer means for holding the first synthetic speech data, and (2-4) the first synthetic speech data generated by the other speech processing means, respectively, (2-5 ) voice data obtained by removing received sound data relating to the transmission destination from received sound data received by a plurality of the above-mentioned sound signal receiving means for each transmission destination; And the above second synthetic speech data Have a third synthesizing means for generating a transmitted sound data, (2-6) the buffer means, cyclically arranged a plurality of buffer portion for holding the first synthesized speech data of one time series The first synthetic speech data is stored in chronological order in the circular buffer, and (2-7) the buffer means holds the predetermined number of first synthetic speech data in the circular buffer, and Starting output of the first synthetic speech data held in

A third aspect of the present invention is the speech processing method executed by the speech processing apparatus, comprising (1) a plurality of speech processing means, and (2) each speech processing means comprising a plurality of sound signal receiving means, a first synthesizing means , The buffer means, the second combining means, and the third combining means, (3) each of the sound signal receiving means receives the received sound data based on the received sound for each time series, (4) The first synthesizing means synthesizes the received sound data received by the plurality of sound signal receiving means to generate first synthesized speech data, and (5) the buffer means generates the first synthesized speech data. (6) the second synthesizing unit synthesizes the first synthesized speech data generated by each of the other speech processing units to generate second synthesized speech data; (7) the above The third combining means receives a plurality of the above-mentioned sound signals for each transmission destination. Stage was that generate transmitted sound data obtained by synthesizing the audio data and the second synthesized speech data excluding the received sound data related to the destination from the received sound data received, (8) the buffer means, one The first synthetic speech data is stored in chronological order in a circular buffer in which a plurality of buffer units for holding the first synthetic speech data in time series are cyclically arranged, and (9) the buffer means comprises the circular buffer The present invention is characterized in that, after holding a predetermined number of first synthetic speech data, output of the first synthetic speech data held by the circulating buffer is started .

  According to the present invention, it is possible to provide an audio processing apparatus capable of performing software mixing processing for more audio channels at low cost.

FIG. 6 is a block diagram showing a configuration example of a media processing thread operating in the multipoint audio mixing device according to the first embodiment. It is the block diagram shown about the hardware constitutions and connection composition of the multipoint audio mixing device concerning a 1st embodiment. It is the block diagram shown about the example of the composition of the terminal connected to the multipoint sound mixing device concerning a 1st embodiment. It is the block diagram shown about the example of the internal configuration of the voice reception processing concerning a 1st embodiment. It is the block diagram shown about the example of the internal configuration of the voice transmission processing concerning a 1st embodiment. It is an explanatory view shown about an example of composition of a circulation buffer concerning a 1st embodiment. It is explanatory drawing shown about the example of the internal structure of the mixer in 1st Embodiment. It is a timing chart shown about an example of operation (operation of a media processing thread) of a multipoint audio mixing device concerning a 1st embodiment. It is a timing chart shown about an example of operation (operation of media processing thread) of a multipoint audio mixing device concerning a 2nd embodiment.

(A) First Embodiment Hereinafter, a first embodiment of a speech processing apparatus, program and method according to the present invention will be described in detail with reference to the drawings. Hereinafter, an example in which the audio processing device and the audio processing program of the present invention are applied to a multipoint audio mixing device will be described.

(A-1) Configuration of First Embodiment FIG. 2 is a block diagram showing an example of the hardware configuration and the connection configuration of the multipoint audio mixing device 10 of this embodiment.

  As shown in FIG. 2, the multipoint audio mixing apparatus 10 includes N × M terminals 20-1_1 to 20-M_N (terminals 20-1_1, 20-1_2,..., 20-1_N, 20-2_1, 20- , 20-2_N,..., 20-M_1, 20-M_2,..., 20-M_N) through the network 40. N and M are arbitrary integers of 2 or more. Further, the N × M terminals 20 are the maximum number of terminals 20 connectable to the multipoint audio mixing apparatus 10, and the number of terminals 20 connected to the multipoint audio mixing apparatus 10 is N × M. It may be the following.

  For example, an IP network can be applied as the network 40, but the network connection configuration between the multipoint audio mixing device 10 and each terminal 20 is not limited. In this embodiment, it is possible to perform real-time transmission and reception of voice (conference voice) data between the multipoint voice mixing device 10 and each terminal 20 by IP communication.

  FIG. 3 is a block diagram showing an example of the internal configuration of the terminal 20. As shown in FIG.

  Each terminal 20 functions as a conference terminal (telephone terminal). The specific configuration of each terminal 20 is not limited to the configuration shown in FIG. 3. For example, a PC or the like on which an application of an IP telephone or a soft phone is installed can be applied.

  In this embodiment, the terminal 20 will be described as the configuration shown in the block diagram of FIG. 3, but the specific configuration of each terminal 20 is not limited to the configuration of FIG. . For example, an IP telephone, a computer functioning as a soft phone (for example, a PC, a smart phone, a tablet PC or the like with an application of a soft phone installed) can be applied.

  The terminal 20 shown in FIG. 3 has a call processing unit 21, a communication unit 22 as a network interface, a microphone 23 for capturing the voice of the user, and a speaker 24 for voice output to the user. The microphone 23 and the speaker 24 function as a handset in the terminal 20, and for example, a handset of a telephone, a speakerphone, a headset or the like can be applied.

  The call processing unit 21 performs processing related to a call, such as processing of voice data / voice signal and call control processing. When the terminal 20 is configured by, for example, a general-purpose computer such as a PC, the call processing unit 21 is a component corresponding to an application of a softphone.

  The call processing unit 21 connects the call of the multipoint audio mixing apparatus 10 with the telephone communication, and transmits and receives audio data in real time. Protocols for call control processing and voice communication between the terminal 20 and the multipoint voice mixing apparatus 10 are not limited, but, for example, protocols such as SIP (Session Initiation Protocol) and RTP (Real-time Transport Protocol) may be used. It is assumed that call control processing and voice communication can be performed.

  The call processing unit 21 transmits, to the multipoint audio mixing apparatus 10, audio data obtained by applying predetermined encoding processing (codecization) to the audio signal captured by the microphone 23. Further, the call processing unit 21 decodes the audio data received from the multipoint audio mixing device 10 into an audio signal, and outputs the audio signal from the speaker 24. The encoding method (codec) of the sound signal used between the multipoint audio mixing device 10 and the terminal 20 is not limited, but, for example, for example, ITU-T G. 711, G. A coding scheme such as 729 can be applied.

  Next, the hardware configuration of the multipoint audio mixing apparatus 10 will be described.

  The multipoint audio mixing device 10 includes a data processing unit 11 and a communication unit 12.

  The multipoint audio mixing apparatus 10 can be configured, for example, by installing the audio processing program and the like of the embodiment in a general-purpose computer (server apparatus) such as a PC or a workstation.

  The communication unit 12 is a communication interface for connecting to the network 40.

  The data processing unit 11 functions as data processing means (computer) that performs audio processing (for example, transmission and reception of audio data, audio mixing processing, etc.) related to telephone communication (conference communication) with the terminal 20 as software. is there.

  The data processing unit 11 includes a processor 111 and a memory 112 for functioning as data processing means (computer).

  Although the processor 111 is illustrated as one block in FIG. 2, it may be physically configured with a plurality of processors. The processor 111 may be configured using a processor (multi-core processor) including a plurality of cores.

  Further, although the memory 112 is illustrated as one block in FIG. 2, the specific configuration is not limited. For example, volatile memory (for example, SRAM, DRAM, etc.) operating at high speed, and non-volatile It may be configured by combining memories of a plurality of media such as a memory (for example, a flash memory, an HDD, etc.). In this embodiment, it is assumed that the voice processing program of the embodiment is installed in a computer configured by the data processing unit 11.

  Next, a functional configuration (configuration of threads) when the voice processing program of the embodiment is installed in the data processing unit 11 will be described with reference to FIG.

As shown in FIG. 1, the data processing unit 11 (audio processing program of the embodiment) of the multipoint audio mixing apparatus 10 includes media processing threads 30 (30-1 to 30-M) as M audio processing means. Using this, voice mixing processing of N × M channels (channels for sending and receiving voice data with N × M terminals 20) is realized.

  The media processing thread 30 is periodically activated, for example, every 10 ms, and in the periodic processing, the media processing thread 30 processes audio data with the period (10 ms in the example of this embodiment) as one processing unit. That is, it is assumed that each media processing thread 30 can process N channels (process a stream of audio data transmitted and received by N terminals 20). In the data processing unit 11, means for generating and managing each media processing thread 30 (30-1 to 30-M) (for example, an environment such as middleware for generating and managing threads and an environment such as a programming language) is not limited. And various configurations can be applied. Further, the interval of periodic activation in the media processing thread 30 and the time of processing unit of audio data are not limited to 10 ms.

  Each media processing thread 30 has N voice reception processes 31. In FIG. 1, the media processing threads 30-1 to 30-M have the audio reception processing 31-1_1 to 31-1_N, 31-2_1 to 31-2_N, ..., 31-M_1 to 31-M_N, respectively. . In addition, the media processing threads 30-1 to 30-M have voice transmission processes 36-1_1 to 36-1_N, 36-2_1 to 36-2_N, ..., 36-M_1 to 36-M_N, respectively. The voice reception processing 31-1_1 to 31-M_N is configured to receive and process voice data (coded voice data) supplied from the terminals 20-1_1 to 20-M_N, respectively. Also, the voice transmission processing 36-1_1 to 36-M_N is for transmitting voice data (coded voice data) subjected to mixing processing to the terminals 20-1_1 to 20-M_N, respectively.

  FIG. 4 is a block diagram showing the internal configuration of each voice reception process 31. As shown in FIG.

  As shown in FIG. 4, each voice reception process 31 has an RTP reception 311, a jitter buffer 312 and a decoder 313. The RTP reception 311 executes reception processing of the RTP packet that has arrived up to the periodic processing, and puts an RTP payload (voice coded data) into the jitter buffer 312. Then, the jitter buffer 312 takes out 10 ms (predetermined processing unit time) of encoded audio data, and demodulates (demodulates it to linear audio data) by the decoder 313 to generate 10 ms audio data (linear audio data). Output

  FIG. 5 is a block diagram showing an internal configuration of each voice transmission process 36. As shown in FIG.

  As shown in FIG. 5, the voice transmission process 36 includes an encoder 361 and an RTP transmission 362. The encoder supplies to the RTP transmission 362 encoded voice data for 10 ms (predetermined processing unit time). Then, the RTP transmission 362 generates an RTP packet when encoded voice data for the RTP packetization cycle can be accumulated, and transmits the RTP packet. In this embodiment, as an example, the RTP packetization period is described as 20 ms. The RTP packetization period is not limited to 20 ms.

  In addition, each media processing thread 30 (30-1 to 30-M) performs addition processing 32 (32-1 to 32-M), circulation buffer 33 (33-1 to 33-M), addition processing 34 (34- 1 to 34-M) and mixers 35 (35-1 to 35-M). For example, the media processing thread 30-1 includes an addition process 32-1, a circulation buffer 33-1, an addition process 34-1, and a mixer 35-1.

  Hereinafter, the media processing threads 30-1 to 30-M will be referred to as first to M-th media processing threads 30, respectively. In the following, it is assumed that an arbitrary m-th (m is an arbitrary integer of 1 to M) media processing thread 30 is represented as a media processing thread 30-m. Also, hereinafter, the (m-1) th media processing thread 30 is referred to as a media processing thread 30- (m-1), and the (m + 1) th media processing thread 30 is referred to as a media processing thread 30- (m + 1). It shall be. Accordingly, the media processing thread 30-m includes the media processing thread 30-m, the audio reception processing 31-m_1 to 31-m_N, the addition processing 32-m, the circulation buffer 33-m, the addition processing 34-m, and the mixer 35-m. And voice transmission processing 36-m_1 to 36-m_N.

  Also, the order of the media processing threads 30-1 to 30-M will be described as being cyclically managed. For example, it is assumed that the media processing thread 30 whose order is adjacent to the media processing thread 30-1 is the media processing thread 30-2 and the media processing thread 30-M. Therefore, for example, in the case of the media processing threads 30- (m + 1) to 30- (m + M-1), the media processing threads 30-1 to 30-M represent other than the media processing thread 30-m. For example, when m = 3, the media processing threads 30- (m + 1) to 30- (m + M-1) represent the media processing threads 30-1 to 30-2 and the media processing threads 30-4 to 30-M. It will be.

  In the following, the internal configuration of each media processing thread 30 will be described with an example centered on the m-th media processing thread 30-m.

  The addition process 32-m adds (synthesizes) voice data (for example, linear voice data for 10 ms) for unit time output from each of the voice reception processes 31-m_1 to 31-m_N, and generates 10 ms for voice The data is written to the circular buffer 33-m. Note that various processings for adding voice data (synthesis technology) should be applied to processing in which the addition processing 32 adds (synthesizes) voices relating to a plurality of voice data and generates voice data after addition (post-synthesis) Detailed description is omitted.

  FIG. 6 is a block diagram showing an internal configuration of the circulation buffer 33-m.

  The circular buffer 33 includes Z buffer planes 331-1 to 331 -Z, a read position pointer group 332, a write plane selection unit 333, a read plane selection unit 334, and a read position pointer selection unit 335. Also, the circular buffer 33-m holds the write position pointer WP (m). Each buffer surface 331 is a buffer capable of holding one unit (in this embodiment, 10 ms of audio data (linear audio data)) when processing audio data by the media processing thread 30. In the following, it is assumed that pointer values (address values) corresponding to the buffer planes 331-1 to 331 -Z are 1 to Z respectively.

  The write position pointer WP (m) manages a pointer value (a pointer value indicating any one of the buffer surfaces 331-1 to 331-Z) to write audio data. The writing surface selection unit 333 updates (increments) the value of the writing position pointer WP (m), and then writes 10 ms of linear audio data to the buffer surface 331 corresponding to the writing position pointer WP (m). . The pointer value of the write position pointer WP (m) is incremented by one at the write trigger, and becomes 1 at the next write trigger when Z is reached. That is, audio data is cyclically written to the buffer surfaces 331-1 to 331 -Z by the writing surface selection means 333.

  The read position pointer group 332 includes read position pointers RP (i.e., M-1) corresponding to media processing threads 30- (m + 1) to 30- (m + M-1) other than self (media processing thread 30-m). The read position pointer RP) is arranged. For example, in FIG. 6, the read position pointer group 332 of the media processing thread 30-m includes read position pointers RP (1), RP (2),..., RP (m-1), RP (m + 1),. RP (M) is arranged. The read position pointer RP holds a pointer value (a pointer value indicating any of the buffer faces 331-1 to 331 -Z) of audio data to be read by the corresponding media processing thread 30.

  The reading position pointer selecting means 335 selects the reading position pointer RP corresponding to any of the media processing threads 30, updates (increments) the pointer value of the reading position pointer RP, and then selects the selected reading position pointer RP. The pointer value is supplied to the reading surface selection means 334. The reading surface selecting unit 334 performs processing of reading voice data from the buffer surface 331 corresponding to the supplied pointer value and outputting it (output to the media processing thread 30 selected by the reading position pointer selecting unit 335).

  The read position pointer selection means 335 also increments the pointer value of the read pointer value RP by 1 at each read opportunity, as the write plane selection means 333 and reaches 1 at the next read opportunity when it reaches Z. Perform cyclical action.

  For example, in the media processing thread 30- (m-1), after updating the reading position pointer RP (m-1), one unit (10 ms) from the buffer surface corresponding to the reading position pointer RP (m-1) The audio data (linear audio data) of) is read. The reading position pointer RP (m-1) increments by one at the reading opportunity, and when it reaches Z, performs a cyclic operation to be 1 at the next reading opportunity. Similar read processing is performed in the media processing threads 30-1 to 30- (m-2) and 30- (m + 1) to 30-M.

  The addition processing 34 requests and acquires output of audio data (audio data held on the buffer surface 331 corresponding to the pointer value of the reading position pointer RP) to another media processing thread 30 (circulation buffer 33), The voice data obtained by adding (synthesizing) them is generated and supplied to the mixer 35. That is, the addition processing 34-m is for 10 ms from the circulation buffer 33 (circulation buffers 33-1 to (m-1) and circulation buffers 33- (m + 1) to 33-M) other than the media processing thread 30-m. The linear voice data is read, added (synthesized), and 10 ms worth of linear voice data is delivered to the mixer 35-m. Note that various processings for adding voice data (synthesis technology) are applied to processing in which the addition processing 34 adds (synthesizes) voices relating to a plurality of voice data, and generates voice data after addition (post-synthesis). Detailed description is omitted.

  Next, the internal configuration of the mixer 35 will be described.

  FIG. 7 is an explanatory view showing an internal configuration of the mixer 35-m constituting an arbitrary media processing thread 30-m.

  The mixer 35-m includes mixer units 351-1 to 351-N for synthesizing the audio data supplied to the audio transmission processes 36-m_1 to 36-m_N.

  The mixer unit 351-n combines (mixing) the audio data supplied from the audio reception processing 31-m_ (n + 1) to 31-m_ (n + N−1) with the audio data supplied from the addition processing 34-m The generated voice data is generated and supplied to the corresponding voice transmission processing 36. Specifically, for example, as shown in FIG. 7, the mixer unit 351-1 supplies the voice data synthesized in the voice transmission process 36-m_1. Therefore, as shown in FIG. 7, the mixer unit 351-1 synthesizes (mixes) the audio data supplied from the audio transmission processing 36-m_2 to 36-m_N and the audio data supplied from the addition processing 34-m. ) Will be. Note that various voice data synthesis techniques can be applied to the processing in which the mixer unit 351 synthesizes voices relating to a plurality of voice data and generates the voice data after the synthesis, so detailed description will be omitted.

  Thus, the terminal 20-1_1 which is the transmission destination of the voice transmission process 36-1_1 is not the voice data (voice data supplied to the voice reception process 31-1_1) transmitted from the own device (terminal 20-1_1). Audio data obtained by synthesizing (mixing) audio data of all the terminals 20-1-2 to 20-M_N is transmitted. The same applies to all the other terminals 20-1-2 to 20-M_N. As described above, in the multipoint audio mixing apparatus 10, the synthesis is performed by combining audio data obtained by excluding the terminals 20 of the respective transmission destinations with respect to all the M × N terminals 20-1_1 to 20-M_N (mixing ) Can be sent.

(A-2) Operation of the First Embodiment Next, the operation (the audio processing method of the embodiment) of the multipoint audio mixing device 10 of the first embodiment having the configuration as described above will be described.

  FIG. 8 is a timing chart showing an example of write / read timing of the circular buffer 33-m according to this embodiment.

  In the example shown in FIG. 8, the number of buffer faces 331 constituting the media processing thread 30 is illustrated as being eight (ie, Z = 8).

  In FIG. 8A, the periodic processing timing of the media processing thread 30-m, the write timing of the circular buffer 33-m, the change timing of the write position pointer WP (m), and the pointer value are illustrated. In (b), the periodic processing timing of the media processing thread 30- (m-1), the reading timing of the circular buffer 33- (m-1), the change timing of the reading position pointer RP (m-1), and the pointer value are illustrated. It is done. Further, in FIG. 8C, the cycle processing timing of the media processing thread 30- (m + 1), the reading timing of the circular buffer 33- (m + 1), the change timing of the reading position pointer RP (m + 1) and the pointer value are illustrated. There is. The write position pointer WP shown in FIG. 8A is a pointer that belongs to the media processing thread 30-m. Also, the read position pointer RP (m-1) shown in FIG. 8B and the read position pointer RP (m + 1) shown in FIG. 8C both correspond to the read position pointer group 332 of the media processing thread 30-m. It belongs to.

  In FIG. 8, data transfer from the media processing thread 30- (m) to the media processing thread 30- (m-1) and data from the media processing thread 30- (m) to the media processing thread 30- (m + 1) Although the transfer is described, the same applies to data transfer from the media processing thread 30- (m) to the media processing threads 30-1 to 30- (m-2) and 30- (m + 2) to 30-M. So I will omit the detailed explanation.

  Here, it is assumed that the media processing thread 30- (m) periodically starts every 10 ms, for example, and the start cycle is not disturbed. Then, in the media processing thread 30- (m), a write trigger to the circular buffer (m) occurs every 10 ms. At the first write trigger, the write position pointer WP (m) is set to 1, and linear voice data for 10 ms is written to the buffer surface 331-1. The write position pointer WP (m) is incremented and writing of the linear audio data for 10 ms is performed to the corresponding buffer surface 331 at the write trigger after the next time. In the media processing thread 30- (m), the write position pointer WP (m) is set to 1 at the next write timing when the write position pointer WP (m) becomes 8 equal to the number of buffer faces, and 10 ms worth of linear audio data is written.

  Here, it is assumed that the media processing thread 30- (m-1) periodically starts every 10 ms as in the media processing thread 30- (m), and the start-up cycle is not disturbed. Then, in the media processing thread 30- (m-1), a read trigger from the circular buffer 33- (m) occurs every 10 ms.

  For example, the media processing thread 30- (m-1) starts reading processing after writing data to the buffer surface 331-1 of the media processing thread 30- (m).

  After the media processing thread 30- (m) writes the linear audio data for 10 ms to the buffer surface 331-1, when the reading trigger of the media processing thread 30- (m-1) occurs, this is the media processing thread 30- It becomes an opportunity of the first reading in (m-1). Then, in the media processing thread 30- (m), the reading position pointer RP (m-1) is set to 1. Then, the audio data (linear audio data for 10 ms) held on the buffer surface 331-1 of the media processing thread 30-(m) is read into the media processing thread 30-(m−1). In the media processing thread 30- (m), the reading position pointer RP (m-1) is incremented at the reading timing by the media processing thread 30- (m-1) on and after next time, and the corresponding buffer surface 331 It will supply 10 ms worth of linear audio data). The read position pointer RP (m-1) is set to 1 at the next read timing which is 8 (= Z) which is the same as the number (maximum number) of faces of the buffer surface 331.

Here, the media processing thread 30- (m + 1), the media processing threads 3 0- (m), similarly to 30- (m-1), shall be cyclically activated for each 10 ms. However, here, as shown in FIG. 8, in the cycle in which the media processing thread 30- (m + 1) reads the audio data of the buffer side 331-5 of the media processing thread 30- (m), It shall be taken. Furthermore, here, as shown in FIG. 8, in the media processing thread 30- (m + 1), the disorder of the activation cycle is at the cycle of reading the buffer planes 331-5, 331-6, 331-7, and 331-8. Suppose that it occurs. In the processing by software (thread) in the data processing unit 11, for example, the occurrence of waiting for writing data to the hard disk may make the period processing time longer than usual, and the activation period may be disturbed.

  In this case, in the media processing thread 30- (m + 1), the RTP transmission processing is delayed due to the disturbance of the start-up cycle in the cycle in which the buffer planes 331-5, 331-6, 331-7, and 331-8 are read. Can occur, but the desired mixing process can be continued.

(A-3) Effects of the First Embodiment According to the first embodiment, the following effects can be achieved.

  As described above, the multipoint audio mixing apparatus 10 synthesizes (mixes) audio data excluding audio data related to the terminals 20 of each transmission destination with respect to all the M × N terminals 20-1_1 to 20-M_N. ) Can be sent.

  In the data processing unit 11, for example, each media processing thread 30 can be operated in parallel by different CPU cores, so that the performance limit of mixing processing of N channels in one CPU core is conventionally N × M channels The mixing process of can be performed. Therefore, the multipoint audio mixing apparatus 10 of the present invention can process N × M audio data in software on a general-purpose OS. In particular, in the multipoint audio mixing apparatus 10, by providing the addition processing 32, the circulation buffer 33, the addition processing 34, and the mixer 35 in each media processing thread 30, voice data synthesized by another media processing thread 30 is collected Synthesis in stages. As a result, the multipoint audio mixing apparatus 10 does not require a configuration for processing the audio data of the plurality of media processing threads 30 at the center, and only the audio data processing distributed to the plurality of media processing threads 30 is N × M channel It realizes the mixing process.

(B) Second Embodiment Hereinafter, a second embodiment of the audio processing device and the mixing program according to the present invention will be described in detail with reference to the drawings. Hereinafter, an example in which the audio processing device and the mixing program of the present invention are applied to a multipoint audio mixing device will be described.

(B-1) Configuration of Second Embodiment The configuration of the multipoint audio mixing device 10 according to the second embodiment can also be shown using FIGS. 1 to 7 described above. Hereinafter, only differences from the first embodiment will be described with respect to the multipoint audio mixing device 10 of the second embodiment.

  Among the media processing threads 30, the operation timing may be shifted between the media processing threads 30 due to the disturbance of the activation cycle or the like. In that case, in some media processing threads 30, the position of any read position pointer RP may overtake the position of the write position pointer WP and process invalid data in time series. Therefore, in the second embodiment, processing is performed to absorb the operation timing difference between the media processing threads 30 due to the disorder of the start cycle.

  Specifically, in the second embodiment, for example, for the media processing threads 30 other than one or more media processing threads 30-(m) serving as an arbitrary reference, a predetermined number of media processing threads 30-(m) After writing is performed on the buffer side 331 (in this embodiment, as an example, after data writing to the buffer side 331-2), the reading process is started. Furthermore, in the second embodiment, the reading position pointer RP of the media processing thread 30 is written at the reading timing of the audio data of each media processing thread 30 (timing of reading from the circulation buffer 33 of another media processing thread 30). A processing configuration for preventing the position pointer WP (m) from being overtaken (for example, no increment is performed when overtaking) is added.

(B-2) Operation of Second Embodiment Next, the operation (the audio processing method of the embodiment) of the multipoint audio mixing device 10 of the second embodiment having the configuration as described above will be described.

  FIG. 9 is a timing chart showing an example of write / read timing of the circular buffer 33-m according to the second embodiment.

  In the example shown in FIG. 9, the number of buffer faces 331 constituting the media processing thread 30 is illustrated as being eight (ie, Z = 8).

  In FIG. 9A, the cycle processing timing of the media processing thread 30- (m), the write timing of the circular buffer 33-m, the change timing of the write position pointer WP (m), and the pointer value are illustrated. In FIG. 9B, the periodic processing timing of the media processing thread 30- (m-1), the reading timing of the circular buffer 33- (m-1), the change timing of the reading position pointer RP (m-1), and the pointer value Is illustrated. Further, in FIG. 9C, the cycle processing timing of the media processing thread 30- (m + 1), the reading timing of the circular buffer 33- (m + 1), and the change timing and pointer value of the reading position pointer RP (m + 1) are illustrated. There is. The write position pointer WP shown in FIG. 9A is a pointer that belongs to the media processing thread 30- (m). Further, the read position pointer RP (m-1) shown in FIG. 9 (b) and the read position pointer RP (m + 1) shown in FIG. 9 (c) are both read position pointer groups of the media processing thread 30- (m). It belongs to 332.

  In FIG. 9, data transfer from the media processing thread 30- (m) to the media processing thread 30- (m-1) and data from the media processing thread 30- (m) to the media processing thread 30- (m + 1) Although the transfer is described, the same applies to data transfer from the media processing thread 30- (m) to the media processing threads 30-1 to 30- (m-2) and 30- (m + 2) to 30-M. So I will omit the detailed explanation.

  The media processing thread 30-m, for example, periodically starts every 10 ms, but in the cycle of writing the buffer surface 331-4, periodic processing takes time, and buffer surfaces 331-5, 331-6, 331-7, It is assumed that the start-up cycle is disturbed in the cycle of writing 331-8 and 331-1. In the processing by software, for example, the occurrence of waiting for writing data to the hard disk may make the periodic processing time longer than usual, and the start-up period may be disturbed.

  It is assumed that the media processing thread 30- (m-1) periodically starts every 10 ms as in the case of the media processing thread 30-m, and that the start cycle is not disturbed. Then, in the media processing thread 30- (m-1), a read trigger from the circular buffer 33-m occurs every 10 ms.

  Further, in this embodiment, the media processing thread 30 other than the media processing thread 30-m starts reading processing after writing data to the buffer surface 331-2 of the media processing thread 30-m. Therefore, the media processing thread 30- (m-1) starts reading processing after writing data to the buffer surface 331-2 of the media processing thread 30-m. Therefore, as shown in FIG. 9, after the media processing thread 30-m writes voice data (linear voice data) for 10 ms to the buffer surface 331-2, reading of the media processing thread 30- (m-1) is performed. When a trigger occurs, this becomes the first read trigger. At this time, the media processing thread 30-m sets the read position pointer RP (m-1) to 1, and reads 10 ms worth of audio data (linear audio data) from the buffer surface 331-1. The media processing thread 30-m increments the read position pointer RP (m−1) at the next read opportunity and reads 10 ms worth of audio data (linear audio data) from the corresponding buffer surface 331. Become.

  As shown in FIG. 9, the writing position pointer WP of the media processing thread 30-m is used at the first reading timing (the reading timing of the media processing thread 30-m) when the reading position pointer RP (m-1) is 5. The value of m) is 5. Therefore, at this point in time, in the media processing thread 30-m, the latest audio data is written to the buffer surface 331-5, and no audio data has been written to the buffer surface 331-6. Therefore, the media processing thread 30- (m-1) reads the audio data of the buffer side 331-5 with the reading position pointer RP (m-1) as the previous value holding (without incrementing) at this reading opportunity. I will In this case, since the media processing thread 30- (m-1) reads the audio data of the buffer side 331-5 twice in succession, the audio continuity is lost for a moment at this trigger, but the subsequent audio Can maintain continuity.

  In the media processing thread 30- (m-1), instead of reading voice data twice in a row, pseudo artificial voice (voice for dummy) is generated from past voice by executing the same process as packet loss compensation in the decoder. ) May be generated. Then, the continuity of voice can be maintained even at this moment. Then, at the next read timing when the read position pointer RP (m-1) becomes 8 equal to the number of buffer faces, the read position pointer RP (m-1) is set to 1 and the buffer of the media processing thread 30-m Audio data (linear audio data) for 10 ms is read from the surface 331-1.

(B-3) Effects of Second Embodiment According to the second embodiment, the following effects can be achieved in addition to the effects of the first embodiment.

  In the multipoint audio mixing device 10 according to the second embodiment, the reading process of another media processing thread 30 is started after the data processing to the plurality of buffer planes 331 is performed by the media processing thread 30 serving as an arbitrary reference. ing. Further, in the multipoint audio mixing apparatus 10 of the second embodiment, the read position pointer RP is the write position pointer WP (read timing pointer) at the reading timing of the media processing thread 30 (timing of reading the audio data from the other media processing thread 30). m) Since the processing to prevent overtaking is added, the audio continuity of the read data can be maintained.

(C) Other Embodiments The present invention is not limited to the above embodiments, and may include modified embodiments as exemplified below.

  (C-1) In the second embodiment, the processing for preventing the reading position pointer RP from overtaking the writing position pointer WP (m) has been described. However, the difference in the number of buffer faces between the write position pointer WP (m) and the read position pointer RP results in a transmission delay of buffer face number difference × activation cycle, and it is not desirable that this transmission delay becomes too large. Therefore, in each media processing thread 30, delay recovery processing of transmission delay may be performed. For example, each media processing thread 30 increments the read position pointer RP by 2 when "write position pointer WP (m)-read position pointer RP X X" (for example, X = 4). It is also possible to carry out the delay recovery processing of the transmission delay by carrying out the data reading. Furthermore, each media processing thread 30 increments the read position pointer RP by two only in a silent section where there is no audio data when "write position pointer WP (m)-read position pointer RP X X" is satisfied. It is good.

  (C-2) In each of the above embodiments, the voice processing apparatus according to the present invention has been described as an example applied to a multipoint voice mixing apparatus, but may be applied to various apparatuses for receiving and mixing voice data. it can. In each of the above embodiments, an example of receiving a packet of encoded audio data and performing audio mixing processing has been described, but the encoding method and division format of audio data at the time of reception are not limited. It is natural.

  DESCRIPTION OF SYMBOLS 10 ... Multipoint audio mixing apparatus, 11 ... Data processing part, 111 ... Processor, 112 ... Memory, 12 ... Communications part, 40 ... Network, 20, 20-1_1 to 20-M_N ... Terminal, 21 ... Call processing part, 22 ... Communication unit, 23 ... Microphone, 24 ... Speaker, 30, 30-1 to 30-M ... Media processing thread, 31, 31-1_1 to 31-M_N ... Voice reception processing, 311 ... RTP reception, 312 ... Jitter buffer, 313: decoder 32, 32, 32-1 to 32-M: addition process 33, 33-1 to 33-M: circular buffer 34, 34-1 to 34-M: addition process 34, 35, 35-1 35-M: mixer, 36, 36-1_1 to 36-M_N: voice transmission processing, 361: encoder, 362: RTP transmission, 331, 331-1 to 331-Z: buffer surface, 3 2 ... read position pointers, 333 ... writing surface selecting means, 334 ... read plane selecting means, 335 ... read position pointer selecting means, 351,351-1~351-N ... mixer.

Claims (7)

Equipped with multiple voice processing means,
Each of the above audio processing means
A plurality of sound signal receiving means for receiving received sound data based on received sound for each time series;
First combining means for combining the received sound data received by the plurality of sound signal receiving means to generate first combined speech data;
Buffer means for holding the first synthetic speech data;
Second synthesizing means for synthesizing the first synthetic speech data generated by each of the other speech processing means to generate second synthetic speech data;
Transmission sound data is generated by combining the second synthetic speech data with voice data obtained by excluding the reception sound data relating to the transmission destination from the reception sound data received by the plurality of sound signal reception means for each transmission destination. And a third synthesis means ,
The buffer means causes the first synthetic speech data to be stored in chronological order in a circular buffer in which a plurality of buffer units holding the first synthetic speech data in one time series are cyclically arranged.
The voice processing characterized in that the buffer means holds a predetermined number of first synthetic speech data in the circulation buffer and then starts outputting the first synthetic speech data held by the circulation buffer. apparatus. The buffer means has a read pointer for managing the read position of the circular buffer and a write pointer for managing the write position of the circular buffer, so that the position of the read pointer does not exceed the position of the write pointer. The voice processing device according to claim 1 , wherein the position of the read pointer is controlled. The above buffer means does not increment the position of the read pointer when the position of the read pointer is the same as the position of the write pointer when reading the first synthetic speech data corresponding to the read pointer. The speech processing apparatus according to claim 2 , characterized in that: When the buffer means reads the first synthetic speech data corresponding to the read pointer, if the difference between the position of the write pointer and the position of the read pointer is greater than or equal to a threshold value, the read position pointer 2 The speech processing apparatus according to claim 2 , wherein the value is incremented. The voice processing apparatus according to any one of claims 1 to 4 , wherein each of the voice processing means is configured by a thread on a computer. Make the computer function as multiple voice processing means,
Each of the above audio processing means
A plurality of sound signal receiving means for receiving received sound data based on received sound for each time series;
First combining means for combining the received sound data received by the plurality of sound signal receiving means to generate first combined speech data;
Buffer means for holding the first synthetic speech data;
Second synthesizing means for synthesizing the first synthetic speech data generated by each of the other speech processing means to generate second synthetic speech data;
Transmission sound data is generated by combining the second synthetic speech data with voice data obtained by excluding the reception sound data relating to the transmission destination from the reception sound data received by the plurality of sound signal reception means for each transmission destination. have a third combining means,
The buffer means causes the first synthetic speech data to be stored in chronological order in a circular buffer in which a plurality of buffer units holding the first synthetic speech data in one time series are cyclically arranged.
The voice processing characterized in that the buffer means holds a predetermined number of first synthetic speech data in the circulation buffer and then starts outputting the first synthetic speech data held by the circulation buffer. program. In the speech processing method executed by the speech processing device,
Equipped with multiple voice processing means,
Each voice processing means comprises a plurality of sound signal receiving means, first combining means, buffer means, second combining means, and third combining means.
Each of the above sound signal receiving means receives received sound data based on the received sound for each time series;
The first combining means combines the received sound data received by the plurality of sound signal receiving means to generate first combined speech data.
The buffer means holds first synthesized speech data;
The second synthesizing unit synthesizes the first synthesized speech data generated by each of the other speech processing units to generate second synthesized speech data.
The third synthesizing means includes, for each transmission destination, voice data obtained by excluding the received sound data relating to the transmission destination from the received sound data received by the plurality of sound signal receiving means and the second synthesized speech data. Generate synthesized transmit sound data ,
The buffer means causes the first synthetic speech data to be stored in chronological order in a circular buffer in which a plurality of buffer units holding the first synthetic speech data in one time series are cyclically arranged.
The voice processing characterized in that the buffer means holds a predetermined number of first synthetic speech data in the circulation buffer and then starts outputting the first synthetic speech data held by the circulation buffer. Method.

Images