JP5931707B2 - Video conferencing system - Google Patents

Description

  The present invention relates to a video conference technology that emphasizes a specific sound and suppresses noise from acoustic signals collected by a plurality of freely arranged clients.

  In a telephone conference device and a video conference device, a voice emphasis device is used to emphasize a voice of a specific speaker or a voice in a specific direction and suppress noise. The speech enhancement device uses a microphone array that can collect sound by synchronizing sampling frequencies of a plurality of microphones, and obtains an arrival time difference between sounds that reach the microphones. For example, Non-Patent Document 1 describes a beamforming technique that can enhance a sound in a specific direction by estimating the direction of a sound source from the difference in arrival time of sounds and adding the same phase. Non-Patent Document 2 describes a technique for emphasizing the voice of a specific speaker by distinguishing the speaker from the sound source direction and designing a filter that maximizes the power ratio between the voice of the specific speaker and other noises. ing.

Yusuke Hioka, Kazunori Kobayashi, Ken’ichi Furuya, and Akitoshi Kataoka, “Enhancement of Sounds in a Specific Directional Area using Power Spectra Estimated from Multiple Beamforming Outputs”, HSCMA, pp.49-52, 2008. Moto Ogasawara, Kentaro Ishizuka, Akiko Araki, Masaki Fujimoto, Tomohiro Nakatani, Kazuhiro Otsuka, “Online Conference Speech Enhancement Using SN Ratio Maximized Beamformer”, Proc. -698, 2009.

  In a conventional speech enhancement apparatus used in a video conference, a method of classifying acoustic signal sections based on a sound source direction using a microphone array has been used. In such a method, the sampling frequency for recording with a plurality of microphones needs to be synchronized, and the relative positional relationship between the microphones needs to be known. Therefore, a dedicated video conference terminal equipped with a microphone array is required.

  In view of these points, the present invention emphasizes a specific sound and suppresses noise by using an acoustic signal obtained by freely arranging a general-purpose terminal device without using a dedicated video conference terminal. It aims to provide video conferencing technology that can be.

  In order to solve the above-described problems, a video conference system according to the present invention includes a plurality of clients and a server including sound collection means and photographing means.

  The client includes an audio acquisition unit that transmits to the server the acoustic signal collected by the sound collection unit, a video acquisition unit that transmits the video signal captured by the imaging unit to the server, and an audio that reproduces the distribution acoustic signal received from the server. A reproduction unit and a video display unit for displaying a distribution video signal received from the server are included.

  The server receives an audio signal of a plurality of channels received from a plurality of clients, arbitrarily selected from an audio processing unit that generates a transmission audio signal by predetermined audio processing, and a plurality of video signals received from a plurality of clients A video selection unit that determines a transmission video signal, an audio distribution unit that transmits a plurality of channels of audio signals as input to a distribution audio signal generated by predetermined audio processing, and a plurality of video signals. And a video distribution unit that transmits the distributed video signal to a plurality of clients.

  The speech processing unit receives a plurality of channels of acoustic signals, and a feature amount sequence acquisition unit that obtains a feature amount obtained by normalizing the magnitude of the acoustic signal of the speech section for each channel by the magnitude of the acoustic signal of the non-speech section; Clustering feature value sequences consisting of feature values obtained for multiple channels, classifying unit to determine the signal section classification to which the feature value sequence belongs, and converting acoustic signals to frequency domain in each of multiple time sections A spectrum calculation unit that obtains a plurality of amplitude spectra and phase spectra, and emphasizes the amplitude spectrum corresponding to the feature amount sequence belonging to the enhanced signal section classification which is one of the signal section classifications with respect to the plurality of amplitude spectra; An emphasis processing unit that performs processing and obtains a plurality of post-processing amplitude spectra, and a phase addition unit that assigns a phase spectrum to the post-processing amplitude spectrum to obtain a complex spectrum. No.

  According to the video conferencing technique of the present invention, it is possible to enhance and extract a specific speaker's voice and suppress noise regardless of the arrangement of clients and the difference in microphone sensitivity. Also, by acquiring video as well as voice from a camera mounted on the client, video of a user who is mainly talking can be distributed to each client and a video conference can be performed.

The figure which illustrates the function structure of a video conference system. The figure which illustrates the function structure of a server and a client. The figure which illustrates the function structure of an audio | voice processing part. The figure which illustrates the structure of a client. The figure which illustrates the processing flow of a server and a client. The figure which illustrates the structure of a video selection part. The figure which illustrates the processing flow of an audio | voice processing part.

  The present invention processes the sound signals of a plurality of channels acquired from a plurality of clients including sound collection means having various sampling frequencies and microphone sensitivities so as to listen to a voice in which a specific speaker is emphasized and noise is suppressed. It is a video conferencing system that can. Furthermore, the present invention acquires, by the imaging means, the sound acquired by the sound collecting means currently included in the client is mainly collected by sending a video signal from the imaging means included in the client to the server. And a technology for selecting an image to be distributed for a video conference.

  Hereinafter, embodiments of the present invention will be described in detail. In addition, the same number is attached | subjected to the component which has the same function in drawing, and duplication description is abbreviate | omitted.

[Embodiment]
This embodiment is a hands-free video conference system using acoustic signals acquired from a plurality of smartphones that are freely arranged and have different sampling frequencies and microphone sensitivities.

<Configuration of video conference system>
With reference to FIG. 1, the structural example of the video conference system 10 of this embodiment is demonstrated. The video conference system 10 includes a server 1 and K (> 1) clients 3 ₁ ,..., 3 _K. The server 1 and the clients 3 ₁ ,..., 3 _K are connected to the network 5. The network 5 only needs to be configured so that the connected devices can communicate with each other. For example, the network 5 can be configured with the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), or the like. The physical medium constituting the network 5 may be a wireless LAN, a mobile phone line typified by 3G or LTE, Bluetooth (registered trademark), etc., regardless of whether it is wired or wireless.

The video conference system 10 realizes a video conference by linking at least two sets. As shown in FIG. 1, when implementing video conferencing by the video conferencing system 10 _A and a video conferencing system 10 _B, the client 3 ₁ of a video conferencing system 10 _A, ..., 3 _K are arranged space and video conferencing system 10 Each of the clients 3 ₁ ,..., 3 _{K of B} is a different space. The server 1 of the video conference system 10 _{A and} the server 1 of the video conference system 10 _B are both connected to the network 6. The network 6 only needs to be configured so that the connected devices can communicate with each other. For example, the network 6 can be configured by the Internet, a WAN (Wide Area Network), a public switched telephone network, a dedicated line, or the like. May be the same as the number of clients 3 included in the number of video conferencing system 10 _B of the client 3 included in the video conferencing system 10 _A, it may be different. In this embodiment, a case where there are two sets of video conference systems 10 will be described as an example. However, three or more sets of video conference systems 10 may be used to join three or more spaces in one video conference.

A configuration example of the server 1 and the clients 3 ₁ ,..., 3 _K according to this embodiment will be described with reference to FIG. The server 1 includes K audio receiving units 11 ₁ ,..., 11 _K , K video receiving units 12 ₁ ,..., 12 _K , an audio processing unit 13, an audio transmission unit 14, an image selection unit 15, and an image transmission unit. 16, a terminal management unit 17, an audio acquisition unit 21, an audio distribution unit 22, a video acquisition unit 23, a video distribution unit 24, and a command transmission unit 25. The server 1 of this embodiment is a special device configured by reading a predetermined program into a known computer including, for example, a CPU (central processing unit), a RAM (random access memory), and the like. Data input to the server 1 and processed data are stored in a memory (not shown) and are read from the processing unit as necessary.

The clients 3 ₁ ,..., 3 _K are respectively a sound acquisition unit 31, a video acquisition unit 32, an audio reproduction unit 33, a video display unit 34, a command reception unit 35, a setting unit 36, a control unit 37, and a sound collecting means such as a microphone. 301, an imaging unit 302 such as a camera, a reproduction unit 303 such as a speaker, and a display unit 304 such as a liquid crystal display. The positions of the clients 3 ₁ ,..., 3 _K and their relative positions may be unknown or may be known. The clients 3 ₁ ,..., 3 _K operate independently of each other. The microphone sensitivities of the sound collecting means 301 may be different from each other or the same. Further, the sampling frequencies of the sound collection means 301 may be different from each other or the same. The clients 3 ₁ ,..., 3 _K may be any devices as long as they have sound collection means, photographing means, reproduction means, and display means. Specific examples of the clients 3 ₁ ,..., 3 _K are general-purpose terminal devices such as smartphones, mobile phone terminals, and laptop computers.

  With reference to FIG. 3, the example of a structure of the audio | voice processing part 13 is demonstrated in detail. The audio processing unit 13 includes an input unit 101, a sampling frequency conversion unit 102, a signal synchronization unit 103, a frame division unit 104, a VAD determination unit 105, a non-audio power storage unit 106, an S / N vector generation unit 107 (feature quantity sequence acquisition) Section), vector classification section 108 (classification section), spectrum calculation section 109, amplitude spectrum storage section 110, phase spectrum storage section 111, filter coefficient calculation section 112 (enhancement processing section), filter coefficient storage section 113, filtering section 114 ( Emphasis processing unit), phase applying unit 115, and time domain converting unit 116.

A configuration example of the clients 3 ₁ ,..., 3 _K will be described more specifically with reference to FIG. This specific example is an example in which the clients 3 ₁ ,..., 3 _K are configured as smartphones. The client 3 _k (1 ≦ k ≦ K) has a built-in microphone 301 near the lower end of the main unit, a camera 302 as a photographing unit near the upper end of the main unit, and a speaker 303 as a reproducing unit. The rectangular liquid crystal display 304 is arranged as a display means over a wide range on the front surface of the main body. The liquid crystal display is desirably a touch panel having a position input function, but may include other pointing devices such as a numeric keypad and a cross key. In this embodiment, the liquid crystal display 304 is divided into a plurality of areas so that different information can be displayed. In this embodiment, description will be made using an example in which display area 1 (3041), display area 2 (3042), operation area (3043), and frame frame (3044) are divided.

<Processing of video conference system>
With reference to FIG. 5, the operation example of the video conference system 10 of this embodiment is demonstrated.

The setting unit 36 included in the client 3 _k (1 ≦ k ≦ K) holds setting information necessary for the client 3 _k to operate, and provides setting information to each unit as necessary. Setting information includes, for example, specify or IP address of the server 1, voice receiving unit 11 ₁ provided in the server 1, ..., 11 _K and image receiver 12 _1, ..., 12 _K port number, and the like. In addition, if the client 3 _k is a smartphone, selection of whether to use a front (front) camera or a back (rear) camera as the photographing unit 302 may be included. The setting information can be input from a setting screen presented to the user through the display unit 304. The setting screen may be activated, for example, by displaying a “menu” button in the operation area (3043) in response to a menu selection by the user.

Control unit 37 included in the client 3 _k transmits a control signal for instructing start or stop of processing for each unit of the client 3 _k. Client 3 _k of the user may select a control signal to be transmitted by a button operation displayed on the client 3 _k of the operation area (3043). Specifically, the client 3 _k displays a “start” button in the operation area (3043) immediately after the activation, and the user presses the button to obtain the audio acquisition unit 31, the video acquisition unit 32, the audio reproduction unit 33, and the video. A control signal for instructing the display unit 34 to start is transmitted. At the same time, the “start” button is redrawn as a “stop” button. The audio acquisition unit 31, the video acquisition unit 32, the audio reproduction unit 33, and the video display unit 34 that have received the control signal for instructing start start the respective processes. Thereafter, when the user presses a “stop” button, a control signal for instructing stop is transmitted to the audio acquisition unit 31, the video acquisition unit 32, the audio reproduction unit 33, and the video display unit. The audio acquisition unit 31, the video acquisition unit 32, the audio reproduction unit 33, and the video display unit 34 that have received the control signal for instructing stop stop the respective processes. In the following description, it is understood that the audio acquisition unit 31, the video acquisition unit 32, the audio reproduction unit 33, and the video display unit 34 have already received a control signal instructing start from the control unit 37 and have started processing. It is assumed.

Terminal management unit 17 provided in the server 1, the client 3 ₁ doing the communication, ..., to manage the number of 3 _K. When communication from the client 3 _k (1 ≦ k ≦ K) is started to the audio receiving unit 11 _k or the video receiving unit 12 _k , the communication start notification is received and the IP address of the connection source client 3 _k is managed. Add as If you can not already added, such as managed exceeds the number of connectable client, voice receiving unit 11 _1, ..., 11 _K and image receiver 12 _1, ..., to the effect that unmanageable to 12 _K Notice. Accordingly, the terminal management unit 17 holds the latest value of the IP addresses of the clients 3 ₁ ,..., 3 _K with which the server 1 is communicating.

The sound acquisition unit 31 included in the client 3 _k performs A / D conversion on the sound observed using the sound collection unit 301 at each sampling frequency, and converts the digital acoustic signal x _k (i _k ) at a plurality of sample points. Obtain (step S31). Here, i _k is an integer index representing a sampling point in the time domain. That is, x _k (i _k ) represents the digital acoustic signal at the sample point represented by the index i _k . The format of the digital audio signal x _k (i _k ) may be any format, but in this embodiment, it is a monaural PCM (pulse code modulation) with a sampling frequency of 16 kHz and a quantization bit number of 16 bits.

A processing sequence for performing processing corresponding to the digital acoustic signal x _k (i _k ) obtained by the voice acquisition unit 31 of the client 3 _k is referred to as a channel k. That is, channel k handles values obtained from digital acoustic signal x _k (i _k ) and digital acoustic signal x _k (i _k ). In this embodiment, there are K channels k = 1,.

The acquired digital acoustic signal x _k (i _k ) is transmitted to the server _1A . Although a known communication protocol can be used as the transmission protocol, RTP (Real-time Transport Protocol) is used in this embodiment. The destination port number at the time of transmission is a port number preset by the setting unit 37. Specifically, if the client 3 ₁ , the 6100 port of the server 1, the client 3 ₂ the 6102 port of the server 1, and the client 3 _k of the server 1 (6100 + 2 (k−1)) As in the case of the port number, they may be assigned regularly in order. However, the present invention is not limited to this, and an arbitrary port number can be assigned as long as it does not overlap on the server 1. In addition, the data length in the case of transmitting by RTP can be arbitrarily set, but may be set in units of 20 milliseconds, for example.

The voice receiving unit 11 _k included in the server 1 receives the digital acoustic signal x _k (i _k ) transmitted from the client 3 _k (step S11). The voice receiving unit 11 _k notifies the terminal management unit 17 of the IP address of the transmission source at the start of communication. When there is a response that the terminal management unit 17 cannot manage the received packet, the received packet is discarded and the process is terminated. The K channel digital acoustic signals x ₁ (i ₁ ),..., X _K (i _K ) received by the audio receivers 11 ₁ ,..., 11 _K are input to the audio processor 13. The audio receiving unit 11 _k may input the digital acoustic signal x _k (i _k ) to the audio processing unit 13 while buffering as necessary.

The video acquisition unit 32 included in the client 3 _k acquires the digital video signal v _k (i _k ) in the format specified using the photographing unit 302 (step S32). Here, i _k is an integer index representing a sampling point in the time domain. That is, v _k (i _k ) represents the digital video signal at the sample point represented by the index i _k . The format of the digital video signal v _k (i _k ) may be any format, but in this embodiment, a still image having a predetermined size is periodically taken. More specifically, JPEG-compressed still images of QVGA size (320 x 240 pixels) are taken every 100 milliseconds.

The acquired digital video signal v _k (i _k ) is transmitted to the server 1. Although a known communication protocol can be used as the transmission protocol, UDP (User Datagram Protocol) is used in this embodiment. The destination port number at the time of transmission is a port number preset by the setting unit 37. Specifically, if the client 3 ₁ , the 6200 port of the server 1, the client 3 ₂ the 6202 port of the server 1, and the client 3 _k of the server 1 (6200 + 2 (k−1)) As in the case of the port number, they may be assigned regularly in order. However, the present invention is not limited to this, and an arbitrary port number can be assigned as long as it does not overlap on the server 1. A sequence number is assigned to the UDP packet to be transmitted. It is desirable that the image size to be captured does not exceed 64 KB, which is the upper limit of the packet size in the UDP protocol, so that one image is not divided into a plurality of UDP packets.

The video receiver 12 _k included in the server 1 receives the input digital video signal v _k (i _k ) transmitted from the client 3 _k (step S12). The video reception unit 12 _k notifies the terminal management unit 17 of the IP address of the transmission source at the start of communication. When there is a response that the terminal management unit 17 cannot manage the received packet, the received packet is discarded and the process is terminated. The K channel digital video signals v ₁ (i ₁ ),..., V _K (i _K ) received by the video receivers 12 ₁ ,..., 12 _K are input to the video selector 15. In this embodiment, every time a JPEG-compressed still image is obtained from the received UDP packet, it is input to the video selection unit 15.

The voice processing unit 13 included in the server 1 emphasizes a specific voice by performing predetermined voice processing on the K channel digital acoustic signals x ₁ (i ₁ ),..., X _K (i _K ), and generates noise. A suppressed transmission acoustic signal y _c (n) is generated (step S13). Here, n is an integer index representing a sample point. c is a classification label number representing a signal section classification (emphasis signal section classification) for emphasizing sound. The classification label number will be described later. Details of “predetermined audio processing” performed by the audio processing unit 13 will also be described later. The transmission acoustic signal y _c (n) is input to the voice transmission unit 14.

The audio transmission unit 14 included in the server 1 transmits the input transmission acoustic signal y _c (n) to the server 1 included in another video conference system 10 participating in the video conference (step S14). When a video conference is performed using three or more sets of video conference systems 10, the same transmission acoustic signal y _c (n) may be transmitted to each server 1.

Video selection unit 15 provided in the server 1, the client 3 ₁ K heights, ..., 3 K number of digital video signals v ₁ received from _K (i _1), ..., arbitrarily selected from v _K (i _K) The transmission video signal w (n) is determined (step S15). Here, n is an index representing a sample point. The transmission video signal w (n) is input to the video transmission unit 16. Selection of the transmission video signal w (n), the display unit such as a display provided in the server 1 (not shown) or the client 3 _1, ..., are performed by the user operation on the display unit 304 with one of the 3 _K is .

With reference to FIG. 6, an operation example for selecting the transmission video signal w (n) will be described. The example of FIG. 6 is an example in the case of operating on the display unit included in the server 1. The video selection unit 15 obtains the number of clients in communication among the clients 3 ₁ ,..., 3 _K from the terminal management unit 17, and the digital video signal v ₁ (i ₁ ),. v Reduce the _K (i _K ) to generate an operation screen in which an iconized image is embedded. The IP address of the server 1 is displayed in the “server IP number” field on the operation screen. The client 3 ₁ in the "received image" column, ..., 3 digital video signals v ₁ received from _K (i _1), ..., v side by side image obtained by reducing the _K (i _K) is displayed. The digital video signal _{v 1 (i 1), ...} , v client 3 ₁ corresponding to the vicinity of the _K (i _K), ..., and displays the IP address of 3 _K. When any image in the “received image” column is selected by the user's operation, the transmission video signal w (n) is determined. When there is one client connected to the server 1, that one may be automatically selected. The selected transmission video signal w (n) is displayed in the “transmission image” field.

When the transmission video signal w (n) is selected by the user's operation on the display unit 304 included in the client 3 _k , an operation screen similar to the operation screen example shown in FIG. 6 is displayed in the display area 2 (3042) shown in FIG. The video selection unit 15 may be notified of the video selected by the user's operation.

  The video transmission unit 16 included in the server 1 transmits the input transmission video signal w (n) to the server 1 included in the other video conference system 10 participating in the video conference (step S16). When a video conference is performed using three or more sets of video conference systems 10, the same transmission video signal w (n) may be transmitted to each server 1.

The audio acquisition unit 21 included in the server 1 receives the distribution acoustic signal y ′ _c (n) transmitted by the audio transmission unit 14 of the server 1 included in another video conference system 10 that participates in the video conference (step S21). . The received distribution acoustic signal y ′ _c (n) is input to the audio distribution unit 22. When a video conference is performed using three or more sets of video conference systems 10, a plurality of distributed audio signals received from other video conference systems 10 can be multiplexed to be distributed audio signals y ′ _c (n). Good.

The audio distribution unit 22 included in the server 1 transmits the distribution acoustic signal y ′ _c (n) to the _K clients 3 ₁ ,..., 3 _K (step S22). Although a known communication protocol can be used as the transmission protocol, RTP (Real-time Transport Protocol) is used in this embodiment. The IP address of the transmission destination is acquired from the terminal management unit 17, and the port number is a preset port number. Any port number can be used as long as it does not overlap on the client _3k . For example, in this embodiment, the 6004th port is assumed. In addition, the data length in the case of transmitting by RTP can be arbitrarily set, but may be set in units of 20 milliseconds, for example.

The client 3 _k, the distribution sound signal y by the audio reproduction unit 33 'receives the _c (n), using the reproduction unit 303, the distribution sound signal y' Play _c (n) (step S33). The sound reproducing unit 33 may reproduce the distribution acoustic signal y ′ _c (n) as necessary, while buffering it.

  The video acquisition unit 23 included in the server 1 receives the distribution video signal w ′ (n) transmitted by the video transmission unit 16 of the server 1 included in another video conference system 10 that participates in the video conference (step S23). The received distribution video signal w ′ (n) is input to the video distribution unit 24. When a video conference is performed using three or more sets of video conference systems 10, a plurality of distribution video signals received from other video conference systems 10 are combined into a single video signal by a method such as frame division and distributed. The video signal w ′ (n) may be used.

The video distribution unit 24 included in the server 1 transmits the distribution video signal w ′ (n) to the _K clients 3 ₁ ,..., 3 _K (step S24). Although a known communication protocol can be used as the transmission protocol, UDP (User Datagram Protocol) is used in this embodiment. The IP address of the transmission destination is acquired from the terminal management unit 17, and the port number is a preset port number. Any port number can be used as long as it does not overlap on the client _3k . For example, in this embodiment, port 6001 is assumed. A sequence number is assigned to the UDP packet to be transmitted.

The client 3 _k receives the distribution video signal w ′ (n) by the video display unit 34 and reproduces the distribution video signal w ′ (n) using the display unit 304 (step S34). In this embodiment, each time a single JPEG compressed still image is obtained from the received UDP packet, the image is displayed on the display unit 304.

The command transmission unit 25 included in the server 1 transmits a request signal based on the transmission video signal w (n) selected by the video selection unit 15 to the _K clients 3 ₁ ,..., 3 _K (step S25). Request signal based on the transmission video signal w (n) is, for example, the client 3 _1, ..., 3 K number of digital video signal received from the _{K v 1 (i 1),} ..., v of _K (i _K) , Information indicating which video signal is selected. Specifically, the first color to the color of the framework surrounding the display means 304 to the client 3 _k which have taken the selected video signal as a transmission video signal w (n) (3044) (e.g., A request signal indicating red), and for a client 3 _k that is capturing a video signal that has not been selected as a transmission video signal w (n), a frame frame (3044) surrounding the display means 304 is displayed. A request signal indicating that the color is the second color (for example, blue) is transmitted. Specifically, a character string in which the RGB value of the color to be drawn in the frame (3044) (for example, “# ff0000” for red, “# 0000ff” for blue, etc.) is set according to a predetermined text format. As a request signal. Although a known communication protocol can be used as the transmission protocol, UDP (User Datagram Protocol) is used in this embodiment. The IP address of the transmission destination is acquired from the terminal management unit 17, and the port number is a preset port number. Any port number can be used as long as it does not overlap on the client _3k . For example, in this embodiment, the port number is 6000. When transmitting by UDP, it is desirable to repeatedly transmit the same request signal periodically in consideration of packet loss, for example, every 20 milliseconds.

The command receiving unit 35 included in the client 3 _k displays information based on the transmission video signal w (n) on the display unit 304 based on the request signal received from the server 1 (step S35). For example, when the video signal v _k (i _k ) captured by the client 3 _k is selected as the transmission video signal w (n), the frame frame (3044) surrounding the display unit 304 is set to the first color ( For example, if the video signal v _k (i _k ) displayed in red) and the client 3 _k captured is not selected as the transmission video signal w (n), a frame frame (3044) surrounding the display unit 304 is displayed. ) In a second color (eg, blue). At this time, if the determined text format does not match or the specified RGB value is an invalid value exceeding 255, the request signal is discarded and the process is terminated. With this configuration, a user who uses the client 3 _k can send a video captured by the client 3 _k (in many cases, to a client included in another video conference system 10 participating in the video conference. It is possible to know whether or not the user's face is displayed.

<Processing details of the voice processing unit>
With reference to FIG. 7, the operation example of the audio | voice processing part 13 is demonstrated in detail.

To an input unit 101, K pieces of voice receiving unit 11 _1, ..., 11 input digital audio signals of K channels received by _{K x 1 (i 1),} ..., x K (i K) is input (step S101 ). The received digital acoustic signals x _k (i _k ) of the plurality of channels k = 1,..., K are input to the sampling frequency converter 102. Since the input digital acoustic signals x _k (i _k ) of different channels k are obtained by different clients 3 _k , the sampling frequencies may be different. The sampling frequency conversion unit 102 aligns the sampling frequencies of the input digital acoustic signals x _k (i _k ) of all channels k = 1,..., K to any same sampling frequency. In other words, the sampling frequency conversion unit 102 converts the input digital acoustic signal x _k (i _k ) of the plurality of channels k = 1,..., K to the sampling frequency, and converts the converted digital acoustic signal cx _k ( i _k ) is _obtained for a plurality of channels k = 1,. The “specific sampling frequency” may be any one of the clients 3 ₁ ,..., 3 _K , or may be another sampling frequency. An example of “specific sampling frequency” is 16 kHz. The sampling frequency conversion unit 102 performs sampling frequency conversion based on the nominal value of the sampling frequency of each client 3 _k . That is, the sampling frequency conversion unit 102 converts a signal sampled at the nominal value of the sampling frequency of each client 3 _k into a signal sampled at a specific sampling frequency. Such sampling frequency conversion is well known. The sampling frequency converter 102 outputs the converted digital acoustic signal cx _k (i _k ) of each channel k obtained as described above (step S102).

The signal synchronizer 103 receives the converted digital acoustic signals cx ₁ (i ₁ ),..., Cx _K (i _K ) of the channels k = 1,. The signal synchronization unit 103 synchronizes the converted digital acoustic signals cx ₁ (i ₁ ),..., Cx _K (i _K ) between the channels k = 1,. Signals sx ₁ (i ₁ ),..., Sx _K (i _K ) are obtained and output (step S103). The details will be described below.

There are individual differences in client _3k . Therefore, even if the nominal value of the sampling frequency of the client 3 _k is f _k , the client 3 _k may perform A / D conversion at the sampling frequency f _k / α _k . However, α _k is a positive parameter representing a frequency shift between the actual sampling frequency of the client 3 _k and the nominal value of the sampling frequency. If a digital acoustic signal obtained by A / D converting the acoustic signal at the sampling frequency f _k is x _k ′ (i _k ), the same acoustic signal is obtained by A / D conversion at the sampling frequency f _k / α _k. The digital acoustic signal to be obtained is x _k ′ (i _k × α _k ). However, “×” represents a multiplication operator. That is, the frequency deviation of the sampling frequency appears as a timing deviation in the time domain of the digital acoustic signal.

The sampling frequency conversion unit 102 performs sampling frequency conversion based on the nominal value f _k of the sampling frequency of each client 3 _k . That is, assuming that “specific sampling frequency” common to all channels k = 1,..., K is F, the sampling frequency conversion unit 102 performs sampling frequency conversion that increases the sampling frequency of each channel k by F / f _k times. It is carried out. Therefore, if the actual sampling frequency of each client 3 _k is f _k / α _k , the sampling frequency of the converted digital acoustic signal cx _k (i _k ) of each channel k is F × α _k . The frequency shift based on the individual difference appears as a timing shift in the time domain of the converted digital acoustic signal cx _k (i _k ) between the channels k = 1,.

Signal synchronization unit 103 in order to reduce the conversion timing shift in the time domain digital audio signal cx _k (i _k) based on individual difference, converts digital audio signals in the time domain _{cx 1 (i 1), ...} , cx K (i _K ) is synchronized between channels k = 1,. For example, the signal synchronizer 103 shifts the converted digital acoustic signals cx ₁ (i ₁ ),..., Cx _K (i _K ) in the time axis direction (sample point direction) so that the cross-correlation between channels is maximized. , Sx _K (i _K ) are obtained after the digital audio signals sx ₁ (i ₁ ),.

For example, the signal synchronizer 103 can sample from the converted digital acoustic signal cx _k (i _k ) of each channel k with a length (for example, 3 seconds) of a sample string cx _k ( 1),..., Cx _k (I) are taken out (step S1031). However, I represents a positive integer. Next, the signal synchronizer 103 uses the sample sequence cx _{k ′} (1),..., Cx _{k ′} (I) of one channel k′∈ {1,. (Step S1032). Next, the signal synchronizer 103 converts the sample sequences cx _{k ″} (1),..., Cx _{k ″} (I) of channels k ″ ∈ {1,..., K} (k ″ ≠ k ′) other than the channel k ′. Sample sequence cx _{k ″} (1 + τ _{k ″} ), ..., cx _{k ″} (I + τ _{k ″} ) shifted to the time axis and reference sample sequence cx _k ′ (1),…, cx _k ′ (I) A delay τ _{k ″} that maximizes the cross-correlation Σ _n {cx _{k ″} (n) × cx _{k ′} (n)} is searched from a predetermined search range, and sx _{k ″} (i _{k ″} ) = cx _{k ″} ( i _{k ″} + τ _{k ″} ) and sx _{k ′} (i _{k ′} ) = cx _{k ′} (i _{k ′} ) (step S1033). Further, the signal synchronizer 103 sets the sample sequence cx _k (1),. The range in which cx _k (I) is cut out is shifted (for example, by shifting only the sample point corresponding to the time of 1 second), and the processing of executing the processing of steps S1031 to S1033 is repeated, and the synchronized digital acoustic signal sx ₁ (i ₁ ), ..., sx _K (i _K ) are obtained and output for all sample points.

The frame dividing unit 104 receives the digital audio signals sx ₁ (i ₁ ),..., Sx _K (i _K ) after synchronization as inputs. The frame dividing unit 104 divides the digital acoustic signal sx _k (i _k ) into frames that are predetermined time intervals for each channel k (step S104). In this frame division processing, the frame cutout section length (frame length) L point and the shift width m point of the cutout section can be arbitrarily determined. However, L and m are positive integers. For example, the cut-out section length is 2048 points, and the shift width of the cut-out section is 256 points. The frame dividing unit 104 cuts out and outputs a digital acoustic signal sx _k (i _k ) having a cut-out section length for each channel k. Further, the frame division unit 104 shifts the cutout section according to the determined shift width of the cutout section, and repeats the process of cutting out and outputting the digital audio signal sx _k (i _k ) having the cutout section length for each channel k. Through the above processing, a digital audio signal of each frame is output for each channel k. Hereinafter, the digital acoustic signal belonging to the r-th frame r of the channel k is expressed as sx _k (i _{k, r, 0} ),..., Sx _k (i _{k, r, L−1} ).

The VAD determination unit 105 receives as input the digital acoustic signals sx _k (i _{k, r, 0} ),..., Sx _k (i _{k, r, L−1} ) belonging to each frame r of each channel k. The VAD determination unit 105 determines whether each frame r of each channel k is a speech section or a non-speech section using the input digital acoustic signal (step S105). The VAD determination unit 105 determines whether the frame r is a speech segment or a non-speech segment using a well-known technique as described in Reference 1, for example.
[Reference 1] Jongseo Sohn, Nam Soo Kim, Wonyong Sung, “A Statistic Model-Based
Voice Activity Detection, ”IEEE SIGNAL PROCESSING LETTERS, VOL.6, NO.1, 1999.

Based on these determinations, the VAD determination unit 105 assigns to each frame r a label θ _r that indicates a determination result as to whether the frame is a speech segment or a non-speech segment. For example, when the “number of channels determined that the frame r is a speech segment” is equal to or greater than the “number of channels determined that the frame r is a non-speech segment”, the VAD determination unit 105 determines that the frame r is It is determined that it is a voice section, and a label θ _r indicating that it is a voice section is given to the frame r. On the other hand, when the “number of channels determined that the frame r is a speech segment” is less than the “number of channels determined that the frame r is a non-speech segment”, the VAD determination unit 105 determines that the frame r is It is determined that it is a non-speech segment, and a label θ _r indicating that it is a non-speech segment is _assigned to the frame r. In addition, among the channels k = 1,..., K, the average power and average S / N ratio of the digital acoustic signals sx _k (i _{k, r, 0} ),..., Sx _k (i _{k, r, L-1} ) A label θ _r indicating the determination result for the channel with the largest may be attached to the frame r. In addition, an example of a label indicating a voice section is θ _r = 1, and an example of a label indicating a non-voice section is θ _r = 0. VAD decision unit 105 outputs each label theta _r.

The S / N vector generation unit 107 obtains the digital acoustic signal sx _k (i _{k, r, 0} ),..., Sx _k (i _{k, r, L−1} ) and the label θ _r for each frame r of each channel k. Receive as input. The S / N vector generation unit 107 obtains a feature amount obtained by normalizing the magnitude of the digital acoustic signal in the speech section for each channel k with the magnitude of the digital acoustic signal in the non-speech section, and obtains the channel k = 1,. An S / N vector (feature quantity sequence) having the obtained feature quantity as an element is obtained and output (step S107). An example of the “feature amount” is a value representing a ratio of the magnitude of the digital acoustic signal in the voice section to the magnitude of the digital acoustic signal in the non-voice section. Examples of “digital audio signal magnitude” include the power and absolute value of the digital audio signal, the average and absolute value of the digital audio signal power, and the total and absolute value of the digital audio signal power. , Their inverted values and function values. An example of the “feature representing the ratio” is “the ratio of the magnitude of the digital acoustic signal in the speech section to the magnitude of the digital acoustic signal in the non-speech section” itself, its reciprocal, and other function values. In the following, the average value of the power of the digital audio signal is referred to as “digital audio signal magnitude”, and “the ratio of the digital audio signal magnitude in the audio section to the digital audio signal magnitude in the non-audio section” itself is “characteristic”. An example of “quantity” is shown.

  The S / N vector generation unit 107 executes the following processing.

[Step S1071]
The S / N vector generation unit 107 initializes r to 1.

[Step S1072]
The S / N vector generation unit 107 determines whether the label θ _r represents a speech section or a non-speech section.

[Step S1073]
When the label θ _r represents a non-speech interval, the S / N vector generation unit 107 performs the digital acoustic signal sx _k (i _{k, r, 0} ), belonging to the frame r for all channels k = 1,. ..., sx _k (i _{k, r, L-1} ) average power P _N (k, r) is calculated (see equation (1)), and average power P _N (k, r) is calculated as the k-th element. The average power vector P _N (r) = (P _N (1, r),..., P _N (K, r)) is stored in the non-speech power storage unit 106.

[Step S1074]
When the label θ _r represents a speech section, the S / N vector generation unit 107 calculates the average power vector P _N (r ′) = (P of the frame r ′ in the non-speech section stored in the non-speech power storage unit 106. _N (1, r '), ..., P _N (K, r')) are taken out. This frame r ′ is preferably close to the frame r to be processed. For example, the S / N vector generation unit 107 extracts the average power vector P _N (r ′) of the frame r ′ in the non-speech interval closest to the frame r. The non-speech power storage unit 106 also stores an initial value of the average power vector. An example of the initial value of the average power vector is a vector having K constants (for example, 1) as elements. When the average power vector of the non-speech section is not obtained, the S / N vector generation unit 107 extracts the initial value of the average power vector from the non-speech power storage unit 106, and obtains it as P _N (r ′) = (P _N (1, r '), ..., P _N (K, r')).

Further, the S / N vector generation unit 107 performs digital acoustic signals sx _k (i _{k, r, 0} ),..., Sx _k (i _k ) belonging to the frame r in the speech section for all channels k = 1,. _{, r, L-1} ) is divided by P _N (k, r ′) to obtain a normalized average power P _V (k, r) (see equation (2)).

By dividing by P _N (k, r ′), the average power of the digital acoustic signal of each channel k can be normalized, and the influence due to the difference in microphone sensitivity of each channel k can be eliminated. The S / N vector generation unit 107 uses the obtained normalized average power P _V (k, r) as the k-th element, and the S / N vector P _V (r) = (P _V (1, r),. , P _V (K, r)) is output.

[Step S1075]
If there is an unprocessed digital acoustic signal, the S / N vector generation unit 107 sets a value obtained by adding 1 to r to a new r, and the process proceeds to step S1072. When there is no unprocessed digital acoustic signal, the process of the S / N vector generation unit 107 is finished.

As described above, the non-speech power storage unit 106 stores the initial value of the average power vector and the average power vector P _N (r) obtained by the S / N vector generation unit 107.

The vector classification unit 108 receives as input a plurality of S / N vectors P _V (r) (a feature amount sequence made up of feature amounts obtained for a plurality of channels). The vector classification unit 108 clusters a plurality of input S / N vectors P _V (r), and determines a signal section classification (cluster) to which each S / N vector P _V (r) belongs (step S108). . Vector classifying portion 108, a plurality of S / N vector P _N (r) (e.g., at intervals corresponding to five seconds S / N vector P _N (r)) each time the input is newly input Clustering may be performed by adding the S / N vector P _N (r) to the clustering target, or each time one S / N vector P _N (r) is input, a new input is performed. Clustering may be performed by adding the S / N vector P _N (r) to the clustering target. An example of clustering is online clustering that is unsupervised learning, and one example is leader-follower clustering (see, for example, Reference 2). The cosine similarity can be used for the distance that is an index for clustering. The distance function of cosine similarity can be defined as follows.

However, CL is a label of each cluster, and the label CL takes a value (for example, an integer of 1 or more) other than a label θ _r (for example, 0) representing a non-voice segment. P _CL is the centroid vector of the cluster CL. d (CL) represents the distance between the centroid vector P _CL of the cluster CL and the input S / N vector P _V (r). A label CL obtained by clustering using the cosine similarity as a distance function represents a signal section classification to which the input S / N vector P _V (r) belongs. Vector classifying portion 108 substitutes the label CL obtained for the input S / N vector P _V (r) to the label theta _r updating the label theta _r. As a result, the label θ _r of the frame r in the voice section becomes the value of the label CL, and the label θ _r of the frame r in the non-voice section becomes a value representing the non-voice section. The vector classification unit 108 outputs the label θ _r of each frame r.
[Reference 2] Richard O. Duda, Peter E. Hart, David G. Stork, “Pattern Classication,” Wiley-Interscience, 2000.

The spectrum calculation unit 109 divides by the frame division unit 104 and belongs to each frame r of each channel k. The digital acoustic signal sx _k (i _{k, r, 0} ),..., Sx _k (i _{k, r, L− 1} ) is received as input. Here, a K-dimensional vertical vector whose element is the digital acoustic signal sx _k (i _{k, r, j} ) of each channel k in the frame r is x (j, r) = [sx ₁ (i _{1, r, j} ), ..., sx _K (i _{K, r, j} )] ^T However, [η] ^T represents transposition of [η]. Further, a K-dimensional vertical vector whose element is a value obtained by converting the elements of the K-dimensional vector x (0, r),..., X (L-1, r) belonging to the frame r into the frequency domain is represented by X ( f, r). That is, the value X (k, f,) obtained by converting the digital acoustic signal sx _k (i _{k, r, 0} ),..., Sx _k (i _{k, r, L-1} ) belonging to the frame r into the frequency domain. A K-dimensional vertical vector having r) as the k-th element is described as a spectrum vector X (f, r) = [X (1, f, r),..., X (K, f, r)] ^T. Here, f is an index representing a discrete frequency. An example of the method of transforming to the frequency domain is discrete Fourier transform such as FFT (Fast Fourier Transform). Further, a K-dimensional vertical vector having the amplitude spectrum A (k, f, r) of X (k, f, r) as the k-th element is expressed as an amplitude spectrum vector A (f, r) = [A (1, f , r), ..., A (K, f, r)] ^T. Further, a K-dimensional vertical vector having the phase spectrum φ (k, f, r) of X (k, f, r) as the k-th element is expressed as a phase spectrum vector φ (f, r) = [φ (1, f , r),..., φ (K, f, r)] ^T. The spectrum calculation unit 109 converts x (j, r) = [sx ₁ (i _{1, r, j} ),..., Sx _K (i _{K, r, j} )] ^T to the frequency domain, and for each frame r , Amplitude spectrum vector A (f, r) consisting of k amplitude spectra A (k, f, r) and phase spectrum vector φ (f, r) consisting of k phase spectra φ (k, f, r) ) And output (step S109).

  The amplitude spectrum vector A (f, r) is stored in the amplitude spectrum storage unit 110, and the phase spectrum vector φ (f, r) is stored in the phase spectrum storage unit 111.

The filter coefficient calculation unit 112 receives the label θ _r of each frame r output from the vector classification unit 108 and the amplitude spectrum vector A (f, r) read from the amplitude spectrum storage unit 110 as inputs. Here, among the values (classification label numbers) that the label θ _r can take, the classification label number representing the signal section classification (emphasis signal section classification) for emphasizing the sound is set as c. Only one classification label number c may be set, or a plurality of classification label numbers c may be set. For example, the classification label number c may be arbitrarily determined, or one or more signal section classifications selected in descending order of the average value or the total value of the norms of the S / N vector P _V (r) to which it belongs are emphasized. The classification label number c may be determined as the section classification, or the classification label number c is defined as a signal section classification in which the average value or the total value of the norms of the S / N vector P _V (r) to which it belongs exceeds the threshold value. May be determined. θ _r = c represents that the frame r is classified into the enhanced signal section classification.

The filter coefficient calculation unit 112 calculates a filter coefficient for filtering that enhances the amplitude spectrum A (k, f, r) corresponding to the S / N vector P _V (r) belonging to the enhanced signal section classification (step S112). ). In the S / N ratio maximizing beamformer disclosed in Reference 3 below, the eigenvector for the maximum eigenvalue is obtained as a filter coefficient using the complex spectrum as it is. On the other hand, the filter coefficient calculation unit 112 of this embodiment forms an S / N ratio maximizing beamformer using the amplitude spectrum vector A (f, r). That is, the filter coefficient calculation unit 112 solves the generalized eigenvalue problem of the following equation (4), and uses the eigenvector value corresponding to the maximum eigenvalue γ (f) as the filter coefficient w that emphasizes the speech of each classification label number c. _c Obtain as (f).

E [ρ] _{θr = c} (subscript θr is θ _r ) represents a matrix composed of the expected values of the elements of the matrix ρ in the section composed of the frame r with θ _r = c. E [ρ] _{θr ≠ c} represents a matrix composed of expected values of elements of the matrix ρ in the section composed of the frame r where θ _r ≠ c. The section for obtaining equations (5) and (6) corresponds to a time of 10 seconds or more, for example. The filter coefficient w _c (f) is a K-dimensional horizontal vector [w _c (f, 1), ..., w _c (f) with the coefficient w _c (f, k) corresponding to the channel k as the k-th element. , K)]. The filter coefficient calculation unit 112 obtains and outputs a filter coefficient w _c (f) for each index f and each classification label number c. Further, the filter coefficient calculation unit 112 corresponds to the largest element among the elements of the S / N vector P _V (r) of each frame r with θ _r = c in the section for obtaining (5) and (6). The channel is obtained as the maximum channel number k _{c, r} . The filter coefficient calculation unit 112 associates the filter coefficient w _c (f) and the maximum channel number k _{c, r} with each classification label number c and stores them in the filter coefficient storage unit 113. In order to cope with the movement of the speaker and the change in noise, the filter coefficient calculation unit 112 updates the intervals for obtaining the equations (5) and (6) periodically (for example, every one minute), and each filter coefficient w _c (f) and the maximum channel number k _{c, r} are obtained, and each filter coefficient w _c (f) and the maximum channel number k _{c, r} stored in the filter coefficient storage unit 113 are updated.
[Reference 3] HL Van Tree, ed., “Optimum Array Processing,” Wiley, 2002.

The filtering unit 114 receives the filter coefficient w _c (f) read from the filter coefficient storage unit 113 and the amplitude spectrum vector A (f, r) read from the amplitude spectrum storage unit 110 as inputs. The filtering unit 114 applies the filter coefficient w _c (f) to the plurality of amplitude spectra A (1, f, r),..., A (K, f, r) constituting the amplitude spectrum vector A (f, r). ) = [w _c (f, 1),..., w _c (f, K)] to obtain a processed amplitude spectrum A _c ′ (f, r) and output it (step S114). For example, the filtering unit 114 sets the inner product of the filter coefficient w _c (f) and the amplitude spectrum vector A (f, r) as the processed amplitude spectrum A _c ′ (f, r) as shown in the following equation (7). obtain.
A _c '(f, r) = w _c (f) A (f, r) (7)

Through the above steps S112 and S114, the S / N vector P _V (r) belonging to the emphasized signal section classification is applied to the plurality of amplitude spectra A (1, f, r),..., A (K, f, r). A process of emphasizing the amplitude spectrum corresponding to is performed, and a plurality of processed amplitude spectra A _c ′ (f, r) are obtained.

The phase assigning unit 115 assigns a phase spectrum corresponding to the processed amplitude spectrum A _c ′ (f, r) to obtain a complex spectrum and outputs it (step S115). In this embodiment, the phase assigning unit 115 reads the maximum channel number k _{c, r} corresponding to each frame r and each classification label number c from the filter coefficient storage unit 113. Phase deposition unit 115, the phase spectrum corresponding to all channels k from the phase spectrum storage section 111 φ (k, f, r ) read out, up from their channel number k _c, the phase spectrum corresponding to the _r phi (k _{c, r} , f, r). Further, the phase adding unit 115 receives the processed amplitude spectrum A _c ′ (f, r) output from the filtering unit 114 as an input. The phase assigning unit 115 assigns the phase spectrum φ (k _{c, r} , f, r) to the processed amplitude spectrum A _c ′ (f, r) as shown in the following equation (8), and the complex spectrum Y _c ( f, r) is obtained and output.
Y _c (f, r) = A _c '(f, r) exp (iφ (k _{c, r} , f, r)) (8)
Where i is an imaginary unit and exp is an exponential function.

The time domain conversion unit 116 receives the complex spectrum Y _c (f, r) as an input, converts the complex spectrum Y _c (f, r) to the time domain, and enhances the acoustic signal y _c (n, r) (n = 0, ..., L-1). Here, n is an index representing a sample point. As a method for converting to the time domain, for example, inverse Fourier transform can be used. Further, the time domain conversion unit 116 synthesizes the enhanced acoustic signal y _c (n, r) (n = 0,..., L−1) using the overlap add method to obtain and output the acoustic signal waveform in the time domain. To do. When there are a plurality of classification label numbers c, the time domain conversion unit 116 outputs a plurality of acoustic signal waveforms corresponding to the classification label numbers c. Or you may output the addition value for every same sample point of the acoustic signal waveform corresponding to each classification label number c.

  The speech processing unit 13 of this embodiment clusters a plurality of S / N vectors obtained by normalizing the magnitude of the digital acoustic signal in the speech section with the magnitude of the digital acoustic signal in the non-speech section. Therefore, it is possible to perform signal section classification based on the sound source position from digital sound signals collected by a plurality of terminal devices such as a plurality of smartphones and laptop computers provided with sound collecting means with different sensitivity. it can.

Further, in the sound processing unit 13 of this embodiment, the cosine similarity is used as a distance measure used for clustering in order to pay attention to the attenuation of sound pressure from the sound source to the sound collecting means 301. Further, in this embodiment, the sampling frequency conversion unit 102 performs sampling frequency conversion to correct a sampling frequency shift between channels, and the signal synchronization unit 103 performs synchronization between channels to perform client 3 ₁ ,. _The effect of individual differences in _K was suppressed. Therefore, even if the nominal values of the sampling frequencies of the sound collecting means 301 of each channel are different from each other or there are individual differences in the sampling frequencies, the signal section classification can be performed with high accuracy.

  Since the section classification result as described above can be used to classify the target sound section and other sound source sections, it can be used as information for designing a filter that suppresses noise and emphasizes the target sound. Therefore, in this embodiment, a specific target sound is emphasized from digital acoustic signals obtained by a plurality of terminal devices such as a smartphone, a mobile phone terminal, and a laptop computer, which have a plurality of freely arranged sampling frequencies and microphone sensitivities. can do.

<Modifications>
The present invention is not limited to the embodiment described above. For example, if the nominal values of the sampling frequencies of the sound collection means 301 of all channels k = 1,..., K are the same, the processing of the sampling frequency conversion unit 102 may not be performed. In this case, the “input digital acoustic signal” may be directly input to the signal synchronization unit 103 as the “converted digital acoustic signal”. In such a case, the sampling frequency conversion unit 102 may not be provided.

  Furthermore, if the nominal values of the sampling frequencies of the sound collecting means 301 of all the channels k = 1,..., K are the same and the influence of their individual differences is small, the sampling frequency converter 102 and the signal synchronizer 103 It is not necessary to perform the process. In this case, the “input digital audio signal” may be directly input to the frame dividing unit 104 as the “digital audio signal”. In such a case, the sampling frequency conversion unit 102 and the signal synchronization unit 103 need not be provided.

In addition, the phase assigning unit 115 assigns the phase spectrum φ (k _{c, r} , f, r) corresponding to the maximum channel number k _{c, r} to the processed amplitude spectrum A _c ′ (f, r). However, the phase spectrum φ (k, f, r) of other channels may be added to the processed amplitude spectrum A _c ′ (f, r).

  The present invention is not limited to the above-described embodiment, and it goes without saying that modifications can be made as appropriate without departing from the spirit of the present invention. The various processes described in the above-described embodiments are not only executed in time series according to the order described, but may be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes.

<Effect of the invention>
As described above, according to the video conferencing technique of the present invention, even if an acoustic signal is collected by a plurality of freely arranged clients using sound collecting means such as microphones having different sampling frequencies and sensitivities. It is possible to enhance / extract speaker's voice and suppress noise. Further, by acquiring not only voice but also video from photographing means such as a camera mounted on the client, video of a user who is mainly talking can be distributed to each client and a video call can be performed.

1 server 3 client 5 network 10 video conference system 11 audio receiving unit 12 video receiving unit 13 audio processing unit 14 audio transmitting unit 15 video selecting unit 16 video transmitting unit 17 terminal managing unit 21 audio acquiring unit 22 audio distributing unit 23 video acquiring unit 24 video distribution unit 25 command transmission unit 31 audio acquisition unit 32 video acquisition unit 33 audio reproduction unit 34 video display unit 35 command reception unit 36 setting unit 37 control unit 301 sound collection unit 302 reproduction unit 303 photographing unit 304 display unit

Claims (7)

A video conference system including a plurality of clients and a server including sound collection means and photographing means,
The client
A client-side voice acquisition unit that transmits an acoustic signal collected by the sound collecting unit to the server;
A video acquisition unit that transmits a video signal captured by the imaging unit to the server;
An audio reproduction unit for reproducing a distribution acoustic signal received from the server;
A video display unit for displaying a distribution video signal received from the server;
Including
The server
An audio processing unit that receives an audio signal of a plurality of channels received from the plurality of clients and generates a transmission audio signal by predetermined audio processing;
An audio transmission unit for transmitting the transmission acoustic signal to a server included in another video conference system;
A video selection unit for determining a transmission video signal arbitrarily selected from a plurality of video signals received from the plurality of clients;
A server-side audio acquisition unit for receiving the distributed audio signal generated by the predetermined audio processing with an audio signal of a plurality of channels as an input from a server included in the other video conference system;
An audio distribution unit for transmitting the distribution acoustic signal to the plurality of clients;
A video distribution unit for transmitting the distribution video signal arbitrarily selected from a plurality of video signals to the plurality of clients;
Including
The voice processing unit
A feature quantity sequence acquisition unit that receives the acoustic signals of the plurality of channels and obtains a feature quantity obtained by normalizing the magnitude of the acoustic signal in the voice section for each channel by the magnitude of the acoustic signal in the non-voice section;
A clustering unit configured to cluster feature quantity sequences made up of the feature quantities obtained for the plurality of channels, and to determine a signal section classification to which the feature quantity sequence belongs;
A spectrum calculation unit that converts the acoustic signal into a frequency domain in each of a plurality of time intervals, and obtains a plurality of amplitude spectra and phase spectra;
Emphasis processing for obtaining a plurality of post-processing amplitude spectra by performing processing for emphasizing the amplitude spectrum corresponding to the feature amount sequence belonging to the emphasis signal section classification which is one of the signal section classifications with respect to the plurality of amplitude spectra. And
A phase adding unit that obtains a complex spectrum by adding the phase spectrum to the processed amplitude spectrum;
A time domain conversion unit that converts the complex spectrum into the time domain to obtain the transmission acoustic signal;
Including video conferencing system. The video conferencing system according to claim 1,
The server includes a command transmission unit that transmits a request signal for displaying predetermined information to the plurality of clients based on the transmission video signal,
The video conference system, wherein the client includes a command receiving unit that displays predetermined information based on the transmission video signal according to the request signal. The video conferencing system according to claim 2,
The video receiving system, wherein the command receiving unit displays information indicating which client has acquired the transmitted video signal. The video conference system according to any one of claims 1 to 3,
The video conference system, wherein the client is a smartphone. The video conference system according to any one of claims 1 to 4,
The enhancement processing unit
A filter coefficient calculation unit for calculating a filter coefficient for filtering that emphasizes an amplitude spectrum corresponding to a feature amount sequence belonging to the enhancement signal section classification;
Filtering the plurality of amplitude spectra with the filter coefficient to obtain the processed amplitude spectrum;
Including video conferencing system. The video conference system according to any one of claims 1 to 5,
The video conference system, wherein the feature amount represents a ratio of a magnitude of the acoustic signal in the voice section to a magnitude of the acoustic signal in the non-voice section. The video conference system according to any one of claims 1 to 6,
The voice processing unit
Sampling frequency conversion of the acoustic signals of the plurality of channels to obtain a converted acoustic signal of a specific sampling frequency; and
A signal synchronization unit that synchronizes the converted acoustic signal between channels and obtains the acoustic signal;
Including video conferencing system.

Images