JP6587088B2 - Audio transmission system and audio transmission method - Google Patents

Description

  The present invention relates to a sound transmission system and a sound transmission method for performing predetermined signal processing and transmitting a sound collected in a sound space in which a plurality of sounds are mixed.

  In a space where a plurality of sounds are mixed (hereinafter referred to as “sound space”), a plurality of microphones are used to pick up the sound, and a sound source (sound signal) in a specific direction is extracted from the picked up sound signal. A sound source separation technique has been studied (see, for example, Patent Document 1).

  The sound source separation device shown in Patent Document 1 specifies independent component analysis that separates sounds without using information on the arrangement of microphones in consideration of the effects of sound diffraction, and specifies a target direction that is weak to sound diffraction. A filtering process that can extract the sound is executed in parallel, the sound power spectrum of the filtering process result is used as a reference signal, a sound having a high time correlation with the reference signal is regarded as a sound having a high noise suppression performance, and the reference A sound having a high time correlation with the signal is selected as a sound source separation signal. As a result, the sound source separation device can perform sound separation in a short time even when the sound is diffracted on the housing, such as a mobile phone.

JP 2011-199447 A

  In Patent Document 1, sound in an arbitrary direction is obtained from sound waveform information (audio signal) collected by a microphone array having a plurality of microphones, corresponding to a housing (for example, a mobile phone) whose product size is defined in advance. Can be extracted.

  However, in the configuration of Patent Document 1, since the number and arrangement of microphones assuming the size of the product are premised, in an actual sound space in which environmental sounds including many speakers and noise are mixed, a plurality of microphones are used. There is a problem that it is difficult to extract a specific sound source from the collected sound.

  In order to solve the above-described problems, the present invention extracts a specific sound source from collected sound in an actual sound space in which environmental sounds including many speakers and noise are mixed, It is an object of the present invention to provide a voice transmission system and a voice transmission method for transmitting high-quality voice to the own terminal or a call partner of the own terminal.

The present invention is an audio transmission system including a first communication terminal that accepts a predetermined input operation , wherein the one or more sound collection units collect sound in a sound space including a plurality of sounds, and the one A storage unit that stores the sound data of the sound collected by the sound collection unit, and the sound data stored in the storage unit is converted into the same number of types of sound data as the one or more sound collection units. separated, a separation unit for generating a respective isolation audio data, among the individual separation audio data generated by the separating unit, either separate sound corresponding to the predetermined input operation to the first communication terminal the data includes a first transmission unit for transmitting to the first communication terminal as the switching voice data, wherein the first communication terminal, you output sound receiving the switching voice data is voice transmission system .

The present invention is also an audio transmission method in an audio transmission system including a first communication terminal that accepts a predetermined input operation, in one or more sound collection devices arranged in a sound space including a plurality of sounds. Collecting sound, storing sound data of the sound collected by the one or more sound collecting devices, and collecting the sound data in the same number of types as the one or more sound collecting devices. Separating into voice data and generating individual separated voice data, and among the separated separated voice data after separation , any one of the separated voice data according to the predetermined input operation to the first communication terminal A voice transmission method comprising: transmitting to the first communication terminal as switched voice data; and receiving the switched voice data and outputting the voice.

  According to the present invention, since a specific sound source can be extracted from the collected sound in an actual sound space in which a lot of speakers and environmental sounds including noise are mixed, high-quality sound after extraction can be obtained. Can be transmitted to the own terminal or a call partner of the own terminal.

The block diagram which shows an example of the system configuration | structure of the audio transmission system of this embodiment The block diagram which shows an example of an internal structure of the communication terminal of this embodiment Explanatory drawing which shows typically the 1st use case example of the audio | voice transmission system of this embodiment. Explanatory drawing which shows typically the 2nd use case example of the audio | voice transmission system of this embodiment. Explanatory drawing which shows typically the 3rd use case example of the audio | voice transmission system of this embodiment. The flowchart explaining an example of the operation | movement outline | summary along the time series of the audio | voice transmission system in the 3rd use case example shown in FIG. Explanatory drawing which shows typically the 4th use case example of the audio transmission system of this embodiment. The flowchart explaining the operation | movement procedure of the 1st operation example of the audio | voice transmission system of this embodiment. The flowchart explaining the operation | movement procedure of the 2nd operation example of the audio | voice transmission system of this embodiment. The flowchart explaining the operation | movement procedure following FIG.

  Hereinafter, an embodiment (hereinafter referred to as “the present embodiment”) that specifically discloses a voice transmission system and a voice transmission method according to the present invention will be described in detail with reference to the drawings.

  In the first operation example of the present embodiment, the sound transmission system picks up sound in one or more microphones arranged in a sound space AR in which a plurality of sounds (for example, voices of a plurality of people) are mixed and included, Audio data of audio collected by one or more microphones is stored in a location (for example, a server device on a cloud network) different from the sound space AR. The sound transmission system separates stored sound data into sound data of the same number as one or more microphones (hereinafter referred to as “separated sound data”) by a predetermined input operation by a user of the communication terminal (sound source). Separation), any one of the separated audio data after the sound source separation is transmitted to the communication terminal.

  Further, in the second operation example of the present embodiment, the audio transmission system further includes one or more audio transmission systems arranged in a sound space AR including a mixture of a plurality of sounds (for example, voices of a plurality of people). The microphone picks up the voice and stores the voice data of the voice picked up by the one or more microphones in a place different from the sound space AR (for example, a server device on the cloud network network). The voice transmission system separates the stored voice data into the same number of types of voice data as one or more microphones (hereinafter referred to as “separated voice data”) by a predetermined input operation by the user of the first communication terminal. (Sound source separation), and by a predetermined input operation by the user of the first communication terminal, any one of the separated voice data is transmitted to the second communication terminal which is a call partner of the first communication terminal.

  Hereinafter, the details of the audio transmission system of the present embodiment will be specifically described. Hereinafter, the user who uses the communication terminal in the first operation example and the first communication terminal in the second operation example (refer to the communication terminal T1 shown in FIG. 1) is referred to as “user A”, and the second in the second operation example. The user who uses the communication terminal (see the communication terminal T2 shown in FIG. 1) is referred to as “user B”.

  FIG. 1 is a block diagram illustrating an example of a system configuration of a voice transmission system 100 according to the present embodiment. In the first operation example described above, the audio transmission system 100 illustrated in FIG. 1 includes the communication terminal T1, one or more microphones M1, M2, M3, and M4 arranged in the sound space AR, and a microphone signal multiplexing transmission device MX. And a server device CDS provided on the cloud network NW1 and a first selection unit SL1 provided on the LTE network NW2. In the second operation example described above, the voice transmission system 100 includes a communication terminal T2 and a first selection unit SL1 provided on the LTE network NW2 in addition to the configuration corresponding to the first operation example described above. Furthermore, it is the structure which contains.

  Microphones M1 to M4, which are examples of the sound collection unit or the sound collection device, are arranged at predetermined positions in the sound space AR, respectively, collect sound in which a plurality of sounds are mixed in the sound space AR, and perform microphone signal multiplexing transmission. Output to device MX.

  The microphone signal multiplex transmission device MX aggregates and multiplexes the audio data collected by the microphones M1 to M4, and the multiplexed audio data (audio signal) is provided on the cloud network NW1. It transmits to the server device CDS. Note that the microphone signal multiplex transmission device MX may be disposed in the sound space AR as long as it is electrically connected to the microphones M1 to M4, or may be disposed in a place other than the sound space AR.

  The server device CDS is provided on a cloud network NW1 as an example of the Internet network or an intranet network, and includes at least a sound signal processing transmission unit PR. In FIG. 1, only the sound signal processing transmission unit PR is shown as a minimum configuration necessary for explaining the operation of the server device CDS of the present embodiment, and the audio signal transmitted from the microphone signal multiplex transmission device MX is shown. The receiving unit for receiving is not shown.

  The sound signal processing transmission unit PR includes at least a control unit PR1, a sound source separation unit PR2, and a memory (not shown) as an example of a storage unit. The controller PR1 and the sound source separator PR2 are configured using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a DSP (Digital Signal Processor). The sound signal processing transmission unit PR demultiplexes the audio signal multiplexed in the microphone signal multiplexing transmission device MX.

  The control unit PR1 receives a control signal (see below) for sequentially switching separated audio data generated after sound source separation by the sound source separation unit PR2 for each predetermined input operation of the users A and B with respect to the communication terminals T1 and T2. Entered. In response to the input of this control signal, the control unit PR1 outputs an instruction for sequentially switching the separated audio data transmitted from the server device CDS to the sound source separation unit PR2. The sound signal processing transmission unit PR transmits the separated audio data output from the sound source separation unit PR2 to the first selection unit SL1 or the second selection unit SL2 in accordance with the switching instruction of the control unit PR1.

  The sound source separation unit PR2 performs sound source separation using the sound data after demultiplexing stored in the memory, thereby providing the same number of types of separated sound data (that is, microphones) as one or more microphones (for example, microphones M1 to M4). When there are as many sound sources as the number of arrangements in the sound space AR, the sound data of the sound output from each sound source is generated. Note that the sound source separation process in the sound source separation unit PR2 is a known technique. For example, the method disclosed in Patent Document 1 described above or the method disclosed in other known techniques may be used. I will omit the details.

  The memory functions as a work memory when the control unit PR1 and the sound source separation unit PR2 are in operation, and further stores audio data of the audio signal after being demultiplexed by the sound signal processing transmission unit PR.

  The first selection unit SL1 and the second selection unit SL2 are configured by, for example, radio base stations provided on an LTE network NW2 as an example of a public telephone network. The network in which the first selection unit SL1 and the second selection unit SL2 are arranged is not limited to LTE (Long Term Evolution), but may be a 3G (3rd Generation) network or an HSPA (High Speed Packet Access) network.

  The first selection unit SL1 receives one of the separated sound data c1 instructed by the control unit PR1 among the separated sound data corresponding to one or more sound sources generated by the sound source separation unit PR2. The first selection unit SL1 transmits any one of the separated voice data c1 to the communication terminal T1 via the LTE network NW2.

  In addition, when the users A and B are calling using the communication terminals T1 and T2, respectively, the first selection unit SL1 calls the user B's call voice transmitted from the communication terminal T2 (in other words, the server device CDS). The voice data b of the live call voice that has not been separated by the sound source) is received, and any one of the separated voice data c1 or the voice data b of the call voice of the user B is transmitted to the communication terminal T1.

  The second selection unit SL2 receives one of the separated sound data c2 instructed by the control unit PR1 among the separated sound data corresponding to one or more sound sources generated by the sound source separation unit PR2. The second selection unit SL2 transmits any separated voice data c2 to the communication terminal T2 via the LTE network NW2.

  In addition, when the users A and B are making a call using the communication terminals T1 and T2, respectively, the second selection unit SL2 transmits the call voice of the user A transmitted from the communication terminal T1 (in other words, the server device CDS). The voice data a) (raw call voice that has not been sound source separated) is received, and any one of the separated voice data c2 or the voice data a of the call voice of the user A is transmitted to the communication terminal T2.

  The communication terminals T1 and T2 use a sound source separation service in the sound transmission system 100 (that is, a service for separating the sound data of the sound collected in the sound space AR into the same number of types of sound data as one or more microphones). Is a wireless communication device that is operated by users A and B who use the mobile phone and has at least a wireless communication function and a telephone call function. The communication terminals T1 and T2 are, for example, mobile phones, smartphones, or cordless telephone handsets.

  Also, applications AP1 and AP2 (simply abbreviated as “application” in FIG. 1) for using the sound source separation service are installed in advance in the communication terminals T1 and T2. In other words, as will be described later with reference to FIG. 2, the programs and data of the applications AP1 and AP2 for using the sound source separation service are stored in advance in the communication terminals T1 and T2, and the users A and B When using the sound source separation service, the applications AP1 and AP2 are activated according to the input operations of the users A and B.

  FIG. 2 is a block diagram illustrating an example of an internal configuration of the communication terminals T1 and T2 according to the present embodiment. The communication terminal T1 shown in FIG. 2 includes a sound collection unit 11, a CPU 12, a memory 13, an input unit 14, a display unit 15, an audio output unit 16, and a wireless communication unit 17 to which an antenna 18 is connected. It is the composition which includes. Similarly, the communication terminal T2 illustrated in FIG. 2 includes a sound collection unit 21, a CPU 22, a memory 23, an input unit 24, a display unit 25, an audio output unit 26, and a wireless communication unit to which an antenna 28 is connected. 27. In the following, since the configurations of the communication terminals T1 and T2 are the same, the description of each unit illustrated in FIG. 2 illustrates the communication terminal T1, and the operation of the communication terminal T2 is different from the operation of the communication terminal T1. Will be described.

  The sound collection units 11 and 21 are voices of conversation when the users A and B hold the communication terminals T1 and T2 (ie, handset calls), or the users A and B hold the communication terminals T1 and T2. The voice of the call when a call (for example, a hands-free call) is made without being picked up and output to the CPU 12. Note that the voice data of the call voice collected by the sound collection units 11 and 21 is an analog signal, but is converted into a digital signal by an ADC (Analog Digital Converter, not shown) included in the sound collection units 11 and 21. Input to the CPUs 12 and 22.

  The CPUs 12 and 22 perform control processing for overall control of operations of the respective units of the communication terminals T1 and T2, data input / output processing with other units, data calculation (calculation) processing, and data storage processing. I do. For example, when the wireless communication units 17 and 27 receive signals (notification signals) for informing the sound source separation service via the antennas 18 and 28, the CPUs 12 and 22 are applications for using the sound source separation service. AP1 and AP2 are activated. Further, after starting the applications AP1 and AP2 for using the sound source separation service, the CPUs 12 and 22, for example, when the map information of the sound space AR is stored in the memories 13 and 23 in advance, the sound space AR Map information is displayed on the display units 15 and 25.

  The memories 13 and 23 are configured using, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a nonvolatile or volatile semiconductor memory, and function as a work memory when the CPUs 12 and 22 are operated. Predetermined programs (for example, applications AP1 and AP2) and data for operating the CPUs 12 and 22 are stored.

  The input units 14 and 24 are user interfaces (UI) for receiving input operations of the users A and B and notifying the CPUs 12 and 22, and are arranged corresponding to the screens of the display units 15 and 25, for example. The touch panel or the touch pad that can be operated by the fingers of the users A and B or the stylus pen is used.

  The display units 15 and 25 are configured using, for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence), and use, for example, a sound source separation service by a predetermined input operation on the input units 14 and 24 of the users A and B. The screens of the applications AP1 and AP2 for displaying are displayed.

  The voice output units (for example, speakers) 16 and 26, for example, according to the input operation of the users A and B, either the separated voice data c1 transmitted from the first selection unit SL1 or the voice data b of the call voice of the user B Is output.

  The wireless communication unit 17 is connected to an antenna 18 for wireless communication (for example, LTE, 3G, HSPA, Wi-fi (registered trademark), Bluetooth (registered trademark)), and is transmitted from the first selection unit SL1. The separated voice data c1 or the voice data b of the call voice of the user B is received. The wireless communication unit 27 is connected to an antenna 28 for wireless communication (for example, LTE, 3G, HSPA, Wi-fi (registered trademark), Bluetooth (registered trademark)), and is transmitted from the second selection unit SL2. The separated voice data c2 or the voice data a of the call voice of the user A is received.

  In addition, the wireless communication unit 17 receives the control signal generated by the CPU 12 in order to sequentially switch the separated audio data generated after the sound source separation by the sound source separation unit PR2 in response to an input operation on the input unit 14 by the user A. 18 to the server device CDS. In response to an input operation on the input unit 24 by the user B, the wireless communication unit 27 switches the CPU 22 to switch between the separated voice data c2 transmitted from the second selection unit SL2 or the voice data a of the call voice of the user A. Is transmitted to the server device CDS via the antenna 28.

  Next, various use case examples by the sound source separation service in the voice transmission system of the present embodiment will be described with reference to FIGS. In order to simplify the explanation of various use case examples, in FIGS. 3, 4, 5, and 7, the microphone signal multiplexing transmission device MX shown in FIG. 1, the first selection unit SL1, and the second selection are used. Illustration of the LTE network NW2 including the part SL2 is omitted.

(First use case example)
FIG. 3 is an explanatory diagram schematically illustrating a first use case example of the audio transmission system 100 according to the present embodiment. In the first use case example, a plurality of microphones M1a, M2a, M3a, M4a, and M5a are arranged in a sound space AR (for example, an interview room), and an English interpreter Tr1 around a speaker Ps1a that speaks Japanese. Assume a situation in which the conversation content of the speaker Ps1a is simultaneously translated into multiple languages by a French interpreter Tr2 and a Chinese interpreter Tr3. In this case, the server device CDS stores voice data in which at least the Japanese voice of the speaker Ps1a, the English voice of the interpreter Tr1, the French voice of the interpreter Tr2, and the Chinese voice of the interpreter Tr3 are collected. The

  In the first use case example shown in FIG. 3, the user Ps2a of the communication terminal T1 in which the application AP1 for using the sound source separation service is installed may or may not be in the sound space AR.

  The server device CDS uses the predetermined input operation of the user Ps2a to the communication terminal T1, and the sound data collected by the microphones M1a to M5a in the sound space AR is converted into the same number of types of sound sources as the number of microphones (for example, , Japanese voice, English voice, French voice, Chinese voice, and other voices), and one of the separated voice data (for example, English voice) is transmitted to the communication terminal T1. The communication terminal T1 sequentially switches and outputs the separated voice data transmitted from the server device CDS for every predetermined input operation of the user Ps2a. Thereby, the user Ps2a listens to the content of the speaker Ps1a who is speaking in Japanese in the interview room which is the sound space AR in a language (for example, English) that the user Ps2a can understand in high quality. Can do.

(Second use case example)
FIG. 4 is an explanatory diagram schematically showing a second use case example of the audio transmission system 100 of the present embodiment. In the second use case example, a plurality of microphones M1b, M2b, M3b, M4b, M5b, M6b, M7b, M8b are arranged in a sound space AR (for example, a concert hall in Tokyo), and a singer, piano player, violin Suppose an orchestra is being played by performers, clarinet players, and flute players. In this case, the server device CDS includes the sound of the singer's song, the piano sound by the piano player, the violin sound by the violin player, the clarinet sound by the clarinet player, and the flute player. The sound data in which at least the flute sound due to and the noise sound due to the noise source are collected are stored.

  In the second use case example shown in FIG. 4, the users Ps1b and Ps2b of the communication terminal T1 in which the application AP1 for using the sound source separation service is installed may be in Osaka, for example, far away from the concert hall. You may be near the concert hall. On the screen of the communication terminal T1 of the users Ps1b and Ps2b, a sound space AR (for example, a sketch of a concert hall) is displayed on the display unit 15 as an example of a screen of the application AP1 for using the sound source separation service. .

  The server device CDS, for example, receives the voice data collected by the microphones M1b to M8b in the sound space AR by a predetermined input operation on the communication terminal T1 of the user Ps1b in Osaka, for the same number as the number of microphones. Sound sources (for example, the sound of the singer's song, the piano sound by the piano player, the violin sound by the violin player, the clarinet sound by the clarinet player, and the flute sound by the flute player) Then, the sound source is separated into noise sound generated by the noise source and other sounds), and any separated sound data (for example, piano sound) is transmitted to the communication terminal T1. The communication terminal T1 sequentially switches and outputs the separated voice data transmitted from the server device CDS for every predetermined input operation of the user Ps1b. Thereby, the user Ps1b can listen only to high-quality sound after the sound source separation of the instrument (for example, piano) specified by the user in the orchestra performed in the concert hall in Tokyo, which is the sound space AR. it can.

  Similarly, the server device CDS receives, for example, audio data collected by the microphones M1b to M8b in the sound space AR by a predetermined input operation on the communication terminal T1 of the user Ps2b near the concert hall. The same number of types of sound sources (for example, the sound of a singer's song, the sound of a piano by a piano player, the sound of a violin by a violin player, the sound of a clarinet by a clarinet player, and the performance of a flute The sound source is separated into a flute sound by a person, a noise sound by a noise source, and other sounds), and any separated sound data (for example, a flute sound) is transmitted to the communication terminal T1. The communication terminal T1 sequentially switches and outputs the separated audio data transmitted from the server device CDS for every predetermined input operation of the user Ps2b. As a result, the user Ps2b can listen only to high-quality sound after the sound source separation of the instrument (for example, flute) specified by the user in the orchestra performed in the concert hall in Tokyo, which is the sound space AR. it can.

(Third use case example)
FIG. 5 is an explanatory diagram schematically showing a third use case example of the audio transmission system according to the present embodiment. FIG. 6 is a flowchart for explaining an example of the operation outline along the time series of the audio transmission system in the third use case example shown in FIG. 5. In the third use case example, a plurality of microphones M1c, M2c, M3c, M4c, M5c, M6c, M7c, M8c, M9c are arranged in a sound space AR (for example, a shopping mall SML in which a plurality of stores are arranged side by side) A situation is assumed in which the user Ps1c comes to the shopping mall. In this case, the server device CDS collects at least the PR speeches of the products or services of each store (specifically, fish shop, general store, book store, shoe store, sushi store, ramen store, information, clothing store). The recorded audio data is stored.

  In the third use case example shown in FIG. 5, the screen of the communication terminal T1 of the user Ps1c has a sound space AR (for example, a shopping mall SML) as an example of a screen of the application AP1 for using the sound source separation service. Map information of each store is displayed on the display unit 15.

  The server device CDS has the same number of voice data as the number of microphones collected by the microphones M1c to M9c in the sound space AR (for example, the shopping mall SML) by a predetermined input operation of the user Ps1c to the communication terminal T1. Type of sound source (for example, audio from a fish shop, audio from a general store, audio from a bookstore, audio from a shoe store, audio from a sushi store, audio from a ramen store, audio from an information store, audio from a clothing store The sound source is separated into voice and other voices, and one of the separated voice data (for example, voice from a bookstore) is transmitted to the communication terminal T1. The communication terminal T1 sequentially switches and outputs the separated voice data transmitted from the server device CDS for every predetermined input operation of the user Ps1c. As a result, the user Ps1c listens to the high-quality sound desired by the user Ps1c himself (for example, the sound from the bookstore) from the sound heard from the plurality of stores in the shopping mall SML as the sound space AR. Can do.

  In FIG. 6, when the user Ps1c who is a shopper enters the shopping mall SML (ST1), the communication terminal T1 possessed by the user Ps1c is in the target area of the sound source separation service that can be realized by the voice transmission system 100 of the present embodiment. A position report (notification signal) is received by, for example, Bluetooth (registered trademark) (ST2). The notification signal is transmitted, for example, from a predetermined sound source separation service providing device (not shown) installed by a shopping mall operator who provides the sound source separation service.

  Upon receiving the notification signal in step ST2, the communication terminal T1 activates the application AP1 for using the sound source separation service (ST3), and the sound space AR (for example, the user Ps1c is currently located on the screen of the display unit 15) A sketch (see FIG. 5) of the shopping mall SML) is displayed (ST4). Note that the process of step ST4 may be omitted in the communication terminal T1.

  As described above, the server device CDS has at least PR speeches of goods or services of each store (specifically, fish shop, general store, book store, shoe store, sushi store, ramen store, information, clothing store). The collected voice data is stored. The server device CDS has the same number of voice data as the number of microphones collected by the microphones M1c to M9c in the sound space AR (for example, the shopping mall SML) by a predetermined input operation of the user Ps1c to the communication terminal T1. The sound source is separated into two types of sound sources, and one of the separated sound data (for example, sound from a bookstore) is transmitted to the communication terminal T1.

  The communication terminal T1 sequentially switches and outputs the separated voice data transmitted from the server device CDS for every predetermined input operation of the user Ps1c (ST5). After the separated voice data desired by the user Ps1c is output from the communication terminal T1, the application AP1 for using the sound source separation service is terminated by the input operation of the user Ps1c (ST6).

(Fourth use case example)
FIG. 7 is an explanatory diagram schematically illustrating a fourth use case example of the audio transmission system 100 according to the present embodiment. In the fourth use case example, a plurality of microphones M1d, M2d, M3d, M4d, and M5d are arranged in the sound space AR (for example, an office room), and the negotiation space Dk arranged around the speaker Ps1 who is speaking loudly. Assume that the speakers Ps2 and Ps3 having a business talk are talking. In this case, the server device CDS stores voice data in which at least the voice of the speaker Ps1, the voice of the speaker Ps2, and the voice of one of the speaker Ps3 are collected.

  In the fourth use case example shown in FIG. 7, the speaker Ps2 who is the user of the communication terminal T1 in which the application AP1 for using the sound source separation service is installed is the communication terminal T2 that is not in the sound space AR. The application AP2 for using the sound source separation service is installed in the communication terminal T2 while talking with the call partner Ps4. In addition, when the telephones Ps2 and the other party Ps4 use the sound source separation service to make a call in a state where the applications AP1 and AP2 are not activated in the communication terminals T1 and T2, respectively, the communication terminals T1 and T2 Voices of the speaker Ps2 and the call partner Ps4 (that is, live call voices collected by the communication terminals T1 and T2) are transmitted / received via, for example, the LTE network NW2.

  The server device CDS obtains the sound data of the sounds collected by the microphones M1d to M5d in the sound space AR by a predetermined input operation on the communication terminal T1 of the speaker Ps2 as many kinds of sound sources as the number of microphones ( For example, the sound source is separated into the voice of the speaker Ps1, the voice of the speaker Ps2, the voice of the speaker Ps3, and other voices, and one of the separated voice data (for example, the voice of the speaker Ps3) is transmitted to the communication terminal T1. The communication terminal T1 sequentially switches and outputs the separated voice data transmitted from the server device CDS for every predetermined input operation of the speaker Ps2. When the speaker Ps2 designates any separated voice data (for example, separated voice data of the speaker Ps3) for the communication terminal T1, the communication terminal T2 performs the designated separated voice data via the server device CDS. The voice of the speaker Ps2 (received voice) is switched to the separated voice data of the speaker Ps3 and output. As a result, the speaker Ps2 transmits the high-quality separated voice data of the conversation contents of the speaker Ps3 who is talking to the speaker Ps2 in the office room which is the sound space AR to the far-end communication partner Ps4 which is not in the office room. Can be heard.

  Next, the operation procedure of the first operation example of the sound transmission system according to the present embodiment will be described with reference to FIGS. 1 and 8. FIG. 8 is a flowchart for explaining the operation procedure of the first operation example of the sound transmission system 100 according to the present embodiment. In the following description, for the sake of simplicity, the operations of the communication terminals T1 and T2 and the server device CDS are mainly described, and the operations of the first selection unit SL1 and the second selection unit SL2 are performed as necessary. The description of the operation of the microphone signal multiplex transmission apparatus MX will be omitted.

  In FIG. 8, the user A performs an activation operation of the application AP1 for using the sound source separation service with respect to the communication terminal T1. Thereby, communication terminal T1 starts application AP1 for using a sound source separation service (ST11). The communication terminal T1 receives a start signal for requesting start of sound source separation in the sound space AR, which is a target area of the sound source separation service, and listening to the separated sound data separated by the sound source, by a predetermined input operation of the user A. Transmit to the device CDS (ST12).

  The server device CDS reads sound data in which a plurality of types of sounds collected in the sound space AR are mixed from a memory (not shown) in the server device CDS, and performs sound source separation using the sound data after demultiplexing. Thus, the same number of types of separated voice data as that of one or more microphones (for example, microphones M1 to M4) (that is, output from each sound source when the same number of sound sources as the number of microphones is present in the sound space AR) Audio data) (ST21).

  The server device CDS initializes the parameter k indicating the ordinal number of the one or more separated speech data generated in step ST21 in the control unit PR1 (ST22), and the separated sound data corresponding to k = 1 is the first selection unit. It transmits to communication terminal T1 via SL1 (ST23).

  The communication terminal T1 receives the separated voice data transmitted in step ST23 and outputs it in the voice output unit 16 (ST13, provisional listening by the user A). At the time of step ST13, any one of the separated audio data c1 generated by the server device CDS is output (reproduced) at the communication terminal.

  Here, when the user A does not like the separated voice data output from the voice output unit 16 (ST14, NO), an input operation to that effect (for example, other separated voice data other than the currently outputted separated voice data) The communication terminal T1 transmits to the server device CDS a next selection request signal for switching to other separated voice data other than the currently outputted separated voice data.

  In response to the next selection request signal transmitted from communication terminal T1, server device CDS increments parameter k indicating the ordinal number of one or more separated audio data generated in step ST21 in control unit PR1 (ST24). The separated voice data corresponding to the parameter k after the increment (for example, k = 2) is transmitted to the communication terminal T1 via the first selection unit SL1 (ST23). As described above, in the first operation example of the voice transmission system 100 according to the present embodiment, transmission / reception of separated voice data is repeated between the communication terminal T1 and the server device CDS until the separated voice data that the user A wants to hear is found. Thus, the communication terminal T1 can continue to listen to the separated voice data after the separated voice data to be heard is found.

  On the other hand, if the user A likes the separated voice data output from the voice output unit 16 (ST14, YES), the user A performs an input operation (for example, output (reproduction) of the currently outputted separated voice data). The communication terminal T1 continues to output (reproduce) the separated audio data that is currently output (ST15, main listening by the user A). When the user A performs an input operation for instructing to end the application AP1 (ST16), the communication terminal T1 ends the application AP1.

  Next, the operation procedure of the second operation example of the sound transmission system according to the present embodiment will be described with reference to FIG. 1, FIG. 9, and FIG. FIG. 9 is a flowchart for explaining an operation procedure of the second operation example of the sound transmission system 100 according to the present embodiment. FIG. 10 is a flowchart for explaining the operation procedure subsequent to FIG. 9 and 10, the communication terminal T1 used by the user A is the transmitting side, and the communication terminal T2 used by the user B is the receiving side. In the description of FIG. 9, the same operations as those in FIG. 8 are given the same step numbers and the description thereof is omitted, and different contents will be described.

  In FIG. 9, the communication terminal T1 starts transmission / reception of call voice with the communication terminal T2 by a predetermined input operation of the user A for starting a call between the user A and the user B, and similarly. The communication terminal T2 starts transmission / reception of call voice with the communication terminal T1 by a predetermined input operation of the user B for starting a call between the user A and the user B (ST51). Therefore, the voice data of the voice transmitted from the communication terminal T1 to the communication terminal T2 is the voice data a of the call voice of the user A, and the voice data of the voice sent from the communication terminal T2 to the communication terminal T1. Is voice data b of user B's call voice.

  The user A performs an activation operation of the application AP1 for using the sound source separation service as the sender mode with respect to the communication terminal T1. Thereby, communication terminal T1 starts application AP1 for using a sound source separation service as a sender mode (ST32). Similarly, the user B performs an activation operation of the application AP2 for using the sound source separation service as the receiver mode with respect to the communication terminal T2. Thereby, communication terminal T2 starts application AP2 for using a sound source separation service as a receiver mode (ST52).

  The communication terminal T1 receives the separated voice data corresponding to the parameter k = 1 transmitted from the server device CDS in step ST23, and outputs it in the voice output unit 16 (ST13, provisional listening by the user A). At the time of step ST13, any one of the separated audio data c1 generated by the server device CDS is output (reproduced) at the communication terminal.

  When the user A likes the separated voice data output from the voice output unit 16 (ST14, YES), an operation to the effect that the voice data (separated voice data) to be transmitted to the user B who is the other party is determined. I do. Thereby, according to the determination operation of the user A, the communication terminal T1 transmits a determination signal indicating that the separated voice data to be transmitted to the communication terminal T2 is determined to the server device CDS and the communication terminal T2 (ST33). Note that the determination signal includes information indicating the type of any one of the separated audio data determined by the user A.

  In response to the determination signal transmitted from communication terminal T1 in step ST33, server apparatus CDS transmits any separated voice data corresponding to the determination signal to communication terminal T2 (ST41).

  On the other hand, when the communication terminal T2 receives the determination signal transmitted from the communication terminal T1 in step ST33 (ST53, YES), the user who has transmitted the received voice output from the voice output unit 26 is transmitted from the communication terminal T1. The voice data a of the call voice A is switched to one of the separated voice data c2 transmitted by the server device CDS via the second selection unit SL2 in step ST41 (ST54).

  In addition, when the user B performs the switching operation of the listening destination (ST55, YES), the communication terminal T2 uses the received voice output from the voice output unit 26 as the call voice of the user A transmitted from the communication terminal T1. And the separated audio data c2 transmitted from the server device CDS via the second selection unit SL2 in step ST41 (ST56).

  After step ST54, when the user B does not perform the switching operation of the listening destination (ST55, NO), the communication terminal T2 continues to transmit and receive call voice between the users A and B with the communication terminal T1. (ST57). Similarly, communication terminal T1 continues to transmit and receive call voice between users A and B with communication terminal T2 (ST34). Note that the voice data of the voice transmitted from the communication terminal T2 to the communication terminal T1 is the voice data b of the call voice of the user B, but the voice voice transmitted from the communication terminal T1 to the communication terminal T2. The data is the voice data a of the call voice of the user A or any one of the separated voice data c2 transmitted from the server device CDS via the second selection unit SL2.

  After step ST34, when the user A performs an input operation for instructing the end of the application AP1 (ST35), the communication terminal T1 ends the application AP1 and notifies the end of the application AP1. An end notification signal is transmitted to each of the server device CDS and the communication terminal T2. When the server device CDS receives the application end notification signal from the communication terminal T1 (ST42, YES), the server device CDS ends the operation in the sound source separation service.

  In addition, when the communication terminal T2 receives the application end notification signal from the communication terminal T1, the communication terminal T2 displays a message to that effect on the screen of the display unit 25, and performs an input operation for instructing the user B to end the application AP2. If it is performed (ST58), the received voice output from the voice output unit 26 is the voice data a of the call voice of the user A transmitted from the communication terminal T1, and the server device CDS sets the second selection unit SL2 in step ST41. Of the separated voice data c2 transmitted via the former (ST59).

  After step ST35, the communication terminal T1 continues to transmit and receive call voice between the users A and B with the communication terminal T2 (ST36). Similarly, communication terminal T2 continues to transmit and receive call voice between users A and B with communication terminal T1 (ST60). Since the sound source separation service applications AP1 and AP2 are terminated at the communication terminals T1 and T2, the voice data of the voice transmitted from the communication terminal T1 to the communication terminal T2 is the voice data of the call voice of the user A. The voice data of voice transmitted from the communication terminal T2 to the communication terminal T1 is the voice data b of the call voice of the user B.

  When communication between the users A and B ends with the communication terminals T2 and T1, the communication terminals T1 and T2 end transmission / reception of the call voice (ST37, ST61).

  As described above, the sound transmission system 100 according to the present embodiment picks up sound in the plurality of microphones M1 to M4 arranged in the sound space AR including a plurality of sounds, and the sound is picked up by these microphones M1 to M4. The voice data of voice is stored in the server device CDS on the cloud network NW1. The server device CDS separates the sound data stored in the sound source separation unit PR2 into the same number of types of sound data (separated sound data) as the microphones M1 to M4, and performs any separation by a predetermined input operation to the communication terminal T1. The voice data is transmitted to the first selection unit SL1 on the LTE network NW2. The first selection unit SL1 transmits any one of the individual separated audio data generated by the sound source separation unit PR2 to the communication terminal T1.

  As a result, the voice transmission system 100 can extract separated voice data serving as a specific sound source from the collected voice in an actual sound space AR in which environmental sounds including many speakers and noise are mixed. Therefore, high-quality separated voice data desired by the user A of the communication terminal T1 can be transmitted to the communication terminal T1 that is the terminal by a predetermined input operation on the communication terminal T1.

  In addition, for each predetermined input operation on the communication terminal T1, the audio transmission system 100 causes the control unit PR1 of the server device CDS to sequentially switch individual separated audio data so as to be transmitted from the sound source separation unit PR2 to the first selection unit SL1. . The communication terminal T1 receives and outputs one of the separated voice data transmitted from the first selection unit SL1 according to a predetermined input operation of the user A.

  As a result, the voice transmission system 100 can perform high-quality separation desired by the user A from the separated voice data separated as one or more sound sources in the server device CDS by a simple input operation of the user A with respect to the communication terminal T1. The voice data can be selectively switched and transmitted to the communication terminal T1.

  In addition, in the server device CDS, the audio transmission system 100 separates the audio data stored in the sound source separation unit PR2 into the same number of types of audio data (separated audio data) as the microphones M1 to M4, and performs predetermined processing for the communication terminal T2. By the input operation, one of the separated voice data is transmitted to the second selection unit SL2 on the LTE network NW2. The second selection unit SL2 includes the voice data transmitted from the communication terminal T1 (for example, the raw call voice of the user A picked up by the communication terminal T1) or individual separated voice data generated by the sound source separation unit PR2. Either is transmitted to the communication terminal T2.

  As a result, the voice transmission system 100 can extract separated voice data serving as a specific sound source from the collected voice in an actual sound space AR in which environmental sounds including many speakers and noise are mixed. Therefore, the voice data of the voice of the user A who is the call partner or the high-quality separated voice data generated by the server device CDS by the predetermined input operation on the communication terminal T2 which is the call partner of the communication terminal T1 is transmitted to the communication terminal T2. Can be selectively transmitted to the communication terminal T2 which is the communication partner of the communication terminal T1 in accordance with the user B's preference.

  Further, the voice transmission system 100 transmits any one of the individual separated voice data generated by the sound source separation unit PR2 to the communication terminal T1, and each separation is performed for each predetermined input operation of the user A with respect to the communication terminal T1. The sound data is sequentially switched and transmitted from the sound source separation unit PR2 to the first selection unit SL1. The communication terminal T1 receives and outputs one of the separated voice data transmitted from the first selection unit SL1 according to a predetermined input operation of the user A.

  As a result, the voice transmission system 100 is requested by the user B who is a call partner from the separated voice data separated as one or more sound sources in the server device CDS by a simple input operation of the user A to the communication terminal T1. Since the high quality separated voice data can be transmitted to the communication terminal T1 in accordance with the desire of the user A, the high quality separated voice data designated by the communication terminal T1 is obtained after listening in advance at the communication terminal T1. It can be transmitted to the communication terminal T2.

  In addition, the sound transmission system 100 causes the control unit PR1 to output any separated sound data from the sound source separation unit PR2 to the second selection unit SL2 in accordance with a predetermined input operation on the communication terminal T1. The second selection unit SL2 transmits any separated audio data output from the sound source separation unit PR2 to the second communication terminal.

  Thereby, the voice transmission system 100 can transmit any one of the separated voice data selected after the user A listens to the separated voice data transmitted to the communication terminal T1 to the communication terminal T2.

  Also, in the audio transmission system 100, when the communication terminal T1 receives a notification signal of a voice separation service that separates voice data of voice collected in the sound space AR into the same number of types of voice data as the microphones M1 to M4. In response to the service start operation for enjoying the voice separation service, one of the separated voice data transmitted from the first selection unit SL1 on the LTE network NW2 is received.

  As a result, even if the user A of the communication terminal T1 does not know the presence of the voice separation service, the voice transmission system 100 can receive a simple service of the user A when receiving the voice separation service notification signal at the communication terminal T1. By the start operation, the separated voice data generated by the sound source separation can be received by the communication terminal T1 and can be heard by the user A, or the separated voice data can be transmitted to the communication terminal T2.

  Further, in the audio transmission system 100, the communication terminal T1 notifies the information regarding the target area of the audio separation service that separates the audio data of the audio collected in the sound space AR into the same number of types of audio data as the microphones M1 to M4. Receive a signal.

  As a result, the voice transmission system 100, when the user A of the communication terminal T1 does not know the information related to the target area of the voice separation service, when the information about the target area of the voice separation service is received, By the input operation, the separated voice data generated by the sound source separation corresponding to any target area designated by the user A can be received at the communication terminal T1 and can be heard by the user A, or the separated voice data Can be transmitted to the communication terminal T2.

  In the sound transmission system 100, the sound source separation unit PR2 is selected according to a selection signal from the communication terminal T1 for selecting one or more microphones from the one or more microphones M1 to M4. Using the sound data of the sound collected by the above microphones (for example, the microphones M1 and M2), the data is separated into the same number of types of sound data as the one or more selected microphones (for example, the microphones M1 and M2).

  Thereby, the audio transmission system 100 wants to listen to the user A by selecting and operating on the communication terminal T1 the microphone around the partner (for example, a speaker) that the user A wants to listen to in the sound space AR. It is possible to efficiently collect the sound of the other party by eliminating the collection of a noise source (for example, a motor sound) at a position far away from the other party (for example, a speaker) to be considered.

  While various embodiments have been described above with reference to the drawings, it goes without saying that the present invention is not limited to such examples. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Understood.

  The present invention extracts a specific sound source from collected sound in an actual sound space in which environmental sounds including many speakers and noise are mixed, and the extracted high-quality sound is transmitted to the own terminal or the own terminal. It is useful as a voice transmission system and a voice transmission method for transmitting to the other party.

11, 21 Sound collection unit 12, 22 CPU
13, 23 Memory 14, 24 Input unit 15, 25 Display unit 16, 26 Audio output unit 17, 27 Wireless communication unit 18, 28 Antenna a, b Voice data c1, c2 of voice data Separated voice data AP1, AP2 Application CDS server device M1, M2, M3, M4 Microphone MX Microphone signal multiplexing transmission device NW1 Cloud network network NW2 LTE network PR Sound signal processing transmission unit PR1 Control unit PR2 Sound source separation unit SL1 First selection unit SL2 Second selection unit T1 , T2 communication terminal

Claims (6)

An audio transmission system including a first communication terminal that accepts a predetermined input operation,
One or more sound collection units for collecting sound in a sound space including a plurality of sounds;
A storage unit for storing audio data of the audio collected by the one or more sound collection units;
A separation unit that separates the sound data stored in the storage unit into the same number of types of sound data as the one or more sound collection units, and generates individual separated sound data;
Among the individual separated voice data generated by the separation unit, any separated voice data corresponding to the predetermined input operation to the first communication terminal is transmitted to the first communication terminal as switching voice data. A first transmitter that
The first communication terminal receives the switching voice data and outputs the voice;
Audio transmission system. The audio transmission system according to claim 1,
A control unit that sequentially switches the individual separated audio data for each predetermined input operation to the first communication terminal and outputs the separated separated data to the first transmission unit;
Audio transmission system. The voice transmission system according to claim 1 or 2,
When the first communication terminal receives a notification signal of a voice separation service that separates voice data of voice collected in the sound space into the same number of types of voice data as the sound collection unit, the voice separation In response to a service start operation, the switching voice data transmitted from the first transmission unit is received.
Audio transmission system. The voice transmission system according to claim 1 or 2 ,
The first communication terminal receives a notification signal of information related to a target area of a voice separation service that separates voice data of voice collected in the sound space into the same number of types of voice data as the sound collection unit,
Audio transmission system. The voice transmission system according to claim 1 or 2 ,
The separation unit is configured to select one or more of the selected one or more of the one or more sound collection units according to a selection signal from the first communication terminal for selecting the one or more sound collection units. Using the voice data of the voice collected by the sound collection unit, the voice data is separated into the same number of types of voice data as the one or more selected sound collection units.
Audio transmission system. An audio transmission method in an audio transmission system including a first communication terminal that accepts a predetermined input operation,
Collecting sound in one or more sound collection devices arranged in a sound space including a plurality of sounds;
Storing audio data of the audio collected by the one or more sound collection devices;
Separating the audio data into the same number of types of audio data as the one or more sound collection devices, and generating individual separated audio data;
A step of transmitting any separated voice data corresponding to the predetermined input operation to the first communication terminal among the separated separated voice data after separation to the first communication terminal as switching voice data;
Receiving the switching voice data and outputting the voice,
Audio transmission method.

Images