JP2022108957A - Data processing device, data processing system, and voice processing method - Google Patents

Abstract

To allow data arrival status to be confirmed without confirmation by conversation between users.SOLUTION: A data processing device includes: a receiving unit that receives first coded data from another data processing device; a decoding unit that decodes the first coded data received by the receiving unit; encoding unit that encodes data obtained by the decoding unit to generate second coded data; and a control unit that controls transmission of the second coded data to the other communication device when it is determined that the first coded data is data transmitted in a specific operation mode from the other communication device.SELECTED DRAWING: Figure 2

Description

The present invention relates to a data processing device, a data processing system, and an audio processing method.

In recent years, companies are accelerating the introduction of telework as a countermeasure against pandemics. In telework, the office and teleworkers at home, or teleworkers at home are connected via the Internet, and remote meetings, training, remote communication, etc. done.

In the above TV conference system and video call software, depending on the type and configuration of the device used, problems may occur in video and audio when participating in a conference. For this reason, a mechanism is known in which a user can check the operation of the TV conference system before participating in the conference. For example, Patent Literature 1 discloses a technology that has a normal mode used in a conference and a self-check mode for preliminary confirmation, and allows confirmation of the voice quality transmitted to the other party during a teleconference. In the technology, in the self-check mode, after the user's uttered voice is compressed, the compressed uttered voice is looped back in the user's terminal, and the uttered voice is reproduced after being decompressed. This allows the user to check the degree of deterioration due to compression and decompression of the voice.

JP 2011-193374 A

However, even if the user's voice data is sent to the terminals of other users participating in the conference, the user's voice may be may not be output from For this reason, each time a conference is held, it is often the case that confirmation is made by conversation between participating users, such as "Are you receiving audio?" This is repeated every time the number of users increases, and in the case of midway participation in the conference, it is not possible to interrupt the conference, and confirmation may not be possible. In addition, since the technology described in Patent Document 1 is a technology that loops back the user's uttered voice in the user's terminal, the technology described in Patent Document 1 allows the user's voice data to participate in the conference. It is difficult to confirm whether or not it reaches the user's terminal.

Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to provide a new and novel method that enables confirmation of data arrival status without confirmation by conversation between users. An object of the present invention is to provide an improved data processing device, data processing system, and voice processing method.

In order to solve the above problems, according to one aspect of the present invention, a receiving unit for receiving first encoded data from another data processing device, and receiving the first encoded data received by the receiving unit, a decoding unit that decodes; an encoding unit that encodes data obtained by the decoding unit to generate second encoded data; and a control unit that controls transmission of the second encoded data to the other communication device if the data is determined to be data transmitted in the above.

The data processing device further includes an output unit that outputs audio or video based on the data obtained by the decoding unit, and the output unit causes the first encoded data to be converted to the other data by the control unit. When it is determined from the processing device that the data is transmitted in the specific operation mode, the audio and video may not be output based on the data obtained by the decoding section.

The encoding unit generates the second encoded data using a second processing method that causes less quality deterioration than the first processing method used to generate the first encoded data. good too.

The control unit may control transmission of the second encoded data using a second communication scheme having higher reliability than the first communication scheme used for communication of the first encoded data. .

Based on the fact that a predetermined flag is added to the first encoded data, the control section controls that the first encoded data is data transmitted in a specific operation mode from the other data processing device. You can judge that there is.

In order to solve the above problems, according to another aspect of the present invention, there is provided a data processing system having a first data processing device and a second data processing device, wherein the first data processing device comprises transmitting first encoded data obtained by encoding data to the second data processing device, the second data processing device receiving the first encoded data from the first data processing device; a decoding unit that decodes the first encoded data received by the receiving unit; and an encoding that encodes the data obtained by the decoding unit to generate second encoded data and the first data of the second encoded data when it is determined that the first encoded data is data transmitted from the first data processing device in a specific operation mode. and a controller for controlling transmission to the processing device.

Further, in order to solve the above problems, according to another aspect of the present invention, receiving first encoded data from another data processing device, and decoding the first encoded data. encoding data obtained by decoding the first encoded data to generate second encoded data; and receiving the first encoded data from the other data processing device in a specific operation mode and controlling the transmission of said second encoded data to said other data processing device if it is determined that said data was transmitted by said method.

In order to solve the above problems, according to another aspect of the present invention, there is provided an encoding unit that encodes input data to generate encoded data; a transmitting unit for transmitting; a receiving unit for receiving first encoded data or second encoded data from said other data processing device; and in a first operation mode, voice based on said first encoded data Alternatively, there is provided a data processing device comprising: a control unit that controls video output, and controls audio or video output based on the second encoded data in a second operation mode.

The first encoded data is data transmitted using a first communication scheme, and the second encoded data is transmitted using a second communication scheme having higher reliability than the first communication scheme. the transmitted data, wherein the receiving unit corresponds to the first communication system and receives the first encoded data; and a second receiver for receiving two encoded data.

The first encoded data is data generated using a first processing method, and the second encoded data is generated using a second processing method with less quality deterioration than the first processing method. and the data processing device corresponds to the first processing method, the first decoding unit for decoding the first encoded data, and the second processing method. and a second decoding unit that decodes the second encoded data.

The data processing device may further include a first buffer that holds input data, and the control section may supply the data held in the first buffer to the encoding section.

The data processing device further comprises a second buffer holding a plurality of data obtained by decoding the plurality of second encoded data received from the plurality of other data processing devices, wherein the control unit , the output of the plurality of data held in the second buffer may be sequentially controlled.

The transmitting section may transmit the encoded data together with a predetermined flag in the second operation mode.

In order to solve the above problems, according to another aspect of the present invention, input data is encoded to generate encoded data, and the encoded data is transmitted to another data processing device. receiving the first encoded data or the second encoded data from the other data processing device; and outputting audio or video based on the first encoded data in a first operation mode. and, in a second mode of operation, controlling output of audio or video based on said second encoded data.

According to the present invention described above, it is possible to confirm the arrival status of data without confirmation by conversation between users.

1 is an explanatory diagram showing the configuration of a data processing system according to one embodiment of the present invention; FIG. 1 is an explanatory diagram showing the configuration of a speech processing device 20 according to one embodiment of the present invention; FIG. FIG. 4 is an explanatory diagram showing a connection sequence in the data processing system according to one embodiment of the present invention; FIG. 10 is an explanatory diagram showing a processing sequence when the speech processing device 20A shifts to test mode; FIG. 4 is an explanatory diagram showing the flow of audio data in the normal mode of the audio processing device 20A and the audio processing device 20B; 3 is an explanatory diagram showing the flow of audio data when the audio processing device 20 operates in the test mode and the audio processing device 20B operates in the normal mode; FIG. 2 is a block diagram showing the hardware configuration of the audio processing device 20; FIG.

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

In addition, in this specification and drawings, a plurality of components having substantially the same functional configuration may be distinguished by attaching different alphabets after the same reference numerals. For example, a plurality of configurations having substantially the same functional configuration or logical significance are differentiated like audio processing devices 20A, 20B and 20C as necessary. However, when there is no particular need to distinguish between a plurality of constituent elements having substantially the same functional configuration, only the same reference numerals are given to each of the plurality of constituent elements. For example, when there is no particular need to distinguish between the audio processing devices 20A, 20B and 20C, each audio processing device is simply referred to as the audio processing device 20. FIG.

<1. Outline of Data Processing System>
One embodiment of the present invention relates to a data processing system that realizes a voice conference in which a plurality of users participate from remote sites. First, with reference to FIG. 1, an outline of a data processing system according to one embodiment of the present invention will be described.

FIG. 1 is an explanatory diagram showing the configuration of a data processing system according to one embodiment of the present invention. As shown in FIG. 1, a data processing system according to one embodiment of the present invention includes audio processors 20A-20F and a conference server 30. As shown in FIG.

These audio processing devices 20A to 20F and conference server 30 are connected via network 12. FIG. Network 12 is a wired or wireless transmission path for information transmitted from devices connected to network 12 . For example, the network 12 may include a public line network such as the Internet, a telephone line network, a satellite communication network, various LANs (Local Area Networks) including Ethernet (registered trademark), WANs (Wide Area Networks), and the like. The network 12 may also include a dedicated line network such as IP-VPN (Internet Protocol-Virtual Private Network).

In the example shown in FIG. 1, the audio processing device 20A, the audio processing device 20B, and the audio processing device 20C belong to the group G1 participating in the same audio conference, and the audio processing device 20D and the audio processing device 20E participate in another audio conference. The voice processing device 20F does not participate in any voice conference.

In the example shown in FIG. 1, the user UA uses the voice processing device 20A, the user UB uses the voice processing device 20B, the user UC uses the voice processing device 20C, and the user UD uses the voice processing device 20C. The voice processing device 20D is used, the user UE uses the voice processing device 20E, and the user UF uses the voice processing device 20F. However, the number of voice processing devices 20 constituting the data processing system and the number of users U using the data processing system may be smaller or larger.

(conference server)
The conference server 30 manages participation and withdrawal of each audio processing device 20 from the conference. For example, the conference server 30 uses a conference protocol such as WebRTC (Web Real-Time Communication) to manage participation and withdrawal of each audio processing device 20 from the conference. Communication of video data and audio data is performed within groups such as the above group G1 and group G2 without going through the conference server 30 . Note that the conference server 30 may not be provided when the audio processing devices 20 are connected to each other by Peer2Peer.

(sound processing device)
The voice processing device 20 is an example of a data processing device, and transmits voice data representing voice uttered by the user of the voice processing device 20 to other voice processing devices 20 . In addition, the voice processing device 20 receives voice data indicating the voice uttered by the user of the other voice processing device 20 from the other voice processing device 20, and uses the other voice processing device 20 based on the voice data. Outputs the voice uttered by the person.

For example, in the example shown in FIG. 1, the voice processing device 20A transmits voice data representing voice uttered by the user UA to the voice processing devices 20B and 20C, and the voice processing devices 20B and 20C A voice uttered by the user UA is output based on the voice data. Further, the voice processing device 20B transmits voice data indicating voice uttered by the user UB to the voice processing device 20A, and the voice processing device 20A outputs the voice uttered by the user UB based on the voice data. Also, the voice processing device 20C transmits voice data indicating voice uttered by the user UC to the voice processing device 20A, and the voice processing device 20A outputs the voice uttered by the user UC based on the voice data. With this configuration, user UA, user UB, and user UC can hold a voice conference.

Note that the audio processing device 20 may transmit and receive video data to and from other audio processing devices 20 in addition to audio data. 1 shows a notebook PC (Personal Computer) as an example of the audio processing device 20, the audio processing device 20 may be a desktop PC, a smart phone, a mobile phone, a PHS (Personal Handyphone System), or the like. It may be another information processing device.

(background)
In such a data processing system, even if a user's voice data is sent to the voice processing devices of other users participating in the conference, problems with the network or other users' voice processing devices may result in A user's voice may not be output from other users' voice processing devices. For this reason, each time a conference is held, it is often the case that confirmation is made by conversation between participating users, such as "Are you receiving audio?" This is repeated every time the number of users increases, and in the case of midway participation in the conference, it is not possible to interrupt the conference, and confirmation may not be possible.

In some cases, the output problem of the receiving-side audio processing device may be pointed out as if it were a problem of the transmitting-side audio processing device, and time may be spent on solving the problem. Conversely, the user may mistakenly think that the problem is the speech processing device on the receiving side even though the problem is the speech processing device on the transmitting side. Furthermore, teleconferencing is greatly affected by the network environment, and packet delays and losses can degrade the quality of voice data, making it difficult to check operations during a conference.

It is also conceivable to configure the conference server so that the video data and the audio data can be looped back on the conference server side to check the operation of the audio processing device. However, as shown in FIG. 1, in a system in which video data and audio data of each audio processing device are transmitted and received without going through the conference server during a conference, it is difficult to transfer audio data from one audio processing device to another audio processing device. It is difficult to check the arrival status during the meeting.

The inventor of the present invention has created an embodiment of the present invention by focusing on the above circumstances. According to one embodiment of the present invention, it is possible to check the delivery status of voice data without confirmation by conversation between users. As a result, it becomes possible to proceed with the audio conference smoothly. Hereinafter, the configuration and operation of such an embodiment of the present invention will be sequentially described in detail.

<2. Overview of audio processing device>
FIG. 2 is an explanatory diagram showing the configuration of the audio processing device 20 according to one embodiment of the present invention. As shown in FIG. 2, the speech processing device 20 according to an embodiment of the present invention includes a speech input section 220, a first encoding section 224, a first communication section 228, a first decoding section 232, a speech output section 236, An operation unit 238 , a control unit 240 , a first buffer 244 , a forwarding unit 248 , a second encoding unit 252 , a second communication unit 256 , a second decoding unit 260 and a second buffer 264 are provided.

(Voice input unit 220)
The speech input unit 220 is configured to receive speech uttered by the user of the speech processing device 20 . The voice input unit 220 converts the voice uttered by the user of the voice processing device 20 into electrical voice data, and supplies the voice data to the first encoding unit 224 and the first buffer 244 . The voice input unit 220 may be configured by a microphone device, or may be a microphone mounted on a camera device used for a conference.

(First encoding unit 224)
The first encoding unit 224 encodes the audio data supplied from the audio input unit 220 using the first processing method to generate encoded audio data. The first processing method may be a lossy compression method. In this specification, the encoded audio data generated by the first encoding unit 224 may also be referred to as first encoded audio data.

(First communication unit 228)
The first communication unit 228 includes a transmission unit (first transmission unit) that transmits the encoded audio data generated by the first encoding unit 224 to the other audio processing device 20 and function as a receiving unit (first receiving unit) that receives encoded audio data generated by the processing method of . The first communication unit 228 communicates using a conference protocol such as UDP as the first communication method. The first communication unit 228 may use TCP as another protocol, but if the packet delay becomes large, it is assumed that the packet is discarded in order to eliminate the delay.

(First decoding unit 232)
The first decoding unit 232 decodes the encoded audio data received from the other audio processing device 20 by the first communication unit 228 using the first processing method.

(Audio output unit 236)
The audio output unit 236 outputs the audio uttered by the user of the other audio processing device 20 based on the audio data obtained by the first decoding unit 232 . The audio output unit 236 may be a speaker device, a headset integrated with a microphone device, a microphone speaker, or the like. The audio output unit 236 also outputs audio supplied from a second buffer 264, which will be described later.

The operation modes of the audio processing device 20 include a normal mode, which is an example of a first operation mode, and a test mode, which is an example of a second operation mode (specific operation mode). In the normal mode, the audio input unit 220, the first encoding unit 224, the first communication unit 228, the first decoding unit 232, and the audio output unit 236 described above operate to enable voice communication between users of the audio processing device 20. It is possible to hold a meeting. In the test mode, whether or not the voice data of the user of the voice processing device 20 has reached another voice processing device 20 by the operations of the second communication unit 256, the second decoding unit 260, the second buffer 264, etc., which will be described later. , it is possible to check the quality of the arrival.

(Operation unit 238)
The operation unit 238 is configured to be operated by the user of the speech processing device 20 to input information or instructions to the speech processing device 20 . For example, the user of the speech processing device 20 inputs an instruction to the operation unit 238 to switch the operation mode of the speech processing device 20 between the normal mode and the test mode.

(control unit 240)
The control unit 240 controls overall operations of the audio processing device 20 . For example, the control unit 240 switches the operation mode of the speech processing device 20 between the normal mode and the test mode according to the user's instruction to the operation unit 238 . In the test mode, the control section 240 performs, for example, the following controls.

The control unit 240 stops the supply of audio data from the audio input unit 220 to the first encoding unit 224 . Instead, the control section 240 supplies the audio data held in the first buffer 244 to the first encoding section 224 . The first encoding unit 224 encodes the audio data supplied from the first buffer 244 to generate encoded audio data. The audio data supplied from the audio input unit 220 through the first encoding unit 224 may be encoded to generate encoded audio data, without using the first buffer 244, according to a user's instruction.

The control unit 240 causes the first communication unit 228 to transmit information about the test mode together with transmission of the encoded audio data. Information about the test mode includes a predetermined flag, and information indicating an address and port for waiting for data from other audio processors 20 . When the first communication unit 228 communicates using WebRTC, the first communication unit 228 may transmit information regarding the test mode using, for example, a data channel.

The control unit 240 receives the encoded audio data by the first communication unit 228, supplies the encoded audio data from the first communication unit 228 to the first decoding unit 232, and decodes the encoded audio data by the first decoding unit 232. , or the supply of audio data from the first decoding unit 232 to the audio output unit 236 is stopped.

With the second decoding unit 260 and the second buffer 264 in the operating state, the encoded audio data received from the other audio processing device 20 by the second communication unit 256 (the second encoding generated by the second processing method) audio data) is decoded by the second decoding unit 260, the audio data obtained by decoding is held in the second buffer 264, and one or more audio data held in the second buffer 264 is sent to the audio output unit 236. output sequentially.

Also, in the normal mode, the control unit 240 performs, for example, the following controls.

Control unit 240 puts second decoding unit 260 and second buffer 264 in a non-operating state.

When the information indicating the test mode is received together with the encoded audio data from the other audio processing device 20 by the first communication unit 228, the control unit 240 transmits the encoded audio data from the other audio processing device 20 in the test mode. It is determined that the data is

If the encoded audio data is data transmitted in the test mode from another audio processing device 20, the control unit 240 outputs the audio data obtained from the encoded audio data by the first decoding unit 232 to the audio output unit 236. do not supply to Instead, the control unit 240 causes the forwarding unit 248 to forward to the second encoding unit 252 . As a result, the second encoding unit 252 encodes the audio data using the second processing method to generate encoded audio data, and the second communication unit 256 transmits the encoded audio data to another audio processing device 20. 2 communication method.

(first buffer 244)
The first buffer 244 temporarily holds part of the audio data supplied from the audio input section 220 as test audio data. In the test mode, audio data is supplied from the first buffer 244 to the first encoding section 224 based on the user's operation on the operation section 238 .

(forwarding unit 248)
When the information indicating the test mode is received together with the encoded audio data from another audio processing device 20 by the first communication unit 228, the forwarding unit 248 transmits the audio data obtained by decoding the encoded audio data to the first communication unit 228. 1 decoding unit 232 to second encoding unit 252 .

(Second encoding unit 252)
The second encoding unit 252 encodes the audio data received from the forwarding unit 248 using the second processing method to generate encoded audio data. The second processing method is a processing method that causes less deterioration in voice quality than the first processing method. The second processing method may be a reversible encoding method that does not degrade audio data. In this specification, the encoded audio data generated by the second encoding unit 252 may also be referred to as second encoded audio data.

(Second communication unit 256)
The second communication unit 256 transmits the encoded audio data generated by the second encoding unit 252 together with information indicating the test mode to the other audio processing device 20 that transmitted the encoded audio data (second transmitter) and a receiver (second receiver) that receives encoded audio data generated by the second processing method from another audio processor 20 . The second communication unit 256 may perform communication using a second communication method that is more reliable than the first communication method. As such a second communication method, for example, there is TCP, which gives top priority to preservation of voice data even if packet delay occurs. Note that the second communication unit 256 and the first communication unit 228 may be functions implemented in the same network device.

(Second decoding unit 260)
The second decoding unit 260 decodes the encoded audio data received from the other audio processing device 20 by the second communication unit 256 using the second processing method.

(Second buffer 264)
A second buffer 264 temporarily holds the audio data obtained by the second decoding unit 260 . When there are a plurality of other audio processing devices 20 communicating with the audio processing device 20, and the second communication unit 256 receives encoded audio data from the plurality of audio processing devices 20, the second buffer 264 stores a plurality of Holds a plurality of audio data obtained by decoding encoded audio data. A plurality of pieces of audio data held by the second buffer 264 are sequentially output as audio from the audio output section 236 under the control of the control section 240 . Note that when there is only one other audio processing device 20 communicating with the audio processing device 20, the second buffer 264 does not hold audio data, and the audio output unit 236 outputs audio based on the audio data. may

<3. Operation>
The configuration of the speech processing device 20 according to one embodiment of the present invention has been described above. Next, the operation of the data processing system according to one embodiment of the present invention will be described.

(connection sequence)
First, referring to FIG. 3, the connection sequence in the data processing system according to one embodiment of the present invention will be described.

FIG. 3 is an explanatory diagram showing the connection sequence in the data processing system according to one embodiment of the present invention. As shown in FIG. 3, first, the user UA performs a conference connection operation on the operation unit 238 of the speech processing device 20A (S1). The connection to the conference may be made by designating a conference room prepared in advance by the conference server 30 or by creating a new conference room. The voice processing device 20A transmits a connection request to the conference server 30 according to the operation from the user UA (S2).

Similarly, first, when the user UB operates the operation unit 238 of the audio processing device 20B to connect to the conference (S3), the audio processing device 20B transmits a connection request to the conference server 30 (S4). When the voice processing device 20A and the voice processing device 20B specify the same conference room, the conference server 30 issues a connection instruction to the voice processing device 20A and the voice processing device 20B so that they communicate with each other by Peer2Peer (S5, S6). , the audio processing device 20A and the audio processing device 20B establish a connection (S7, S8).

After the connection is established, when the user UA speaks to the voice processing device 20A (S9), the first encoding unit 224 of the voice processing device 20A generates encoded voice data, and transmits the encoded voice data to the voice processing device. 20B (S10). Then, the first decoding unit 232 of the audio processing device 20B decodes the encoded audio data, and the audio output unit 236 of the audio processing device 20B outputs the audio of the user UA based on the audio data obtained by decoding ( S11).

Similarly, when user UB speaks toward speech processing device 20B (S12), first encoding unit 224 of speech processing device 20B generates encoded speech data, and sends the encoded speech data to speech processing device 20A. Send (S13). Then, the first decoding unit 232 of the audio processing device 20A decodes the encoded audio data, and the audio output unit 236 of the audio processing device 20A outputs the audio of the user UB based on the audio data obtained by decoding ( S14). During this time, encoded voice data does not pass through the conference server 30 . Even if the number of voice processing devices 20 participating in the conference room increases, each voice processing device 20 can be connected by the same sequence.

(Sequence in test mode)
Next, a processing sequence when the audio processing device 20A shifts to the test mode after the connection between the audio processing device 20A and the audio processing device 20B is established will be described.

FIG. 4 is an explanatory diagram showing a processing sequence when the speech processing device 20A shifts to the test mode. When the user UA performs a speech test start operation on the speech processing device 20A (S21), the speech processing device 20A shifts to the test mode (S22). In the test mode, a flag indicating that the encoded audio data is transmitted to the partner device is set to indicate that the encoded audio data is test audio data.

After that, the user UA issues an instruction to speak in the test mode or to use the voice data held in the first buffer 244 (S23). When the user UA makes a test utterance (S24), the first buffer 244 of the speech processing device 20A holds speech data for a certain period of time (S25: first buffering), and transfers the held speech data to the first buffer. 1 encoding unit 224 encodes using the first processing method to generate encoded audio data (S27: first encoding).

On the other hand, when using the audio data held in the first buffer 244, if there is audio data held in the first buffer 244, the control unit 240 supplies the audio data to the first encoding unit 224 ( S26), the first encoding unit 224 encodes the audio data using the first processing method to generate encoded audio data (S27: first encoding).

In any case, the encoded audio data is transmitted from the first communication unit 228 of the audio processing device 20A to the audio processing device 20B along with the test mode information including the predetermined flag (S28). The information about the test mode includes, in addition to a predetermined flag, information indicating an address and port for waiting for data from the audio processing device 20B.

When the first communication unit 228 of the audio processing device 20B receives the encoded audio data, the first decoding unit 232 of the audio processing device 20B decodes the encoded audio data using the first processing method (S29: first 1 decode). Here, when the control unit 240 of the audio processing device 20B determines that the encoded audio data is data transmitted in the test mode based on the setting of a predetermined flag, the forwarding unit 248 of the audio processing device 20B decodes The obtained audio data is forwarded to the second encoding unit 252, and the second encoding unit 252 encodes the audio data using the second processing method (S30: second encoding). Then, the second communication unit 256 of the audio processing device 20B transmits the encoded audio data generated by encoding to the audio processing device 20A (S31).

When the second communication unit 256 of the audio processing device 20A receives the encoded audio data, the second decoding unit 260 of the audio processing device 20A decodes the encoded audio data using the second processing method (S32: second decoding), and the second buffer 264 temporarily holds the audio data obtained by decoding (S33: second buffering).

After that, the audio output unit 236 outputs audio based on the audio data held in the second buffer 264 (S34). When there are a plurality of devices on the other end of the audio processing device 20A, the audio output unit 236 reproduces the plurality of audio data held in the second buffer 264 in arbitrary order. Then, the control unit 240 performs processing for terminating the test mode, and removes the setting of the predetermined flag (S35).

(Audio data flow)
Here, with reference to FIGS. 5 and 6, the flow of audio data in the normal mode and the flow of audio data in the test mode will be organized.

FIG. 5 is an explanatory diagram showing the flow of audio data in the normal mode of the audio processing devices 20A and 20B. In FIG. 5, a solid line indicates the flow of voice data of the user UA of the voice processing device 20A, and a two-dot chain line indicates the flow of voice data of the user UB of the voice processing device 20B.

As indicated by the solid line in FIG. 5, in the normal mode, the voice data of the user UA is input to the voice input section 220 and the first encoding section 224 of the voice processing device 20A (may pass through the first buffer 244). , the first communication unit 228, the first communication unit 228, the first decoding unit 232, and the audio output unit 236 of the audio processing device 20B. Similarly, as indicated by the two-dot chain line in FIG. 5, the voice data of user UB is input to voice input section 220 and first encoding section 224 (may be passed through first buffer 244) of voice processing device 20B. , the first communication unit 228, the first communication unit 228, the first decoding unit 232, and the audio output unit 236 of the audio processing device 20A.

FIG. 6 is an explanatory diagram showing the flow of audio data when the audio processing device 20 operates in the test mode and the audio processing device 20B operates in the normal mode. In FIG. 6, a solid line indicates the flow of voice data of the user UA of the voice processing device 20A, and a two-dot chain line indicates the flow of voice data of the user UB of the voice processing device 20B.

As indicated by the solid line in FIG. 6, in the test mode, the voice data of the user UA is input to the voice input unit 220 and the first encoding unit 224 (may be passed through the first buffer 244) of the voice processing device 20A. , the first communication unit 228, the first communication unit 228, the first decoding unit 232, the forwarding unit 248, the second encoding unit 252, and the second communication unit 256 of the audio processing device 20B. The encoded audio data generated by the processing method of (1) is transmitted to the audio processing device 20A. After that, the encoded audio data is processed in the flow of the second communication unit 256, the second decoding unit 260, the second buffer 264, and the audio output unit 236 of the audio processing device 20A, and is used by the audio output unit 236 of the audio processing device 20A. The voice of the person UA is output. In the audio processing device 20B, since audio data is not supplied from the first decoding unit 232 to the audio output unit 236, the audio data of the user UA is not output from the audio output unit 236 of the audio processing unit 20B.

On the other hand, as indicated by the two-dot chain line in FIG. 6, when the speech processing device 20A is operating in the test mode, the speech data of the user UB is input to the speech input unit 220 of the speech processing device 20B, the first Processing is performed in the flow of the encoding unit 224 (which may be via the first buffer 244), the first communication unit 228, the first communication unit 228, and the first decoding unit 232 of the audio processing device 20A. Since the audio data obtained by the first decoding section 232 is not supplied to the audio output section 236, the audio data of the user UB is not output from the audio output section 236 of the audio processing device 20A.

<4. Action effect>
According to one embodiment of the present invention described above, various functions and effects are exhibited. For example, in the test mode, the speech processing device 20 according to one embodiment of the present invention receives the user's speech data by returning it from another speech processing device 20, and outputs the user's speech based on the speech data. . Therefore, it is possible to confirm whether or not the user's voice data is in an environment where the voice data of the user reaches the other voice processing device 20 without confirming by conversation such as "Is the voice reaching you?" be. As a result, the progress of the conference is not hindered by such conversations, so that the conference can proceed smoothly.

In addition, the audio processing device 20 according to one embodiment of the present invention can be used in a network-compatible TV conference system or video call software on a PC, and the encoding method (first processing method) and speed during normal conferences and conversations. In addition to the transmission/reception protocol (first communication method) that emphasizes importance, it supports a reversible encoding method (second processing method) and a transmission/reception protocol that emphasizes certainty (second communication method). Then, the second processing method and the second communication method are used for looping back the audio data from the other audio processing device 20 . Therefore, the user of the speech processing device 20 can check the quality of the speech that will be output by other speech processing devices 20 based on his/her own speech output from the speech processing device 20 . Also, if there is no problem with the quality of the own voice output from the voice processing device 20, it can be seen that there is no problem with the internal processing of the voice processing device 20 and the other voice processing device 20, and with the networks of both. It is possible to reduce the time loss for investigating the cause.

Also, the audio processing device 20 according to the embodiment of the present invention does not output audio based on normal audio data received from other audio processing devices 20 in the test mode. Therefore, the user of the voice processing device 20 can clearly hear the voice that is returned and received from the other voice processing device 20, thereby more accurately grasping the quality of the voice. Further, since the other audio processing device 20 does not output audio based on the audio data transmitted from the audio processing device 20 operating in the test mode, it is possible to prevent the conference from being disturbed by the test audio.

In addition, since the speech processing device 20 according to one embodiment of the present invention includes the second buffer 264, even when the speech processing device 20 conducts a conference with a plurality of other speech processing devices 20, each of the other speech processing devices 20 It is possible to listen to their own voices in order, which are returned from and received.

<5. Variation>
An embodiment of the present invention has been described above. Several modifications of the above-described embodiment are described below. In addition, each modification described below may be applied to the above-described embodiment alone, or may be applied in combination to the above-described embodiment. Further, each modification may be applied instead of the configuration of the embodiment described above, or may be applied additionally to the configuration of the embodiment described above.

As the second processing method, a method different from the encoding method (first processing method) used in normal meetings and conversations has been described above. However, the type of the second processing scheme may be the same as the type of the first processing scheme, and the parameters applied to the second processing scheme may differ from the parameters applied to the first processing scheme. Also, the second processing method may be a near-lossless method instead of a completely reversible method.

In addition to the encoding method (first processing method) for normal meetings and conversations and the speed-oriented transmission/reception protocol (first communication method), the speech processing device 20 also uses a reversible encoding method ( 2nd processing method) and a transmission/reception protocol emphasizing certainty (second communication method) have been described. Or it does not have to correspond to both. That is, encoded audio data generated by the first processing method in another audio processing device 20 may be looped back by the first communication method. In this case, voice data whose quality has deteriorated from that of the voice data reaching the other voice processing device 20 is returned to the voice processing device 20. It is possible to confirm whether or not it has reached 20.

Moreover, although the example in which the voice processing device 20 has the function for looping back the voice data has been described above, the function may also be implemented in the conference server 30 . In this case, the voice processing device 20 receives the voice data returned from both the conference server 30 and the other voice processing device 20, and compares the voice data of both to investigate the cause of the trouble. For example, when normal audio data is returned from the conference server 30, but audio data is not returned from the other audio processing devices 20 (or if there is a quality problem even if the audio data is returned), the audio processing It can be seen that there is no problem with the network lines on the device 20 side and the audio processing device 20 side, and there is some problem on the other audio processing device 20 side.

In the above description, audio data was described as an example of data, and encoded audio data was described as an example of encoded data, but an embodiment of the present invention can also be applied to video data and encoded video data. . That is, the audio processing device 20 may have a function for folding back video data. In this case, a video input unit such as a camera is used in addition to or instead of the audio input unit 220 described with reference to FIG. 2, and a video display unit such as a display is used in addition to or instead of the audio output unit 236. Used. With this configuration as well, it is possible to confirm whether or not the video of the user of the audio processing device 20 will reach another audio processing device 20, and if so, what quality it will have.

<6. Hardware Configuration>
An embodiment of the present invention has been described above. Information processing such as encoding and decoding of the audio data described above is realized by cooperation between software and hardware of the audio processing device 20 described below.

FIG. 7 is a block diagram showing the hardware configuration of the audio processing device 20. As shown in FIG. The audio processing device 20 includes a CPU (Central Processing Unit) 201 , a ROM (Read Only Memory) 202 , a RAM (Random Access Memory) 203 and a host bus 204 . The audio processing device 20 also includes a bridge 205, an external bus 206, an interface 207, an input device 208, a display device 209, an audio output device 210, a storage device (HDD) 211, a drive 212, a network and an interface 215 .

The CPU 201 functions as an arithmetic processing device and a control device, and controls overall operations within the sound processing device 20 according to various programs. Alternatively, the CPU 201 may be a microprocessor. The ROM 202 stores programs, calculation parameters, and the like used by the CPU 201 . The RAM 203 temporarily stores programs used in the execution of the CPU 201, parameters that change as appropriate during the execution, and the like. These are interconnected by a host bus 204 comprising a CPU bus or the like. The CPU 201, the ROM 202, the RAM 203, and the software cooperate to operate the first encoding unit 224, the first decoding unit 232, the control unit 240, the forwarding unit 248, the second encoding unit 252, the second communication unit 256, and the like. can be realized.

The host bus 204 is connected via a bridge 205 to an external bus 206 such as a PCI (Peripheral Component Interconnect/Interface) bus. It should be noted that the host bus 204, bridge 205 and external bus 206 do not necessarily have to be configured separately, and these functions may be implemented in one bus.

The input device 208 includes input means for the user to input information, such as a mouse, keyboard, touch panel, button, microphone, sensor, switch, and lever, and generates an input signal based on the user's input, and outputs the input to the CPU 201. It consists of a control circuit and the like. By operating the input device 208, the user of the speech processing device 20 can input various data to the speech processing device 20 and instruct processing operations.

The display device 209 includes, for example, a liquid crystal display (LCD) device, a projector device, an OLED (Organic Light Emitting Diode) device and a lamp. Also, the audio output device 210 includes audio output devices such as speakers and headphones.

The storage device 211 is a data storage device configured as an example of a storage unit of the audio processing device 20 according to the present embodiment. The storage device 211 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded on the storage medium, and the like. The storage device 211 is configured by, for example, a HDD (Hard Disk Drive), an SSD (Solid Storage Drive), or a memory having equivalent functions. The storage device 211 drives storage and stores programs executed by the CPU 201 and various data.

The drive 212 is a reader/writer for storage media, and is built in or externally attached to the audio processing device 20 . The drive 212 reads out information recorded in the attached removable storage medium 24 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and outputs it to the RAM 203 or the storage device 211 . Drive 212 can also write information to removable storage medium 24 .

The network interface 215 is, for example, a communication interface configured with a communication device or the like for connecting to the network 12 . The network interface 215 may be a wireless LAN (Local Area Network) compatible communication device or a wired communication device that performs wired communication.

Note that the hardware configuration of the audio processing device 20 described above can also be applied to the conference server 30 .

<7. Supplement>
Although the preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention belongs can conceive of various modifications or modifications within the scope of the technical idea described in the claims. , of course, are also understood to belong to the technical scope of the present invention.

For example, each step in the processing of the speech processing device 20 of this specification does not necessarily have to be processed in chronological order according to the order described as a sequence diagram or flowchart. For example, each step in the processing of the audio processing device 20 may be processed in an order different from the order described as the flowchart, or may be processed in parallel.

It is also possible to create a computer program for causing hardware such as the CPU, ROM, and RAM built into the audio processing device 20 to exhibit functions equivalent to the components of the audio processing device 20 described above. A storage medium storing the computer program is also provided.

20 audio processing device 220 audio input unit 224 first encoding unit 228 first communication unit 232 first decoding unit 236 audio output unit 238 operation unit 240 control unit 244 first buffer 248 forwarding unit 252 second encoding unit 256 second Communication unit 260 Second decoding unit 264 Second buffer 30 Conference server

Claims (14)

a receiving unit that receives the first encoded data from another data processing device;
a decoding unit that decodes the first encoded data received by the receiving unit;
an encoding unit that encodes the data obtained by the decoding unit to generate second encoded data;
Controlling transmission of the second encoded data to the other communication device when it is determined that the first encoded data is data transmitted from the other communication device in a specific operation mode a control unit that
A data processing device comprising: The data processing device further comprises an output unit that outputs audio or video based on the data obtained by the decoding unit,
The output unit outputs the data obtained by the decoding unit when the control unit determines that the first encoded data is data transmitted from the other data processing device in the specific operation mode. 2. The data processing apparatus of claim 1, wherein no audio or video is output based on. The encoding unit generates the second encoded data using a second processing method that causes less quality deterioration than the first processing method used to generate the first encoded data. 3. A data processing apparatus according to claim 1 or 2. 3. The control unit controls transmission of the second encoded data in a second communication scheme having higher reliability than the first communication scheme used for communication of the first encoded data. 4. The data processing device according to any one of 1 to 3. Based on the fact that a predetermined flag is added to the first encoded data, the control section controls that the first encoded data is data transmitted in a specific operation mode from the other data processing device. 5. The data processing device according to any one of claims 1 to 4, wherein the data processing device determines that there is a A data processing system having a first data processing device and a second data processing device,
The first data processing device,
transmitting first encoded data obtained by encoding data to the second data processing device;
The second data processing device is
a receiving unit that receives first encoded data from the first data processing device;
a decoding unit that decodes the first encoded data received by the receiving unit;
an encoding unit that encodes the data obtained by the decoding unit to generate second encoded data;
transmitting the second encoded data to the first data processing device when it is determined that the first encoded data is data transmitted from the first data processing device in a specific operation mode; a control unit for controlling the transmission of
A data processing system comprising: receiving first encoded data from another data processing device;
decoding the first encoded data;
encoding data obtained by decoding the first encoded data to generate second encoded data;
transmitting the second encoded data to the other data processing device when it is determined that the first encoded data is data transmitted from the other data processing device in a specific operation mode; and
An audio processing method, comprising: an encoding unit that encodes input data to generate encoded data;
a transmission unit that transmits the encoded data to another data processing device;
a receiving unit that receives the first encoded data or the second encoded data from the other data processing device;
Control for controlling audio or video output based on the first encoded data in the first operation mode, and controlling audio or video output based on the second encoded data in the second operation mode Department and
A data processing device comprising: The first encoded data is data transmitted using a first communication scheme, and the second encoded data is transmitted using a second communication scheme having higher reliability than the first communication scheme. is the data sent,
The receiving unit
A first receiving unit that corresponds to the first communication method and receives the first encoded data, and a second receiving unit that corresponds to the second communication method and receives the second encoded data ,
9. A data processing apparatus according to claim 8, comprising: The first encoded data is data generated using a first processing method, and the second encoded data is generated using a second processing method with less quality deterioration than the first processing method. is data generated using
The data processing device is
a first decoding unit that corresponds to the first processing method and decodes the first encoded data; and
a second decoding unit that corresponds to the second processing method and decodes the second encoded data;
10. A data processing apparatus according to claim 8 or 9, comprising: The data processing device further comprises a first buffer that holds input data,
11. The data processing device according to claim 8, wherein said control section supplies the data held in said first buffer to said encoding section. The data processing device further comprises a second buffer holding a plurality of data obtained by decoding a plurality of the second encoded data received from a plurality of other data processing devices,
12. The data processing device according to claim 8, wherein said control unit sequentially controls output of a plurality of data held in said second buffer. 13. The data processing apparatus according to any one of claims 8 to 12, wherein said transmission section transmits said encoded data together with a predetermined flag in said second operation mode. encoding input data to generate encoded data;
transmitting the encoded data to another data processing device;
receiving first encoded data or second encoded data from the other data processing device;
controlling audio or video output based on the first encoded data in a first operation mode, and controlling audio or video output based on the second encoded data in a second operation mode; When,
An audio processing method, comprising:

Images