JP2005184383A - System, apparatus and method for real-time data communication - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable long-time continuous reproduction by real-time transmission by coping with the positional shift of a clock unique to network transmission. <P>SOLUTION: This real-time data communication apparatus is provided with a receiver side terminal 21 having: a data receiving unit 22 for receiving real-time data transmitted from a transmitter terminal via a network; a sound reproducing unit 41 for reproducing the received real-time data; and a central operation processing unit 24 for controlling the statuses of the unit 22 and the unit 41 to receive the real-time data. The apparatus is also provided with: an increase/decrease determining unit 49 for monitoring the data quantity of a sound card buffering unit 45 of the unit 44 to synchronize a recording unit of the transmitter side terminal with the unit 41 of the terminal 21 to transmit the real-time data; and a thinning/interpolating unit 50 for controlling the data of a memory unit 43. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a real-time data communication system for performing real-time data communication, for example.

  Conventionally, in a system in which sound data is transmitted from a server to a terminal and the sound data is played back on the terminal side, the server and the terminal are synchronized by sending a clock together with the sound data from the server to the terminal in a short-distance connection via a cable. I was trying to take it. Further, when connecting at an arbitrary distance via a network, the server and the terminal are synchronized by sending a clock to the sound data packet from the server to the terminal.

  Such a synchronization method is called phase synchronization, and is performed so as to synchronize the operation clock between the arithmetic processing unit of the server and the terminal. There has also been a synchronization method called AV (Audio / Video) synchronization that synchronizes sound data and video data.

Patent Document 1 discloses a data processing device that can prevent the occurrence of data underflow or overflow without increasing the cost. In this data processing apparatus, for this purpose, the clock deviation measuring unit measures the clock deviation from the increase / decrease in the amount of data stored in the memory. Then, the control determination unit determines the position to perform the correction process and what kind of correction is to be performed based on the determination result of the clock deviation measurement unit and the data stored in the memory. Further, the data amount control unit executes the correction process by deleting or generating the data at the position where the correction process is performed based on the determination result of the control determination unit and the data stored in the memory. The data in the memory is rewritten with the correction data.
JP 2003-256191 A

  In the conventional synchronization method described above, digital audio data is generally sampled at a constant sampling frequency and transmitted by transmitting the clock component together with the digital audio data in the transmission via the network. This method has a disadvantage in that it cannot cope with a minute clock shift occurring between each terminal peculiar to transmission via a network and cannot withstand continuous reproduction for a long time, so that real-time transmission cannot be performed. .

  Also, if the clocks are shifted between the central processing unit and the sound playback unit inside the terminal device, it can be avoided by exchanging synchronization signals between them. The terminal has a disadvantage that such processing is impossible.

  In Patent Document 1, it is possible to prevent the occurrence of underflow or overflow of data inside the apparatus, but it is possible to deal with a minute clock shift that occurs between terminals when data transmitted over a network is processed. There was an inconvenience that it was not possible.

  Therefore, the present invention has been made in view of such a point, and copes with a clock shift peculiar to network transmission and enables real-time data communication system and real-time data communication apparatus capable of continuous reproduction for a long time by real-time transmission. It is another object of the present invention to provide a real-time data communication method.

  In order to solve the above-described problems and achieve the object of the present invention, a real-time data communication system of the present invention transmits a recording unit that records input real-time data, a transmission unit that transmits real-time data, and transmits real-time data. Therefore, a transmission unit having a recording unit and a control unit that controls the state of the transmission unit, a reception unit that receives real-time data transmitted from the transmission unit via the network, and a reproduction unit that reproduces the received real-time data And a receiving unit having a control unit for controlling the state of the receiving unit and the reproducing unit to receive the real-time data, and a recording unit of the transmitting unit and a reproducing unit of the receiving unit are synchronized to transmit the real-time data. And synchronization means for controlling the operation.

  Thus, in the real-time data communication system, the synchronization means controls to synchronize the recording unit of the transmission unit and the reproduction unit of the reception unit to transmit real-time data. A clock shift does not occur between the reproduction units.

  In addition, the real-time data communication apparatus of the present invention includes a receiving unit that receives real-time data transmitted from the transmission side via a network, a reproducing unit that reproduces the received real-time data, and a reception unit that receives the real-time data. A receiving unit having a control unit for controlling the state of the recording unit and the reproducing unit, a recording unit for recording real-time data input on the transmission side to transmit real-time data, and a reproducing unit of the receiving unit And synchronization means for controlling.

  Thus, in the real-time data communication apparatus, the synchronization means controls to synchronize the recording unit on the transmission side and the reproduction unit on the reception unit to transmit real-time data. A clock shift does not occur between the reproduction units.

  Further, in the real-time data communication method of the present invention, the transmitting unit records the input real-time data by the recording unit, the step of transmitting the real-time data by the transmitting unit, the recording unit for transmitting the real-time data, and A step of controlling the state of the transmission unit by the control unit, and at the reception unit, the step of receiving the real-time data transmitted from the transmission unit via the network by the reception unit, and the received real-time data by the reproduction unit A step of reproducing and a step of controlling the states of the receiving unit and the reproducing unit by the control unit in order to receive the real-time data, and for transmitting the real-time data, And a step for performing control to achieve synchronization by means of synchronization means. .

  Thus, in the real-time data communication method, in the step of obtaining synchronization, control is performed to synchronize the recording unit of the transmission unit and the reproduction unit of the reception unit in order to transmit real-time data. This eliminates the occurrence of a clock shift between the reproduction units.

  According to the real-time data communication system of the present invention, it is possible to eliminate the occurrence of clock deviation between the recording unit of the transmission unit and the reproduction unit of the reception unit, thereby improving the quality of real-time transmission. In addition, a real-time data communication system that enables continuous reproduction for a long time can be obtained.

  According to the real-time data communication apparatus of the present invention, it is possible to eliminate a clock shift between the recording unit on the transmission side and the reproduction unit of the reception unit, thereby improving the quality of the reproduction operation by real-time transmission. Therefore, a real-time data communication apparatus that can continuously reproduce for a long time can be obtained.

  In addition, according to the real-time data communication method of the present invention, it is possible to eliminate the occurrence of clock shift between the recording unit of the transmission unit and the reproduction unit of the reception unit, thereby improving the quality of real-time transmission. Thus, a real-time data communication method that enables continuous reproduction for a long time can be obtained.

Embodiments of the present invention will be described below with reference to the drawings as appropriate.
FIG. 1 is a diagram showing a configuration of a real-time data communication system applied to the embodiment of the present invention.
The present invention assumes a device configuration that transmits real-time data such as sound data. For ease of explanation, both terminals A and B indicated by 1 and 3 are on the transmitting side and the receiving side, but both may be capable of transmitting and receiving at the same time.

  In FIG. 1, when sound data is transmitted and received by terminals A and B indicated by 1 and 3 via the network 2, in order to correct a phase shift caused by a clock difference between the terminals, synchronization is performed to synchronize. A module is required. In general, by performing phase synchronization for matching operation clocks between operation processing units between terminals, it is possible to exchange data such as voice and video between different terminals on the network 2 in real time.

  If audio data and image data are exchanged without a synchronization module, it is expected that the delay in the operation on the reception side will increase with respect to the operation on the transmission side due to the clock difference between the terminals. For this reason, real-time reproduction cannot be performed on the receiving side. Alternatively, if data has not arrived at a certain reproduction timing required for real-time transmission, it may be misrecognized by the arithmetic processing unit on the receiving side, and real-time reproduction processing may not be performed. Therefore, in the present invention, the synchronization processing in the playback unit as described below is performed so that real-time playback can be performed on the receiving side.

FIG. 2 is a diagram showing the internal configuration of the terminal in the real-time data communication system.
In FIG. 2, the sound data input from the input unit 12 is supplied to the sound recording unit 13 in the terminal 11 inside the transmission side, that is, the capture side. The sound recording unit 13 temporarily stores the input sound data in a memory so that signal processing such as encoding is possible, and reads it out as necessary. The central processing unit 15 controls signal processing such as encoding of the sound data of the sound recording unit 13 and writing or reading operation.

  The sound data read by the sound recording unit 13 and subjected to signal processing such as encoding is supplied to the data transmission unit 14. The data transmission unit 14 performs signal processing such as shuffling for transmission and packet generation on the sound data, and transmits the sound data subjected to signal processing for transmission via the network 2 shown in FIG. To the playback-side terminal 21. The central processing unit 15 controls a signal processing operation for transmitting the sound data of the sound recording unit 13.

  Further, the data receiving unit 22 in the terminal 21 on the receiving side, that is, the reproduction side receives the sound data transmitted from the data transmitting unit 14 in the terminal 11 on the capturing side via the network 2. The data receiving unit 22 performs signal processing such as packet extraction and deshuffling for reception on the received sound data, and supplies the sound data subjected to the signal processing for reception to the sound reproduction unit 23. The central processing unit 24 controls operations of the data receiving unit 22 for reception.

  The sound reproducing unit 23 decodes and reproduces the sound data, generates an acoustic output signal, and supplies it to the output unit 25. The central processing unit 24 controls the decoding and sound output operations of the sound playback unit 23. The output unit 25 performs sound output based on the sound output signal.

  Here, the central processing unit 15 in the terminal 11 on the capture side and the central processing unit 24 in the terminal 21 on the playback side are synchronized in phase using the synchronization module as shown by 26. . However, as shown by 27, a synchronization shift peculiar to network transmission may occur in the sound recording unit 13 in the terminal 11 on the capture side and the sound reproduction unit 23 in the terminal 21 on the reproduction side.

  Sound reproduction that reproduces sound data even when phase synchronization is achieved as indicated by 26 in which the operation clocks are synchronized between the central processing units 15 and 24 of the transmission side terminal 11 and the reception side terminal 21 by the synchronization module described above. When there is a clock shift between the clock of the unit 23 and the central processing unit 24, a minute synchronization error occurs between transmission and reception.

  If the clock of the sound playback unit 23 is earlier than the clock of the central processing unit 24, there is a high possibility that a buffer underrun will occur in the receiving terminal 21 in which the amount of stored sound data is too small relative to the buffer memory. As a result, the sound output is interrupted. On the other hand, when the clock of the sound playback unit 23 is slow, the buffer overflow occurs in the receiving terminal 21 where the amount of stored sound data is too large for the buffer memory, resulting in a delay in sound output. Similarly, a phenomenon similar to that of the reception-side terminal 21 described above may occur on the transmission terminal 11 side due to a clock shift.

  In general, a synchronization module that performs phase synchronization used for network transmission synchronizes the clocks of the central processing units 15 and 24 between the terminals that perform transmission and reception as described above. The phase synchronization used for the above-described network transmission is used for the clock difference generated between the arithmetic processing unit 15 and the sound recording unit 13 and the clock difference generated between the slave-side reception sound reproduction unit 23 and the slave side. It cannot be solved only by the synchronization module.

  Therefore, as indicated by 33 in FIG. 3, it is necessary to establish synchronization between the sound recording / reproducing unit 31 and the sound recording / reproducing unit 32 between the terminals 1 and 3 that perform transmission / reception via the network 2. In practice, this method is not a practical solution because it is not possible to perform direct synchronization control between the sound recording / reproducing unit 31 and the sound recording / reproducing unit 32. Therefore, it can be said that performing the above-described phase synchronization and internal terminal synchronization is a direct solution.

  However, in the case of a terminal that cannot perform clock synchronization between the master and the slave, the above-described problem always occurs. Therefore, by using the method of the present invention as described below, sound data can be smoothly recorded and reproduced via a network.

FIG. 4 is a diagram illustrating a configuration example of a receiving terminal at the time of sound reproduction.
FIG. 4 shows a configuration in which the sound card unit 44 subsequent to the sound playback unit 41 is uniquely provided with a sound card buffer unit 45. However, the sound card unit 44 is not limited thereto, and the sound card unit 44 has its own sound card. If the buffer unit 45 is not provided, the sound playback unit 41 also plays a role of the sound card buffer unit 45 of the sound card unit 44.

  Explaining each function, the sound card buffer unit 45 of the sound card unit 44 passes the data stored in the buffer to the sound card 46 that performs digital / analog conversion at any time, so that the sound output from the output unit 25 is output. It has a mechanism to play and output without interruption. From the central processing unit 24 in the user space (left side of the dotted line), it is only necessary to control the timing of writing or reading sound data to the sound card buffer unit 45 of the sound card unit 44 and digital / analog conversion of the sound card 46. The flow of sound data in the sound card buffer unit 45 of the sound card unit 44 cannot be directly controlled.

  Therefore, the increase / decrease determination unit 49 provided in the sound control unit of the sound playback unit 41 and the decoding unit 42 monitors the amount of sound data in the sound card buffer unit 45 of the sound card unit 44 via the memory unit 43. Memory control is performed so that sound data is transferred to the sound card buffer unit 45.

  Here, the thinning / interpolating unit 50 provided in the memory unit 43 of the sound reproducing unit 41 is 47 based on the increase or decrease of the sound data amount of the sound card buffer unit 45 of the sound card unit 44 by the increase / decrease determination unit 49. As shown, the memory control is performed so that the data amount is thinned out or interpolated so that the amount of sound data transferred to the sound card buffer unit 45 is constant.

  Next, the data flow will be described. Thick line arrows indicate the flow of digital sound data. A thin line indicates a flow of control data from the central processing unit 24 to each unit. The data receiving unit 22 depacketizes the data sent from the transmission side shown in FIG. 2 via the network 2 shown in FIG. 1, that is, the capture-side terminal 11, that is, removes the header from the packet. Retrieve the data part. The depacketized digital sound data is passed to the sound playback unit 41.

FIG. 5 is a diagram illustrating an example of memory control for real-time performance.
In FIG. 5, using the synchronization module used for network transmission as described above, the CPU (Central Processing Unit) 54 on the transmission side 51 and the clock of the CPU 58 on the reception side 56 are synchronized by performing phase synchronization. Shall. Further, the CPUs 54 and 58 shown in FIG. 5 correspond to the central processing units 15 and 24 shown in FIG. 2, and the buffers 55 and 57 shown in FIG. 5 are added to the data transmission unit 14 and data reception unit 22 shown in FIG. Correspondingly, the sound card buffer 53 shown in FIG. 5 corresponds to the sound recording unit 13 shown in FIG. Note that the values of the deviations D1 to Dn shown below are examples of the delay time from the input of the digital sound data, and are not limited thereto.

  Here, it is assumed that the delay time generated between the sound card buffer 53 on the transmission side 51 and the CPU 54 is 32 ms as shown by the deviation D1, and the delay time generated between the CPU 58 on the reception side 56 and the buffer 57. When it is assumed that the time is 30 ms as shown by the deviation D2, the delay time generated between the CPU 58 and the sound card buffer 59 by performing the above-described memory control shown in 65 between the buffer 57 and the sound card buffer 59. As shown by the deviation D3, it should be within 38 ms.

  At this time, the data amount is thinned out so that the upper limit threshold value of the deviation D3 becomes 38 ms by memory control, or the data amount is interpolated so that the lower limit threshold value of the deviation D3 becomes 15 ms. Sound data is supplied from 57 to the sound card buffer 59.

  As a result, the delay time generated between the input side buffer 57 of the receiving side 56 and the output side speaker 60 can be within 68 ms as indicated by D4 (= D2 + D3). Thereby, in this network transmission system, the delay time generated between the input terminal 52 on the transmission side 51 and the speaker 60 on the reception side 56 can be made within 100 ms as indicated by Dn (= D1 + D2 + D3). For example, 100 ms indicated by Dn (= D1 + D2 + D3) is a value that is a fixed time that guarantees real-time transmission in network transmission.

  It should be noted that in the above description, there is no problem with the delay due to the delay with the CPU 54 buffer 55 on the transmission side 51 between the input side buffer 57 on the reception side 56 and the output side speaker 60. This is because there is no problem as long as the delay time generated in (1) can be set within 68 ms as indicated by D4 (= D2 + D3).

FIG. 6 is a flowchart showing the control of the sound control unit of the sound playback unit 41.
In FIG. 6, data is received in step S1. Specifically, the sound control unit and the decoding unit 42 of the sound reproduction unit 41 illustrated in FIG. 4 receive the sound data supplied from the data reception unit 22.

  In step S2, it is determined whether or not decoding is necessary. Specifically, the sound control unit and the decoding unit 42 of the sound reproduction unit 41 determine whether or not the received data has been subjected to an encoding process such as compression encoding.

  If the data received in step S2 has been subjected to an encoding process such as compression encoding, the process proceeds to step S3, where the data subjected to the encoding process such as compression encoding is decoded. Specifically, the sound control unit and the decoding unit 42 of the sound reproducing unit 41 perform decoding processing such as decompression decoding.

  Thereafter, in step S4, the data amount length of the sound card buffer unit is checked. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 monitors the amount of sound data in the sound card buffer unit 45 of the sound card unit 44.

  In step S5, it is determined whether or not there is a tendency to increase or decrease. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound playback unit 41 increases the result of monitoring the data amount of the sound data in the sound card buffer unit 45 of the sound card unit 44. Or it is judged whether it is a decreasing tendency.

If the monitoring result is increasing or decreasing in step S5, the process proceeds to step S6.
Control the memory section. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 transmits the sound data transferred to the sound card buffer unit 45 to the thinning / interpolation unit 50 of the memory unit 43. Memory control is performed so that the data amount is thinned out or interpolated so that the amount becomes constant. If the monitoring result is not increasing or decreasing in step S5, the process proceeds directly to step S7.

  In step S7, data is transferred to the sound card buffer unit. Specifically, the sound control unit and decoding unit 42 of the sound playback unit 41 passes the digital sound data to the sound card buffer unit 45 of the sound card unit 44 via the memory unit 43.

FIG. 7 is a flowchart showing the operation of the increase / decrease determination unit of the sound control unit. FIG. 7 shows the specific processing of the increase / decrease determination in step S5 of FIG.
In step S11, the data amount length of the sound card buffer unit is held. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 illustrated in FIG. 4 holds the data amount of the sound data in the sound card buffer unit 45 of the sound card unit 44. To do.

  In step S12, an average is obtained from the average Ave of the past data amount and the data amount length. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound playback unit 41 is configured to calculate the past average Ave of the amount of sound data in the sound card buffer unit 45 of the sound card unit 44 and the present The average is obtained from the data amount length. The reason for obtaining the average is that the operation system OS operates by time sharing, so the monitoring timing is not constant.

  In step S13, it is determined whether or not a timeout has occurred. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 determines whether or not the monitoring period has exceeded a certain time for guaranteeing real-time transmission, for example, 100 ms.

  If it is not timed out in step S13, the process proceeds to step S14 to recognize a state that is not timed out. More specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound playback unit 41 has a monitoring period exceeding a certain time for guaranteeing real-time transmission, for example, 100 ms as shown in FIG. Recognize that not.

  If it is timed out in step S13, the process proceeds to step S15, and it is determined whether or not the past average Ave> the average Ave_old of the previous time interval. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 is configured to calculate the average Ave_old of the data amount in the previous time section in which the average Ave of the past data amount is the output unit. It is determined whether or not.

  If it is determined in step S15 that the past average Ave> the average Ave_old of the previous time interval, the process proceeds to step S16 to determine whether the past average Ave> the upper limit threshold value. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 determines whether the average Ave of the past data amount is larger than the upper limit threshold 38 ms of the deviation D3 shown in FIG. Determine whether.

  If it is determined in step S16 that the past average Ave> the upper limit threshold, the process proceeds to step S17, where it is determined that the increase has occurred. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound playback unit 41 determines that the amount of sound data in the sound card buffer unit 45 of the sound card unit 44 has increased.

  If it is determined in step S15 that the past average Ave is not greater than the average Ave_old of the previous time interval, the process proceeds to step S18 to determine whether the past average Ave <the lower limit threshold value. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 determines whether or not the average Ave of the past data amount is smaller than the lower limit threshold 15 ms of the deviation D3 shown in FIG. Determine whether.

  If it is determined in step S18 that the past average Ave <the lower limit threshold, the process proceeds to step S19, where it is determined that the value is decreasing. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 determines that the amount of sound data in the sound card buffer unit 45 of the sound card unit 44 has decreased.

  When it is not time-out at step S13, when past average Ave> the upper limit threshold is not satisfied at step S16, or when past average Ave <the lower limit threshold is not satisfied at step S18, the process directly proceeds to step S14.

  In FIG. 7, the increase / decrease tendency is examined only when a threshold value exceeding a reference value corresponding to the number of samples of the timer or reproduced sound data is exceeded. By doing so, it is possible to reduce the overall calculation amount and to suppress frequent changes in the sampling frequency. Specifically, the average value Ave is obtained at regular time intervals, compared with the average Ave_old of the previous time interval, and the upper threshold value and the lower threshold value are compared to determine increase or decrease.

  In fact, in a program that uses a plurality of processes, there is a possibility that the timing of turning to the control unit may be shifted depending on the scheduling of the operation system OS. In addition, due to the late arrival or early arrival of sound data, fluctuations in the buffer amount may become severe. As a result, the amount of data stored in the sound card buffer unit is fluctuated, and thus a mechanism for determining the increase or decrease as described above is required. Furthermore, the upper and lower limit threshold values set here depend on how much real-time property is maintained.

  That is, the larger the upper limit threshold, the greater the delay. Therefore, it is necessary to reduce the delay to some extent in order to reduce the delay. The lower threshold is a threshold for preventing the buffer underrun of the sound card buffer from occurring. If this value is too small, the buffer underrun of the sound card buffer may occur frequently. Must be set to a value.

FIG. 8 is a flowchart showing the thinning and interpolation control of the memory unit of the sound control unit.
In FIG. 8, it is determined in step S21 whether or not there is an increase. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 illustrated in FIG. 4 increases the amount of sound data in the sound card buffer unit 45 of the sound card unit 44. Judge whether or not.

  If it is increased in step S21, the process proceeds to step S22, and the data stored in the memory unit is thinned out. Specifically, the decimation / interpolation unit 50 provided in the memory unit 43 of the sound playback unit 41 is 47 based on the increase in the amount of sound data in the sound card buffer unit 45 of the sound card unit 44 by the increase / decrease determination unit 49. As shown, the memory control is performed so that the data amount is thinned out so that the amount of sound data transferred to the sound card buffer unit 45 is constant.

  If it is not increased in step S21, the process proceeds to step S23 to interpolate the data stored in the memory unit. Specifically, the decimation / interpolation unit 50 provided in the memory unit 43 of the sound playback unit 41 is 47 based on the decrease in the amount of sound data in the sound card buffer unit 45 of the sound card unit 44 by the increase / decrease determination unit 49. As shown, the memory control is performed so that the amount of data is interpolated so that the amount of sound data transferred to the sound card buffer unit 45 is constant.

  In the control method of the sound control unit shown in FIG. 8, the amount of sound data in the memory unit 43 is interpolated by thinning or padding. When the buffer amount tends to increase, the sound data amount supplied to the sound card buffer unit 45 is adjusted so as to reduce the sound data amount of the memory unit 43, and when the buffer amount tends to decrease, the sound data buffer amount 45 is adjusted. The amount of adjustment at that time is Delta / Div, where Delta is the amount of digital sound data handled every hour. Here, Div is an integer and depends on the system.

FIG. 9 is a flowchart showing specific control of the memory unit of the sound control unit.
In FIG. 9, it is determined in step S31 whether or not there is an increase. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 illustrated in FIG. 4 increases the amount of sound data in the sound card buffer unit 45 of the sound card unit 44. Judge whether or not.

  If it is an increase in step S31, the process proceeds to step S32 to determine whether or not there is a sudden change. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 illustrated in FIG. 4 increases the data amount of the sound data in the sound card buffer unit 45 of the sound card unit 44. It is determined whether or not is changing rapidly.

  If there is no abrupt change in step S32, the process proceeds to step S33, and the data is deleted. Specifically, the thinning / interpolating unit 50 provided in the memory unit 43 of the sound reproducing unit 41 is adapted to abrupt changes in the increase in the amount of sound data in the sound card buffer unit 45 of the sound card unit 44 by the increase / decrease determination unit 49. Based on the absence, the data in the memory unit 43 is deleted so that the amount of sound data transferred to the sound card buffer unit 45 becomes constant as indicated by 47.

  If it is not increased in step S31, the process proceeds to step S34 to determine whether or not there is a sudden change. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 illustrated in FIG. 4 reduces the data amount of the sound data in the sound card buffer unit 45 of the sound card unit 44. It is determined whether or not is changing rapidly.

When there is no sudden change in step S34, the process proceeds to step S35, and the previous data is repeated. Specifically, the decimation / interpolation unit 50 provided in the memory unit 43 of the sound playback unit 41 is adapted to abrupt change in decrease in the amount of sound data in the sound card buffer unit 45 of the sound card unit 44 by the increase / decrease determination unit 49. Based on the absence, the data in the previous time section of the memory unit 43 is repeatedly copied so that the amount of sound data transferred to the sound card buffer unit 45 becomes constant as indicated by 47.
If there is a sudden change in step S32, or if there is a sudden change in step S34, the process ends immediately.

  As described above, when the data in the memory unit 43 is thinned, there is a sudden change in the volume level or the like with reference to the data in the previous time interval of the data, or the data in the subsequent time interval if some delay is allowed. In such a case, the memory control process is not performed, and data is deleted if no sudden change is observed. On the contrary, in the case of inflating, similarly, only when there is no sudden change, it is possible to inflate by reinserting the data of the previous time interval.

FIG. 10 is a flowchart showing the operation of the increase / decrease determination unit of the sound control unit when clearing all buffers.
In step S41, the data amount length of the sound card buffer unit is held. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 illustrated in FIG. 4 holds the data amount of the sound data in the sound card buffer unit 45 of the sound card unit 44. To do.

  In step S42, it is determined whether data amount length> threshold value. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 determines whether or not the data amount length of the sound card buffer unit 45 is larger than the abnormal threshold. This abnormal threshold value is, for example, a value twice the upper limit threshold value 38 ms of the deviation D3 shown in FIG.

  If the data amount length> the threshold value in step S42, the process proceeds to step S43, and the sound card buffer is cleared. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 clears the data in the sound card buffer unit 45.

  In step S45, an average is obtained from the average Ave of the past data amount and the data amount length. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound playback unit 41 is configured to calculate the past average Ave of the amount of sound data in the sound card buffer unit 45 of the sound card unit 44 and the present The average is obtained from the data amount length. The reason for obtaining the average is that the operation system OS operates by time sharing, so the monitoring timing is not constant.

  In step S46, it is determined whether or not a timeout has occurred. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 determines whether or not the monitoring period has exceeded a certain time for guaranteeing real-time transmission, for example, 100 ms.

  If it is not timed out in step S46, the process proceeds to step S44 to recognize a state that is not timed out. More specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound playback unit 41 has a monitoring period exceeding a certain time for guaranteeing real-time transmission, for example, 100 ms as shown in FIG. Recognize that not.

  If it is timed out in step S46, the process proceeds to step S47, and it is determined whether or not the past average Ave> the average Ave_old of the previous time interval. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 is configured to calculate the average Ave_old of the data amount in the previous time section in which the average Ave of the past data amount is the output unit. It is determined whether or not.

  If it is determined in step S47 that the past average Ave> the average Ave_old of the previous time interval, the process proceeds to step S48 to determine whether the past average Ave> the upper limit threshold value. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 determines whether the average Ave of the past data amount is larger than the upper limit threshold 38 ms of the deviation D3 shown in FIG. Determine whether.

  When it is determined in step S48 that the past average Ave> the upper limit threshold, the process proceeds to step S49, and it is determined that the increase has occurred. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound playback unit 41 determines that the amount of sound data in the sound card buffer unit 45 of the sound card unit 44 has increased.

  If it is determined in step S47 that the past average Ave> the average Ave_old of the previous time interval is not reached, the process proceeds to step S50, and it is determined whether or not the past average Ave <the lower limit threshold value. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 determines whether or not the average Ave of the past data amount is smaller than the lower limit threshold 15 ms of the deviation D3 shown in FIG. Determine whether.

  If it is determined in step S50 that the past average Ave <the lower limit threshold, the process proceeds to step S51, and it is determined that the value is decreasing. Specifically, the increase / decrease determination unit 49 provided in the sound control unit and the decoding unit 42 of the sound reproduction unit 41 determines that the amount of sound data in the sound card buffer unit 45 of the sound card unit 44 has decreased.

  If the sound card buffer is cleared in step S43, if it is not timed out in step S46, if the past average Ave is not the upper threshold value in step S50, or if the past average Ave is not less than the lower threshold value in step S48, the process proceeds directly to step S44. To do.

FIG. 11 is a diagram illustrating an example of thinning, interpolation, and clear.
In FIG. 11, when the amount of sound data supplied to the sound card buffer unit 45 from the memory unit 43 shown in FIG. Here, Div is an integer and depends on the system.

  Here, at time T, when the amount of sound data in the sound card buffer unit 45 is larger than the upper limit threshold 111, the amount of sound data in the memory unit 43 is thinned as indicated by 115. At this time, as shown at 112, the data of the current time interval (T) is thinned, and as shown at 113, the data of the previous (T-1) time interval is processed so as not to be thinned.

  At time T-2, when the amount of sound data in the sound card buffer unit 45 is smaller than the upper limit threshold 111, the amount of sound data in the memory unit 43 is interpolated by padding as shown at 115, and the amount of sound data is, for example, 32 ms. Set the data amount Delta corresponding to minutes.

  Further, at time Tn-3, when the amount of sound data in the sound card buffer unit 45 is smaller than the lower threshold 116, the amount of sound data in the memory unit 43 is indicated by 117, as shown in Tn-2, Tn-1, Interpolation is performed until the lower threshold 116 at the time point Tn is exceeded.

  Further, when the amount of sound data in the sound card buffer unit 45 exceeds an abnormal threshold value (= upper limit threshold value × 2) indicated by 118, only the current data is output to the sound card buffer side as indicated by 119 and indicated by 120. Clear the abnormal value in the sound card buffer as follows.

  As described above, when the amount of sound data in the sound card buffer unit 45 shown in FIG. 4 tends to increase, the sound card buffer tends to decrease when the amount of sound data in the memory unit 43 tends to decrease. The amount of sound data supplied to the unit 45 is adjusted.

  In the processing described above, the thinning / interpolating unit 50 provided in the memory unit 43 of the sound reproducing unit 41 shown in FIG. 4 performs the increase / decrease determination unit 49 of the sound card buffer unit 45 of the sound card unit 44 with respect to the memory unit. This is based on an increase or decrease in the amount of sound data.

FIG. 12 is a diagram illustrating an example of an abrupt change.
In FIG. 12, the thinning / interpolating unit 50 provided in the memory unit 43 of the sound reproducing unit 41 shown in FIG. 4 is the sample level of the sound data 125 of the sound card buffer unit 45 of the sound card unit 44 by the increase / decrease determination unit 49. 121, the data in the memory unit 43 is thinned out so that the amount of sound data transferred to the sound card buffer unit 45 is constant as indicated by 124 based on the fact that there is no rapid change in the amount of sound data. Like that.

  Further, when the sample level 121 of the sound data 123 is viewed and there is a sudden change in the volume level of the sound data, the thinning process is not performed. Here, the determination of the abrupt change is made by, for example, | past average Ave−average Ave_old |> 1000 bytes of the previous time interval. In real-time transmission, 1000 bytes corresponds to the amount of data transmitted in a time of 16 ms. 2000 bytes corresponds to the amount of data transmitted in the above-described time of the upper threshold 38 ms.

  Here, the same processing is performed for the case of a sudden decrease that shows only the case of a sudden increase. Similarly, in this process, the thinning / interpolating unit 50 provided in the memory unit 43 of the sound reproducing unit 41 shown in FIG. It is performed based on a sudden increase or decrease in the amount of sound data of 45.

  As described above, in order to deal with a case where sound data accumulates in the sound card buffer unit due to some unexpected failure, all sound card buffer units are forcibly cleared when a certain abnormal threshold is exceeded. In this way, there is a high possibility that the sound output will be cut off, but when considering real-time communication rather than delaying due to the accumulation of digital sound data in the sound card buffer unit, such a safety valve mechanism was adopted. More stable operation is possible. Furthermore, by doing in this way, it becomes possible to keep the delay between transmission and reception below a threshold value.

  By performing the memory control described above, it is possible to avoid buffer underrun and buffer overflow of the sound card buffer unit. Further, by determining whether the sound data is thinned out or padded, the sound control unit of the sound reproduction unit 41 and the increase / decrease determination unit 49 provided in the decoding unit 42 determine the control state. You can't do that.

In addition, when the data amount of the sound card buffer unit is greatly increased, it is possible to avoid a sudden increase in delay by providing an abnormal threshold value.
Moreover, since a large amount of buffering is not required, it can be said that it is suitable for real-time communication via a network.

  The embodiment of the present invention is not limited to the sound data described above, but can be applied to other continuous data used for real-time transmission, for example, video data.

It is a figure which shows the structure of the real-time data communication system which performs the phase synchronization applied to embodiment of this invention. It is a figure which shows the structure inside the terminal in a real-time data communication system. It is a figure which shows an unreal synchronization method. It is a figure which shows the structural example of the receiving side terminal at the time of sound reproduction. It is a figure which shows the example of the memory control for real-time property. It is a flowchart which shows control of a sound control part. It is a flowchart which shows operation | movement of the increase / decrease judgment part of a sound control part. It is a flowchart which shows control of the thinning-out and interpolation of the memory part of a sound control part. It is a flowchart which shows the concrete control of the memory part of a sound control part. It is a flowchart which shows operation | movement of the increase / decrease judgment part of the sound control part at the time of clearing all the buffers. It is a figure which shows the example of thinning-out, interpolation, and clear. It is a figure which shows the example of an abrupt change.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Terminal A, 2 ... Network, 3 ... Terminal B, 11 ... Terminal inside (capture side), 12 ... Input part, 13 ... Sound recording part, 14 ... Data transmission part, 15 ... Central processing unit, 21 ... inside terminal (playback side), 22 ... data receiving unit, 23 ... sound playback unit, 24 ... central processing unit, 25 ... output unit, 26 ... phase synchronized, 27 …… Out of synchronization 31 …… Sound recording / playback unit 32 …… Sound recording / playback unit 33 …… Synchronization 41 …… Sound playback unit 42 …… Sound control unit and decoding unit 43 …… Memory 44 ... Sound card unit 45 ... Sound card buffer unit 46 ... Sound card 47 ... Constant rate 48 ... Monitoring 49 ... Increase / decrease judgment unit 50 ... Decimation / interpolation unit 51 …… Sender 2 ... Input terminal, 53 ... Sound card buffer, 54 ... CPU, 55 ... Buffer, 56 ... Receiver, 57 ... Buffer, 58 ... CPU, 59 ... Sound card buffer, 60 ... Speaker , 61... Phase synchronization, 62... Constant rate, 111... Upper limit threshold, 112... Thinning out current (T) data, 113. Thinning, 115 ... Interpolation, 116 ... Lower threshold, 117 ... Interpolation, 118 ... Abnormal threshold, 119 ... Output only current data to the sound card buffer, 120 ... Abnormal value to sound card Clear with buffer, 121 …… View sample level, 122 …… Do not thin out, 123 …… Data with sudden change, 124 …… Thin out, 125 …… No sudden change Over data, 126 ...... criteria of rapid change

Claims (14)

In a real-time data communication system for receiving and reproducing real-time data transmitted from a transmission side via a network on a reception side,
A recording unit that records input real-time data, a transmission unit that transmits the real-time data, and a transmission unit that controls a state of the recording unit and the transmission unit to transmit the real-time data;
A receiving unit that receives real-time data transmitted from the transmitting unit via the network, a reproducing unit that reproduces the received real-time data, and controls the states of the receiving unit and the reproducing unit to receive the real-time data A receiving unit having a control unit,
A real-time data communication system comprising: synchronization means for controlling to synchronize the recording unit of the transmitting unit and the reproducing unit of the receiving unit for transmitting the real-time data. The real-time data communication system according to claim 1, wherein
The real-time data communication system according to claim 1, wherein the synchronization means is provided in the reproduction unit of the reception unit. The real-time data communication system according to claim 1, wherein
The real-time data communication system, wherein the synchronization means is used together with a phase synchronization means for synchronizing the control section of the transmission section and the control section of the reception section. The real-time data communication system according to claim 2,
The real-time data communication system according to claim 1, wherein the synchronization means includes a determination unit that determines whether the output data amount of the reproduction unit of the reception unit is increased or decreased. The real-time data communication system according to claim 4,
The synchronization means includes a thinning-out / interpolating unit for thinning out or interpolating the data amount so that the output data amount becomes constant based on the increase or decrease in the output data amount by the determining unit. Data communication system. In the real-time data communication apparatus on the receiving side that receives and reproduces the real-time data transmitted from the transmission side via the network,
A receiving unit that receives real-time data transmitted from the transmitting side via the network, a reproducing unit that reproduces the received real-time data, and controls the states of the receiving unit and the reproducing unit to receive the real-time data A receiving unit having a control unit,
A recording unit for recording the real-time data input on the transmission side and a synchronization unit for controlling to synchronize the reproduction unit of the reception unit for transmitting the real-time data are provided. Real-time data communication device. The real-time data communication apparatus according to claim 6, wherein
The real-time data communication apparatus according to claim 1, wherein the synchronization unit is used together with a phase synchronization unit that synchronizes the control unit that controls the transmission side and the control unit of the reception unit. The real-time data communication apparatus according to claim 6, wherein
The real-time data communication apparatus according to claim 1, wherein the synchronization means includes a determination unit that determines whether the output data amount of the reproduction unit of the reception unit is increased or decreased. The real-time data communication apparatus according to claim 8,
The synchronization means includes a thinning-out / interpolating unit for thinning out or interpolating the data amount so that the output data amount becomes constant based on the increase or decrease in the output data amount by the determining unit. Data communication device. In a real-time data communication method in a real-time data communication system for receiving and reproducing real-time data transmitted from a transmission side via a network on a reception side,
In the transmitter,
A step of recording input real-time data by a recording unit; a step of transmitting the real-time data by a transmission unit; and a step of controlling a state of the recording unit and the transmission unit by a control unit to transmit the real-time data; Have
In the receiver,
A step of receiving real-time data transmitted from the transmission unit via a network by a receiving unit; a step of reproducing the received real-time data by a reproducing unit; and the receiving unit and the reproducing unit for receiving the real-time data And a step of controlling the state of
A real-time data communication method comprising: a step of controlling the synchronization unit to synchronize the recording unit of the transmission unit and the reproduction unit of the reception unit for transmitting the real-time data. The real-time data communication method according to claim 10.
The real time data communication method according to claim 1, wherein the synchronization step is provided during the reproduction step of the reception unit. The real-time data communication method according to claim 10.
The step of performing control for synchronization is used in combination with the step of phase synchronization for synchronizing the control unit of the transmission unit and the control unit of the reception unit. The real-time data communication method according to claim 11.
A real-time data communication method comprising: a step of determining, by a determination unit, an increase or decrease in the amount of output data of the reproduction unit of the reception unit during the step of performing control for synchronization. The real-time data method according to claim 13.
During the step of controlling to achieve the synchronization, the data amount is thinned out or interpolated by the thinning / interpolating unit so that the output data amount becomes constant based on the increase or decrease in the output data amount by the determining step. The real-time data communication method characterized by providing the step to perform.

Images