JPH07170292A - Transmitter - Google Patents

Abstract

PURPOSE:To reduce the time delay from input to output of a sound data buffer memory and a video data buffer memory and to make the time lag between media in the output small as well. CONSTITUTION:This transmitter is provided with the sound data buffer memory 82, the video data buffer memory 83, a buffer content amount measuring circuit 86 for respectively measuring the respective content amounts of the respective buffer memories 82 and 83 and an arithmetic judgement circuit 84 and a compression parameter specifying circuit 85 for comparing the respective measured results of the content amounts of the memories 82 and 83 with the respective upper limit values of the content amounts of buffers and specifying a compression parameter for turning compressibility in a sound data compression circuit 67 and a video data compression circuit 69 to high compressibility in the compression parameter specifying circuit 85 when it is judged that the respective measured values exceed the respective upper limit values.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a transmitter used in the technical field of communication such as a computer network, and more particularly to a transmitter capable of supporting multimedia data such as audio and video data. .

[0002]

2. Description of the Related Art In recent years, the performance of personal computers, workstations and the like has been remarkably improved and is becoming widespread. In addition, recently, there has been a demand for utilizing them for daily communication like televisions and telephones, and a demand for utilizing these information processing systems in a wider range of applications. Therefore, in order to meet these demands, it is necessary for the above-mentioned personal computers and workstations to be able to interactively handle multimedia data such as voice and moving images in a distributed environment.

Here, as the information processing communication system used in the conventional multimedia communication, the configuration shown in FIG. 11 is known.

In FIG. 11, the input device 100 or 1
From 01, for example, audio data, moving image data, etc. are supplied. As the input devices 100 and 101, for example, A / A that converts an analog audio signal from a microphone or an analog video signal from a video camera into a digital signal
A D converter can be mentioned. This input device 10
The data from 0 and 101 are sent to the input data processing devices 102 and 103, respectively, and subjected to predetermined data processing. Examples of data processing in the input data processing devices 102 and 103 include data compression processing. The input data processing device 102, 103
The compressed data from is subjected to multiplexing processing by the multiplexer 104, is stored in the transmission buffer memory 105, is then read out, and is transmitted to the network transmission unit 106.
Is sent to the network receiving unit 111 on the receiving side via a network such as Ethernet.

The data received by the network receiving section 111 is temporarily stored in the receiving buffer memory 112 and then read out, and further, the separating device 113 performs a separating process corresponding to the multiplexing process on the transmitting side. It is sent to the output data processing devices 114 and 115.
The output data processing devices 114 and 115 perform data expansion processing corresponding to the input data processing devices 102 and 103, and then send the data to the reproducing devices 116 and 117. A D / A converter can be given as an example of the reproducing devices 116 and 117.

[0006]

However, the above-described conventional information processing communication system has the following problems.

First, in the above-described conventional information processing communication system, the processing may be delayed due to, for example, fluctuations in the load of the communication path, and the contents in the transmission buffer memory 105 may increase too much. As described above, when the content of the transmission buffer memory 105 increases too much, the time delay from the input of data to the output of the transmission buffer memory 105 increases.

Further, in the example of FIG. 11, the transmission buffer memory is single, but in the case where there are a plurality of transmission buffer memories, for example, the difference in the internal capacity between the plurality of transmission buffer memories becomes large. If this happens, the time lag between the buffer memories at the time of output (that is, the time lag between the media) becomes large even though these multiple transmission buffer memories should originally be synchronized. .

This conventional drawback will be described more specifically. For example, multimedia data such as moving images and audio have properties that are essentially different from the numerical data and text data processed by computers.
That is, the multimedia data is that firstly, the multimedia data is not just a byte string, but has an attribute of time explicitly or implicitly. Secondly, the amount of data is essentially huge and redundant. However, even if the processing capacity of the hardware is improved, it is necessary to compress the data in order to handle it efficiently. Thirdly, interactive processing is required, and not only throughput but also performance such as response and delay are required. It has the characteristics of being valued.

Therefore, in order to handle these in a distributed environment, it is necessary to introduce new methods for media synchronization, communication, processing, resource management, and the like.

Therefore, the present invention has been made in view of the above situation, and provides a transmitting apparatus in which the time delay from input to output is small and the time difference between media at output is small. The purpose is.

[0012]

The transmitting apparatus of the present invention is proposed in order to achieve the above-mentioned object, and the difference between the signal acquisition time and the transmission time is set within a predetermined time, and the voice is transmitted. And / or a real-time transmission device for transmitting a video signal, the capturing means for capturing the signal, the compression means for compressing the audio and / or video signal, and the audio and / or
Alternatively, a transmitting means for transmitting the moving image signal, a first detecting means for detecting that the difference between the capturing time of the voice and / or moving image signal and the transmitting time is within the first time as the upper limit, the voice and And / or second detecting means for detecting that the difference between the capturing time of the moving image signal and the transmitting time is within the second time as the lower limit, buffer means for buffering the signal, and capturing time of the signal When it is detected that the difference between the transmission time and the transmission time exceeds the first time, the difference between the signal acquisition time and the transmission time is calculated between the first time and the second time. In order to achieve the above, there is provided a control means for controlling the compression of the audio and / or moving image signal.

Further, the transmitting apparatus of the present invention is provided with a third detecting means for detecting that the transmission signal is a signal excluding the voice and / or the moving image signal, and transmits the signal excluding the voice and / or the moving image signal. At the same time, when the third detection means detects that the transmission signal is a signal excluding the audio and / or moving image signals, the signal excluding the audio and / or moving image signals is processed as it is.

Further, the transmitting apparatus of the present invention is further provided with a fourth detecting means for detecting a predetermined portion of the transmission signal, and when the fourth detecting means detects the predetermined portion, the predetermined portion. Is processed as it is, and transmission of other parts is stopped.

[0015]

According to the present invention, the control means transmits the signal acquisition time when it detects that the difference between the signal acquisition time and the transmission time exceeds the lower limit of the first time. Controlling the compression of the audio and / or video signals to make the difference between the first time and the second time.
That is, the difference between the acquisition time and the reception time is the first
If the time exceeds the first time, the time delay from the input to the output in the buffer means becomes large. Therefore, when the difference between the acquisition time and the transmission time exceeds the first time, the audio and / or video signal is output. Is controlled to bring the difference between the first time and the second time so that the time delay from the input to the output of the buffer means is restored to an appropriate value.

Further, according to the transmitting apparatus of the present invention, a third detecting means is further provided, and when it is detected by the third detecting means that the transmission signal is a signal excluding a voice signal and / or a moving image signal, By performing the processing as it is, it is possible to prevent the content of the data from changing when the transmission signal is, for example, text data.

Further, according to the transmitting apparatus of the present invention, a fourth detecting means is further provided, and when a predetermined portion of the transmission signal is detected by this fourth detecting means, the other portion except the predetermined portion is detected. By stopping the transmission of the buffer means, the time delay from the input to the output of the buffer means is restored to an appropriate value, and by processing the predetermined part as it is, the signal of this predetermined part is transmitted. To

[0018]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 shows the configuration of a transmitter according to the embodiment of the present invention. As shown in FIG. 1, the transmitter of the present embodiment is a real-time transmitter that transmits a voice signal and a moving image signal with a difference between a signal acquisition time and a transmission time being within a predetermined time. And A / D conversion circuits 65 and 66 and a frame buffer 66 as a capturing means for capturing a video signal, an audio data compression circuit 67 and a video data compression circuit 69 for compressing audio and video data, and buffering audio and video data. The audio data buffer memory 82 and the video data buffer memory 83 as the buffer means for performing the operation, and the network transmission section 7 as the transmission means for transmitting the audio and video data.

Furthermore, in the transmitting apparatus of this embodiment, the difference between the acquisition time and the transmission time of the audio and moving image data (measured value D described later) is the upper limit (upper limit value Dma described later).
x) the first detection means for detecting that it is within the first time, and the difference between the acquisition time and the transmission time for the audio and moving image data is the lower limit (lower limit value Dmin described later).
Of the buffer memory internal capacity measuring circuit 86 as a second detecting means for detecting that it is within the second time, and the difference between the fetching time and the transmission time (measurement value D ) Has exceeded the first time (upper limit value Dmax), the difference between the acquisition time and the transmission time is calculated as the difference between the first time (upper limit value Dmax) and the second time. Within time (lower limit value Dmi
n)), an operation judgment circuit 84 and a compression parameter designating circuit 85 are provided as control means for controlling the compression parameters in the audio data compression circuit 67 and the video data compression circuit 69.

That is, in the transmitter of this embodiment, the difference (measurement value D) between the fetch time and the transmission time in the buffer memory internal capacity measuring circuit 86 exceeds the first time (upper limit value Dmax). When it is detected, the calculation determining circuit 84 controls the compression parameter designating circuit 85 to change the compression rate in the audio data compressing circuit 67 and the video data compressing circuit 69. As a result, the transmitter of this embodiment controls the amounts of audio data and video data to be sent to the audio data buffer memory 82 and the video data buffer memory 83.

Further, the transmitting apparatus of the present embodiment can also transmit signals other than audio and moving picture signals, and therefore the third signal for detecting that the transmitted signal is a signal excluding audio and moving picture signals. A detection circuit 93 is further provided as a detection means of the. When the detection circuit 93 detects that the transmission signal is a signal excluding voice and moving image, the data processing circuit 26 processes the signal as it is. The signals other than the audio and moving image signals may include, for example, text data, program data, numerical data, and other binary data.

Further, the detection circuit 93 of the transmission apparatus of the present invention also operates as a fourth detection means for detecting a significant portion of a transmission signal, for example, an important portion of video or audio, and the fourth detection means. The detection output from the detection circuit 93 as a means is also sent to the calculation determination circuit 84. In the calculation determination circuit 84, the compression parameter designating circuit 85 is controlled based on the detection output, so that only the predetermined portion is directly output to the audio data compression circuit 67 and the video data compression memory 69 or low. Control is performed such that the processing is performed at the compression rate and the other parts are not output or are processed at the high compression rate.

Here, first, prior to a detailed description of the transmitter of the embodiment of the present invention shown in FIGS. 1 and 9 to 10, multimedia data of an information processing system to which the transmitter of the present invention is applied. The basic functions of the compatible information processing device, the model of the information processing device, and the so-called application programming interface (API: applicationpr)
The following describes the gramming interface), the positioning of the information processing device supporting the multimedia in the information processing communication system, and the evaluation of the information processing, divided into the following items.

1. Functions of multimedia-compatible information processing device in information processing communication system The goals of this information processing device are (1) There are various restrictions, but they are built on a standard personal computer or workstation and are compatible with existing systems. To maintain compatibility, (2) leave audio and video communication and specific processing to the information processing device, and the client program only controls it. (3) Media data attributes and purpose of client program Flexibly according to the following, (4) Capable of responding to changes in the load of resources such as processors and networks, (5)
The model is simple, and so on.

From these requirements, the information processing apparatus needs to have the following functions.

1.1 Media Type Media data handled by the information processing device basically includes audio and moving images. These are given temporal attributes such as the amount of data per unit time and the generation time, and these are used for processing in the information processing device. For example, in the case of voice data having a data width of 8 bits and a sampling frequency of 8 kHz, the unit time is 1/8000 (sec) and the data amount per unit time is 1 byte.

1.2 Information Processing Device and Media Device The information processing device receives data from a plurality of input devices,
Output to the output device while synchronizing in time.

The input device is (1) a hardware device such as an audio interface or a video input interface, (2) a multimedia data file such as a sound file, a moving image file or a movie file, (3) a multicast address, (4) ) Support client process etc.

Output devices include (1) devices such as audio interfaces and windows, (2) multimedia files, (3) multicast addresses, (4)
Supports client processes, etc.

There is only one information processing device in a host, and the device that it directly handles must be on that host. If you need the input and output devices to be on different hosts,
The client accesses each information processing device on each host and connects the information processing devices.

1.3 Transfer and Synchronization of Media Data The following can be considered as criteria for evaluating the quality of objects provided to users as a multimedia system. For example,
(1) Allowable limit of transfer delay, (2) Allowable limit of inter-media synchronization, (3) Throughput, (4) When data loss is allowed and when it is not allowed.

In order to obtain a satisfactory quality on a given transfer path, various parameters at the time of transfer such as data amount, compression method, protocol and packet size are controlled. The information processing device performs inter-media synchronization according to the quality evaluation standard.

1.4 Network Protocol In the network protocol of multimedia data used by the information processing apparatus, it is necessary to dynamically control the flow and the like while considering quality evaluation criteria. Fiber of the so-called Ethernet (Ethernet) compliant with the IEEE 802.3 standard of the current network environment and 100 Mbit / sec token passing system using optical fiber proposed in SC13 of International Standards Organization (ISO) Distributed
Data interface (FDDI: fiber-distributed
Since network devices such as data interface) share and share resources individually in a distributed environment, it is difficult to provide services that secure network resources in advance. However, considering the compatibility with the current network environment, the Internet Protocol (IP: internet
It is necessary to use the so-called transmission control protocol (TCP) as a network protocol this time.
and a so-called user datagram protocol (UDP) which is a connectionless protocol.

1.5 Data Compression / Expansion The information processing apparatus has a software / hardware compression / expansion function for audio data and moving image data. As the voice compression method, the International Governor of Telegraph and Telephone Communications Advisory Committee (CCITT) Recommendation G. 711,
G. 721, G.I. 722, G.I. Supports standards such as 728. The image compression method is the so-called JPEG (J) of the international standardization work group of the color still image coding method of the International Telegraph and Telephone Communication Advisory Committee (CCITT).
oint Picture Expert Group) and video coding recommendations for video conferencing systems. 261, MPEG (Moving Picture E) of the International Standardization Working Group on color video coding
xpert Group). Since each compression method has its own characteristics, it is necessary to use them properly depending on the application.

1.6 Resource Control The information processing apparatus has a function of processing input / output requests from a plurality of client programs. For example, when there are voice input requests from a plurality of clients even though there is only one voice input device, the following processing method can be considered. (1) Copy and distribute to all requesters. (2) Prioritize first-come-first-served requests and reject second-come-first-served requests. (3) Switch clients sequentially. This function is used by programs such as a multimedia manager, which corresponds to the window manager in the window system.

1.7 Control of Physical Device When handling multimedia data such as voices and moving images, it is necessary to connect so-called audio-visual equipment (AV equipment) such as a video camera and a VCR. The information processing apparatus supports a predetermined protocol for controlling AV equipment for these controls.

2. Model of Information Processing Device and API 2.1 Model of Information Processing Device to which AV Equipment is Connected As shown in FIG. 2, the information processing device has one execution control unit (AVobj) for each client. To generate. When the client wants to input / output multimedia data, it makes a request to the information processing device by the following procedure. First, a virtual media device (AVdev) is opened in the execution control unit (AVobj). Since the virtual media device (AVdev) is not a physical device, exclusive control and opening from multiple devices can be realized.

As shown in FIG. 3, the execution control unit (AVobj) that opens the virtual media device (AVdev) for input and the execution control unit (AVobj) that opens the media device (AVdev) for output. ) Is connected to secure a transfer path for multimedia data. The synchronization of media between devices opened in the same execution control unit (AVobj) is guaranteed.

Further, as shown in FIG. 4, by generating and connecting an execution control unit (AVobj) in an information processing device on different hosts connected by a network, multimedia data can be transferred in workstations in a distributed environment. Be seen.

A telephone can be realized by using a sound device as a device (AVdev) opened by the execution control unit (AVobj) on the transmission and reception sides. In addition, opening a video device enables a videophone call. When a movie file is specified as an input device and a sound device and a video device are opened as output devices, a movie player is created. As described above, by changing the input / output devices, various multimedia applications can be easily created.

2.2 API of Information Processing Device The following are prepared in the library of the information processing device, for example.

Int avs_new (char * hostname);
An execution control unit (AVobj) is generated in the information processing device activated on the host name (hostname). If an error occurs, null is returned. In the case of normal termination, the ID (identification information) of the execution control unit (AVobj) is returned. All instructions for the execution control unit (AVobj) are executed using the ID of this execution control unit (AVobj).

Int avs_open (int net, char * devname,
int mode); This opens the device (AVdev). The arguments are the ID of the execution control unit (AVobj), the device name,
Mode.

Int avs _connect (int net1, int net2);
It connects two execution control units (AVobj) point-to-point. When one of the execution control units (AVobj) connected by this becomes the transfer state by the function (avs_transfer) described below, the other execution control unit (AVobj) can receive the data and automatically receives it. Process the data. Since one reception execution control unit (AVobj) can be connected to one transmission execution control unit (AVobj), one-to-many data transfer can be processed.

Int avs_transfer (int net, int dev, i
nt length); this controls the transmission of the execution control unit (AVobj). If 0 is specified as the device ID, the execution control unit (AVobj) is valid for all the opened devices. If a positive number is specified for the length, data is transferred for that length (unit is msec). If you specify zero here, transfer is performed until the next function (av_transfer) is given. Specify a negative number to stop immediately.

Int avs_destroy (int net); This releases the execution control unit (AVobj) generated by the function (av_new).

Int avs _interval (int net, int dev, i
nt interval); This sets the transfer interval. The unit is msec. When zero is set as the device ID, it is applied to all the devices opened in the execution control unit (AVobj).

Int avs _resize (int net, int dev, int
width, int height); This requires the video device to be resized. The unit of size is pixel. When zero is set as the device ID, it is applied to all video devices opened in the execution control unit (AVobj).

Int avs _nettype (int net, char * typ
e); This sets the type of network that connects the execution control units (AVobj). Currently TCP
And UDP is supported. This is a function (avs_conn
ect) must be done before executing.

Int avs_fd (int net); This returns the file descriptor of the control connection connection between the client and the execution control unit (AVobj).

Int avs _codec (int net, char * type, i
nt quality); This specifies the compression method of the media data. Here, only the JPEG compression / expansion is supported for image data.

In addition, there are the following libraries for directly accessing the media data to the device (AVdev). For example, int avs _read (int net, int dev, int shmid, int si
ze); int avs _write (int net, int dev, int shmid, int s
ize); int avs _ioctl (int net, int dev, int request, int
shmi);

3. Mounting of Information Processing Device Positioning of the information processing device in the entire system is as shown in FIG. 3.1 Use of Threads Using threads makes the context switch time shorter than that achieved by using multiple processes, the environment such as memory can be shared between threads, and programming becomes easy. The following two methods are conceivable as methods of assigning threads to models for realizing an information processing device.

For example, each function is assigned to a thread,
There is a thread allocation for each data stream.
The former is a method of allocating threads for each function of screen input / output, audio input / output, network, etc. to form a pipeline, and the latter is a method of allocating threads for processing to perform input to output for each media data.

Even if a pipeline is formed for each function, there is no advantage unless in a multiprocessor environment, and the latter is suitable for performing schedule priority control in stream units in a plurality of streams. The method of allocating threads for each data stream was also used.

3.2 Transmission / Reception Buffer and Delay of TCP / IP In order to realize high reliability in the TCP / IP, packet ordering, checksum, timeout and retransfer are performed, resulting in a large overhead and a data transfer delay. It becomes a problem.

Therefore, a buffer is prepared here for transmission and reception. For example, in a workstation, the default is 8K (bytes).
The accumulation of data in this buffer causes a delay. For example, in the case of a screen having a resolution of 160 × 120 and a depth of 16 bits, one screen has approximately 38 Kbytes, and even if the buffers on both the transmitting side and the receiving side are combined, no frame is buffered. In this case, increasing the buffer size improves the transfer efficiency. However, when the image is compressed to a size of about 1/10, ie, about 4 Kbytes, about 4 frames are accumulated in both buffers. If transfer is performed at a speed of 5 frames per second, this alone results in a delay of about 1 second.

However, conversely, if the buffer is made too small, the transfer efficiency will deteriorate, and there is a trade-off here. The size of this send / receive buffer is set by the function (setsockop
It can be changed by t).

4. Evaluation of information processing device The evaluation when the information processing device is mounted on a workstation is as follows.

4.1 Transmission / Reception Buffer, Transfer Delay, and Efficiency As an example of actually measuring the size of the transmission / reception buffer, transfer delay, and efficiency, transfer of moving image data between two workstations connected to the same network. Was done. Here, the workstation is connected by Ethernet, the resolution is 160 × 120, 16-bit depth uncompressed and JPEG compressed images are transferred and measured. One frame size is uncompressed, approximately 38 Kbytes, JPEG compressed. It is about 5K bytes.

The transfer delay was measured by adjusting the clock using a function (timed (8)) and measuring the time from image capture to transfer and display on the remote side. Furthermore, as the transfer efficiency, the maximum transfer frame rate is measured. Figure 6
Shows the relationship between the transmission / reception buffer size of an image that has not been compressed, and FIG.

As shown in FIG. 6, when the size of one frame is large without compression, the difference in transfer delay with respect to the size of the buffer remains almost unchanged because one frame does not fill the transmission / reception buffer. When the buffer size is 8 Kbytes, good results are obtained in terms of transfer delay and efficiency.
According to this FIG. 6, it can be understood that the transfer efficiency by TCP / IP is the most efficient in that case also from the fact that the default transmission / reception buffer size is 8 Kbytes.

In comparison, when transferring the JPEG-compressed image shown in FIG. 7, JPE is executed by software.
The maximum frame rate is small because G is compressed and expanded.
Moreover, since several frames are accumulated in the transmission / reception buffer, the delay increases as the size of the buffer increases. Therefore, when image data is sufficiently smaller than the bandwidth of the network by performing compression such as JPEG, the transfer delay of media data can be shortened by reducing the transmission / reception buffer size so that transfer efficiency does not decrease.

4.2 Screen Transfer Speed Under the same environment, the screen size, frame rate and transfer delay are measured. First, when Ethernet was used as the network, the results in the following table were obtained. Table 1
Is a 16-bit deep uncompressed image, Table 2 is JPEG
The maximum transfer frame rate and delay performance of compressed images are shown.

[0066]

[Table 1]

[0067]

[Table 2]

Next, as a network, an integrated service digital network (ISDN) is integrated.
Table 3 shows the performance when using 2B) of 1B (64K). In addition, when ISDM is used, the frame rate becomes 0.2 (fps) even if the image data is not compressed and the minimum screen size (80 × 60) is not practical. Evaluation of image transfer performance was omitted.

[0069]

[Table 3]

By using such an information processing apparatus, a multimedia application can be easily created without being conscious of capturing and compressing / decompressing images and sounds, and network programming.

4.4 Realization of synchronization between media This information processing apparatus synchronizes and transfers media data by interleaving into a single stream. Further, since TCP / IP is used as the network protocol, the packet order is guaranteed. Furthermore, if a plurality of pieces of media data are captured at the same time and interleaved to guarantee synchronization during data transmission, it can be assumed that the plurality of pieces of media data are also synchronized on the data receiving side. In fact, the synchronization of audio and video media data transferred over a single network in this way has been satisfactory.

4.5 Relationship with Delay In the above-mentioned implementation, TCP / IP, which is standard in the operating system UNIX, is used as the network protocol.
Is used. Here, as shown in FIG. 8, in an environment in which media data can be directly sent, the information processing apparatus can control the buffers on the transmitting side and the receiving side, and fine flow rate control can be performed.

Now, returning to FIG. 1, a transmitting apparatus according to an embodiment of the present invention that specifically realizes the above will be described.

In FIG. 1, audio signals and video signals (moving image signals) from the microphone 1 and the video camera 2 are sent to the corresponding A / D conversion circuits 65 and 66, respectively.
/ D converted. The audio data from the A / D conversion circuit 65 is sent to the audio data compression circuit 67, and as described above, as the audio compression method, for example, G.264. 711, G.I. 7
21, G.I. 722, G.I. The compression is performed using any one of 728 and the like. Further, the A / D conversion circuit 66
The video data from the video data is sent to the video data compression circuit 69 via the frame buffer 68, and as described above, the moving picture compression method is, for example, JPEG, H.264. 261, M
The compression is performed using any one of PEG and the like.

The compressed audio data and video data from the compression circuits 67 and 69 are sent to the audio data buffer memory 82 and the video data buffer memory 83 provided correspondingly.

Further, the buffer memory internal capacity measuring circuit 86
Measures the internal capacity of the memories 82 and 83 from the write address and the read address of the audio data buffer memory 82 and the video data buffer memory 83, respectively, and sends the measurement result to the calculation determination circuit 84.

In the calculation judgment circuit 84, the contents of the video data buffer memory 83 are determined based on the internal capacity measurement results (measurement value D described later) of the audio data buffer memory 82 and the video data buffer memory 83 from the measurement circuit 86. When it is detected that the amount exceeds the upper limit (Dmax described later), a signal to that effect is sent to the compression parameter designating circuit 85 as a control signal.

Further, in the operation judging circuit 84, the internal capacity measurement result (measurement value D described later) from the measuring circuit 86 indicates the internal capacity of the audio data buffer memory 82 and the video data buffer memory 83. If the value is below the lower limit (Dmin described later), audio and video may be interrupted, so a signal to that effect (exceeding the upper limit) is sent to the compression parameter designating circuit 85 as a control signal.

The compression parameter designation circuit 85 designates compression parameters for the audio data compression circuit 67 and the video data compression circuit 69 based on the control signal.

When the audio data compression circuit 67 and the video data compression circuit 69 receive the above-mentioned compression parameters, the audio data compression circuit 67 and the video data compression circuit 69 perform a compression process on the supplied audio data and video data, respectively.

For example, when the compression parameter indicates a high compression rate, the amount of data sent to the corresponding buffer memories 82 and 83 can be reduced, and conversely, the compression parameter indicates a low compression rate. In some cases, the amount of data sent to each buffer memory 82, 83 can be increased.

That is, in the transmitting apparatus of this embodiment, the compression parameter designation control is performed by performing the processing of the flowchart shown in FIG. 9 below.

In FIG. 9, in step S51,
The measured values D of the internal capacities of the audio data buffer memory 82 and the video data buffer memory 83 are obtained.
The measured value D is calculated by the following formula. D = (buffer read address)-(buffer write address)

At the next step S52, from the measured values D of the audio data and the video data, and the respective upper limit values Dmax of the internal capacities of the audio data buffer memory 82 and the video data buffer memory 83, respectively, If D> Dmax is determined, and if NO is determined, the process proceeds to step S53. Conversely, if YES is determined, the process proceeds to step S54.

In step S54, the compression parameter designating circuit 85 designates a compression parameter for increasing the compression rate in the audio data compression circuit 67 and the video data compression circuit 69 respectively, and the audio data compression circuit 67 is designated.
To the video data compression circuit 69. This step S54
After that, the process returns to step S61.

On the other hand, if the result of the determination in step S52 is NO, then in step S53 the measured values D
And D <Dmin from the respective lower limit values Dmin of the internal capacities of the audio data buffer memory 82 and the video data buffer memory 83. When it is determined No in step S53, the process returns to step S51,
If YES is determined, the process proceeds to step S55.

In step S55, the compression parameter designating circuit 85 designates a compression parameter for reducing the compression rate in each of the audio data compression circuit 67 and the video data compression circuit 69, and the audio data compression circuit 67 is designated.
To the video data compression circuit 69. This step S55
After that, the process returns to step S61.

As a result, in the transmitting apparatus of this embodiment, the internal capacities of the audio data buffer memory 82 and the video data buffer memory 83 increase, and the buffer memory 8
Even if the time delay from the input to the output and the time lag in 2 and 83 exceed the proper values, the time feed and the time lag can be restored to the proper values,
Further, even if the contents of the audio data buffer memory 82 and the video data buffer memory 83 become small and there is a risk that audio or video will be interrupted, the audio or video can be prevented from being interrupted.

The compressed audio data read from the audio data buffer memory 82 as described above and the compressed video data read from the video data buffer memory 83 are multiplexed by the multiplexer 5. After that, it is sent to the network transmission unit 7, and from the network transmission unit 7 via the network 8 such as Ethernet, the network reception unit 11 on the reception side.
Sent to.

Further, the input terminal 89 of the transmitter of the present embodiment.
A signal, such as text data, program data, numerical data, or other binary data, which is a signal excluding audio and moving image signals, is input as a transmission signal. This data is sent to the detection circuit 93. The detection circuit 9
In 3, the data processing circuit 9 outputs the data as it is when the data other than the audio and the moving image is detected.
Send to 4.

The data processing circuit 94 performs, for example, error correction processing as the predetermined processing on the supplied data, and then sends it to the data buffer memory 95. The data buffer memory 95 temporarily stores the data and then sends the data to the multiplexer 5.

As described above, the transmitting side of the present embodiment has the above-mentioned detection circuit 93 so that other data other than the voice and the moving picture can be transmitted as the transmission signal.

Further, on the transmitting side of this embodiment, the above A /
The respective audio data and video data from the D conversion circuits 65 and 66 are also sent to the detection circuit 93.

In the detection circuit 93 at this time, for voice data, for example, a low-frequency portion is detected from the voice data in consideration of human auditory characteristics, or particularly important such as a human voice portion. Detects various voice parts. Also,
For video data, for example, in consideration of human visual characteristics, a particularly visually important video portion is detected from the video data, or a particularly important portion in the video content is detected.
The detection circuit 93 outputs detection signals corresponding to the audio data and the video data, respectively, and the detection signals are sent to the calculation determination circuit 84.

When the arithmetic decision circuit 84 receives the detection signal corresponding to the audio data from the detection circuit 93, it outputs a control signal to the compression parameter designation circuit 85.

When the compression parameter designating circuit 85 receives the control signal for audio data from the operation judging circuit 84, it compresses, for example, an important part of the audio data in the compression processing in the audio data compression circuit 67. The parameter is controlled to be a low compression rate or to be a parameter when compression is not performed. On the contrary, the compression parameter designating circuit 85 controls the non-important part of the audio so that the compression parameter has a high compression rate according to the internal capacity of the audio data buffer memory 82.

As described above, in the transmitting apparatus of the present embodiment, the compression rate is increased for the portion excluding the important portion of the voice so that the time delay from the input to the output of the voice data buffer memory 82 is set to an appropriate value. Can be restored to
Also, by setting a low compression rate or non-compression for the important part of the voice, it becomes possible to maintain high quality of the transmitted voice.

On the other hand, the compression parameter designating circuit 85 described above.
When receiving the control signal for the video data from the arithmetic decision circuit 84, for example, the compression parameter at the time of the compression processing in the video data compression circuit 69 for the important part of the video data is set to a low compression rate. Or control to set the parameter when compression is not performed. On the contrary, the compression parameter designating circuit 85 controls the non-important part of the video so that the compression parameter has a high compression rate in accordance with the internal capacity of the video data buffer memory 83.

As a result, in the transmitting apparatus of this embodiment, the compression rate is increased in the portions other than the important portions of the video so that the time delay from the input to the output of the video data buffer memory 83 becomes a proper value. Can be recovered,
Further, by setting a low compression rate or non-compression to the important part of the video, it becomes possible to maintain high quality of the video to be transmitted.

Next, on the receiving side of this embodiment, the network receiving section 11 for receiving the above-mentioned transmission data.
The data received by is separated into the compressed audio data and video data by the separation circuit 12 and sent to the corresponding audio data buffer memory (FIFO memory) 13 and video data buffer memory (FIFO memory) 14, respectively. .

The audio data and the video data read from the audio data buffer memory 13 and the video data buffer memory 14 are respectively subjected to the corresponding decompression processing and D / A.
It is sent to the decompression D / A conversion circuits 18 and 19 which perform conversion.
In the expansion D / A conversion circuits 18 and 19, the A / D
Decompression processing corresponding to each compression processing in the conversion compression circuits 3 and 4 is performed, and then D / A converted and output.

The audio signal from the expansion D / A conversion circuit 18 is sent to the speaker 19, and the expansion D / A conversion circuit 1 is transmitted.
The video signal from 9 is sent to the monitor display 24.

The received signal supplied to the separation circuit 12 is also sent to the detection circuit 25 via the separation circuit 12. The detection circuit 25 detects signals such as the text data, program data, numerical data, and other binary data, which are signals obtained by removing the audio and moving image signals from the received signal.

When the detection circuit 25 detects data other than the audio and moving image signals, the detection circuit 25 outputs a detection signal to the separation circuit 12. When the detection signal from the detection circuit 25 is supplied, the separation circuit 12 separates only the data corresponding to the detection signal from the reception signal and sends it to the data processing circuit 26.

In the data processing circuit 26, the supplied data is subjected to predetermined processing and then output from the output terminal. It should be noted that examples of the predetermined processing include error correction processing and the like.

As described above, the receiving side of this embodiment has the detection circuit 25, and the separation circuit 12 responds to the detection signal from the detection circuit 25 to detect data other than voice or moving image from the reception signal. By separating the above, it is possible to process data other than the above-mentioned audio and moving images.

Next, the configuration of another embodiment of the present invention is shown in FIG.
Shown in.

In FIG. 10, a monitor display 40 and a microphone 4 are provided on the transmitting side host computer.
1 and the video camera 42 are connected, and the network 4
3 is connected to the receiving side host computer.

In the transmitting host computer,
The CPU 30 controls each unit by using the program data held in the main memory 31, and at the same time, the buffer memory internal capacity measuring circuit 86, the operation judging circuit 84, the detecting circuit 93, the data processing circuit 94, etc. of FIG. 1 described above. It has the function of.

The video signal from the video camera 42 is A /
The A / D conversion is performed by the D conversion compression processing unit 38 and the compression processing as described above is performed, and the buffer memory 34
Temporarily stored in. The buffer memory 34 has a part that functions as the video data buffer memory 83 of each of the above-described embodiments and a part that also functions as the frame buffer 68. However, although an example in which the video data before compression is taken in the frame buffer 68 is given, the buffer memory 34 shown in FIG.
In the above example, the compressed video data is loaded.

On the other hand, the audio signal from the microphone 41 is A / D converted by the A / D conversion compression processing section 37, subjected to the above-described compression processing, and temporarily stored in the buffer memory 33. This buffer memory 33
Has a function as the audio data buffer memory 82 of FIG.

In the frame memory 32, for example, C
The video frame data formed by the PU 30 and the video frame data based on the video imaged by the video camera 42 are stored. The video frame data from the frame memory 32 is sent to the monitor display 40 via the display interface 36 and displayed.

Further, the compressed audio data and video data stored in the buffer memories 33 and 34 are multiplexed, for example, by the CPU 30, and then temporarily stored in the buffer memory 35 and then read. The buffer memory 35 also functions as the data buffer memory 95 of FIG. The data read from the buffer memory 35 is the network interface 39.
It is sent to the receiving side host computer via the network 43 connected to.

Next, a monitor display 61 and a speaker 62 are connected to the receiving side host computer, and further connected to the transmitting side host computer via the network 43.

In the receiving host computer,
The CPU 52 uses the program data held in the main memory 53 to control each unit and perform various calculations.

The data sent from the sending host computer via the network 43 is transferred to the buffer memory 51 via the network interface 50.
It is stored once in and then read.

From the received data read from the buffer memory 51, for example, the compressed audio data and video data are separated by the CPU 52 and sent to the corresponding buffer memories 54 and 55, respectively.

The compressed video data sent to the buffer memory 54 is read out from the buffer memory 54 and sent to the data expansion processing unit 56, where expansion processing corresponding to the compression in the transmission side host computer is performed. It The decompressed video data is stored in the frame memory to form a frame, and the display interface 58
It is sent to and displayed on the monitor display 61 via.

Further, the compressed audio data sent to the buffer memory 55 is read from the memory 55 and sent to the data expansion processing unit 59, where the expansion processing corresponding to the compression in the transmission side host computer is performed. Done. The decompressed audio data is converted to an analog audio signal by the D / A conversion processing unit 60 and then sent to the speaker 62.

As described above, according to each of the embodiments of the present invention, each measured value D of the internal capacity of the audio data buffer memory 82 and the video data buffer memory 83 corresponds to the internal capacity of each of the buffer memories 82 and 83. When the upper limit value Dmax and the lower limit value Dmin of the audio data buffer memory 82 and the video data buffer memory 83 are detected, the difference between the upper limit value Dmax and the lower limit value Dmin is detected. Since the specification of the compression parameters of the audio and moving image signals is controlled in order to maintain the interval, the time delay and the time lag from the input to the output in the audio data buffer memory 82 and the video data buffer memory 83 are controlled. It is possible to recover to an appropriate value, and it is possible to prevent interruptions in audio and video.

Further, the transmitting apparatus of this embodiment is provided with the detecting circuit 93, and when the detecting circuit 93 detects that the received signal is the data excluding the moving image and the sound, the data processing circuit 94 directly processes it. By doing so, when the transmission signal is, for example, text data, the content of the data is prevented from changing.

Furthermore, in the transmitter of this embodiment,
When the detection circuit 93 detects a particularly important part in the video from the transmission signal, the arithmetic determination circuit 84 controls the compression parameter designating circuit 85, and the other parts other than these important parts are compressed at a high compression rate. By doing so, the time delay from the input to the output of the audio data buffer memory 82 and the video data buffer memory 83 is restored to an appropriate value, and the particularly important part is processed as it is by a low compression rate or non-compression. By doing so, it is possible to prevent the quality of audio and video from deteriorating for the signal in this portion.

[0123]

As described above, in the transmitting apparatus of the present invention, when it is detected that the difference between the signal acquisition time and the transmission time exceeds the first time, the signal acquisition time and the transmission time are transmitted. The time delay from the input to the output of the buffer means is adjusted to an appropriate value by controlling the compression ratio of the audio and / or video so that the difference between the time and the time between the first time and the second time is controlled. It is possible to recover to the above and to prevent interruption of audio and video.

Further, in the transmitting apparatus of the present invention, the third
When the transmission means detects that the transmission signal is a signal excluding the audio and / or moving image signals, the processing is performed as it is, and the content of the data is changed when the transmission signal is, for example, text data. It can be prevented.

Further, according to the transmitting apparatus of the present invention, the fourth
When the predetermined portion of the transmission signal is detected by the fourth detecting means, the transmission of the other portion excluding the predetermined portion is stopped, so that the time from the input to the output of the buffer means is increased. The delay can be recovered to an appropriate value, and by processing the predetermined portion as it is, the signal of the predetermined portion can be transmitted.

[Brief description of drawings]

FIG. 1 is a block circuit diagram showing a schematic configuration of a transmitting apparatus and an apparatus on the receiving side corresponding to the transmitting apparatus of an embodiment of the present invention.

FIG. 2 is a diagram showing a model of an information processing device of an information processing communication system to which the device of the present invention is applied.

FIG. 3 is a diagram for explaining transfer of multimedia data.

FIG. 4 is a diagram for explaining transfer of multimedia data between hosts connected by a network.

FIG. 5 is a diagram for explaining the positioning of the information processing device in the entire system.

FIG. 6 is a characteristic diagram showing a relationship between a transmission / reception buffer size, transfer delay, and efficiency in the case of no compression.

FIG. 7 is a characteristic diagram showing a relationship between a transmission / reception buffer size of an image in JPEG compression, transfer delay, and efficiency.

FIG. 8 is a diagram for explaining a relationship between an information processing device and delay.

FIG. 9 is a flowchart for explaining a process of measuring an internal capacity of a video data buffer memory and designating a compression parameter in the apparatus of this embodiment.

FIG. 10 is a block circuit diagram showing a schematic configuration of a transmission-side host computer and a reception-side host computer according to another embodiment of the present invention.

FIG. 11 is a block circuit diagram showing a schematic configuration of a conventional transmitter and a transmitter.

[Explanation of symbols]

1, 41 ... Microphone 2, 42 ... Video camera 5 ... Multiplexer 6 ... Data buffer memory 7 ... Network transmitter 8 ... Network 9 ... Network receiver 12 ... Separation circuit 13, 82 ... Audio data buffer memory 14, 83 ... Video data buffer memory 18 ... Audio data expansion D / A conversion circuit 19 ... Video data expansion D / A conversion circuit 24, 40, 61 ... Monitor display 25, 93 ... Detection circuit 26, 94 ... Data processing circuit 27, 62 ... Speaker 30, 52 ... CPU 31, 53 ... Main memory 32, 57 ... Frame memory 33, 34, 35, 51, 54, 55 ... Buffer memory 36, 58 ... Display interface 37 ..Voice data A / D conversion compression unit 38 ... Video data A / D conversion compression unit 56 ... Audio data data expansion unit 59 ... Video data data expansion unit 60 ... Audio data D / A conversion unit 65, 66 ... A / D conversion circuit 67 ... Audio data compression circuit 68 ... Frame buffer 69 ... Video data compression circuit 84 ... Calculation judgment circuit 85 ... Compression parameter designation Circuit 86 ... Buffer memory capacity measuring circuit 95 ... Data buffer memory

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification number Reference number within the agency FI Technical indication location H04N 5/38 7/10 7/24 (72) Inventor Mitsuru Tanabe 6-chome Kitashinagawa, Shinagawa-ku, Tokyo 7th 35th Sony Corporation

Claims (3)

[Claims] 1. A real-time transmitting device for transmitting a voice and / or video signal within a predetermined time with a difference between a signal capturing time and a transmitting time, comprising: a capturing means for capturing a signal; And / or a compression means for compressing the moving picture signal, a transmitting means for transmitting the sound and / or the moving picture signal, and a difference between the capturing time of the sound and / or the moving picture signal and the transmitting time is within the first time as an upper limit. And a second detecting means for detecting that the difference between the capturing time of the audio and / or moving image signal and the transmitting time is within the second time as the lower limit. And a buffer means for buffering the signal, and when detecting that the difference between the signal acquisition time and the transmission time exceeds the first time, the signal acquisition time and the transmission time The difference in order to achieve a between the first time and the second time, transmission apparatus characterized by a control means for controlling the compression of audio and / or video signals. 2. A third detection means for detecting that the transmission signal is a signal excluding a voice and / or a moving image signal is provided, and a signal excluding a voice and / or a moving image signal is also transmitted, and at the same time as the third aspect. The transmission signal is voice and / or detected by the detection means.
Alternatively, when it is detected that the signal is a signal excluding the moving image signal, the signal other than the sound and / or the moving image signal is processed as it is. 3. A fourth detecting means for detecting a predetermined portion of the transmission signal is provided, and when the fourth detecting means detects the predetermined portion, the predetermined portion is processed as it is,
The transmitting device according to claim 1, wherein transmission is stopped for other portions.