JPH07170290A - Communication system - Google Patents

Abstract

PURPOSE:To reduce time delay from input to output and to prevent the interruption of video images. CONSTITUTION:A reception side 91 measures the content amount of a video data buffer memory 14 and when it is detected that the measured value V of the content amount exceeds an upper limit value Vmax or falls below a lower limit value Vmin, the control signal of that effect is transmitted to a transmission side 90. When the control signal from the reception side 91 is received, the transmission side 90 varies the image quality selection of moving images or a frame dropping number corresponding to the contents. Namely, by varying the image quality selection of the moving images or the frames dropping number in a frame buffer 68 when the transmission side 90 receives the control signal from the reception side 91, the amount of moving image data to be transmitted to the reception side 91 is controlled.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a communication system used in the technical field of communication such as a computer network, and more particularly to a communication system capable of handling multimedia data such as audio and video data. .

[0002]

2. Description of the Related Art In recent years, the performance of personal computers, workstations and the like has been remarkably improved and is becoming widespread. In addition, recently, there has been a demand for utilizing them for daily communication like televisions and telephones, and a demand for utilizing these information processing systems in a wider range of applications. Therefore, in order to meet these demands, it is necessary for the above-mentioned personal computers and workstations to be able to interactively handle multimedia data such as voice and moving images in a distributed environment.

Here, as an information processing communication system used in the conventional multimedia communication, a configuration shown in FIG. 13 is known.

In FIG. 13, the input device 100 or 1
From 01, for example, audio data, moving image data, etc. are supplied. As the input devices 100 and 101, for example, A / A that converts an analog audio signal from a microphone or an analog video signal from a video camera into a digital signal
A D converter can be mentioned. This input device 10
The data from 0 and 101 are sent to the input data processing devices 102 and 103, respectively, and subjected to predetermined data processing. Examples of data processing in the input data processing devices 102 and 103 include data compression processing. The input data processing device 102, 103
The compressed data from is subjected to multiplexing processing by the multiplexer 104, is stored in the transmission buffer memory 105, is then read out, and is transmitted to the network transmission unit 106.
Is sent to the network receiving unit 111 on the receiving side via a network such as Ethernet.

The data received by the network receiving section 111 is temporarily stored in the receiving buffer memory 112 and then read out, and further, the separating device 113 performs a separating process corresponding to the multiplexing process on the transmitting side. It is sent to the output data processing devices 114 and 115.
The output data processing devices 114 and 115 perform data expansion processing corresponding to the input data processing devices 102 and 103, and then send the data to the reproducing devices 116 and 117. A D / A converter can be given as an example of the reproducing devices 116 and 117.

[0006]

However, the above-described conventional information processing communication system has the following problems.

First, in the above-mentioned conventional information processing communication system, for example, the output data processing device 11 on the receiving side is used.
The processing may be delayed due to fluctuations in load such as 4,115, and the contents in the reception buffer memory 112 may become too large. In this way, the reception buffer memory 11
When the content of 2 becomes too large, the time delay from the input of data to the output of the reception buffer memory 112 becomes large.

Further, in the example of FIG. 13, the number of the reception buffer memories is one, but in the case where there are a plurality of reception buffer memories, for example, the difference in the internal capacity between the plurality of reception buffer memories becomes large. If too many of these receive buffer memories are originally supposed to be synchronized, the time lag between the buffer memories at the time of output (that is, the time lag between media) becomes large. .

This conventional drawback will be described more specifically. For example, multimedia data such as moving images and audio have properties that are essentially different from the numerical data and text data processed by computers.
That is, the multimedia data is that firstly, the multimedia data is not just a byte string, but has an attribute of time explicitly or implicitly. Secondly, the amount of data is essentially huge and redundant. However, even if the processing capacity of the hardware is improved, it is necessary to compress the data in order to handle it efficiently. Thirdly, interactive processing is required, and not only throughput but also performance such as response and delay are required. It has the characteristics of being valued.

Therefore, in order to handle these in a distributed environment, it is necessary to introduce new methods for media synchronization, communication, processing, resource management, and the like.

Therefore, the present invention has been made in view of the above circumstances, and provides a communication system in which the time delay from input to output is small and the time difference between media at output is small. The purpose is.

[0012]

The present invention has been proposed in order to achieve the above-mentioned object, and the difference between the time for capturing the moving image signal at the transmitting side and the time for reproducing at the receiving side is within a predetermined time. In the real-time communication system, the transmitting side includes a capturing means for capturing a moving image signal, a varying means for selecting the image quality of the moving image or varying the number of dropped frames, a control means for controlling the varying means, and the above control. A control signal input means to which a control signal from the receiving side to the means is input,
The receiving side confirms that the capturing means for capturing the transmitted moving image signal, the reproducing means for reproducing the moving image signal, and the difference between the capturing time of the moving image signal and the reproducing time are within the first time as the upper limit. First detecting means for detecting, and second detecting means for detecting that the difference between the capturing time of the moving image signal and the reproducing time is within a second time as a lower limit,
When it is detected that the difference between the buffer means for buffering the signal and the signal acquisition time and the reproduction time exceeds the first time period and / or the second time period falls below the second time period, To control the image quality or the number of frames to be transmitted within a unit time for the transmission side in order to make the difference between the signal acquisition time and the reproduction time between the first time and the second time. And a control signal transmitting means for transmitting a control signal.

Further, the receiving side of the communication system of the present invention is provided with third detecting means for detecting that the received signal is a signal excluding the moving image signal, and the receiving side also receives the signal excluding the moving image signal. At the same time, when the third detecting means detects that the received signal is a signal excluding the moving image signal, the signal excluding the moving image signal is processed as it is.

Further, the receiving side is provided with a fourth detecting means for detecting that the moving image signal is a predetermined portion, and the receiving side detects the predetermined portion of the moving image signal by the fourth detecting means. If so, only the predetermined part is processed as it is, and the other parts are discarded.

[0015]

According to the communication system of the present invention, on the receiving side, the difference between the signal acquisition time and the reproduction time may exceed the first time period and / or fall within the second time period. When detected, the difference between the signal acquisition time and the reproduction time is set between the first time and the second time. Send control signals. Upon receiving this, the transmission side performs control to select the image quality of the moving image or change the number of dropped frames for the moving image signal to be transmitted.

That is, according to the communication system of the present invention, when the difference between the signal acquisition time and the signal reproduction time on the receiving side exceeds the first time which is the upper limit, from the input to the output of the buffer means. Since the time delay becomes large, if the difference between the signal acquisition time and the reproduction time on the receiving side exceeds the first time, the image quality of the moving image to be sent or the frame dropping is performed on the transmitting side. As a result, the time delay from the input to the output of the buffer means on the receiving side is restored to an appropriate value.

Further, according to the communication system of the present invention,
If the difference between the signal acquisition time and the reproduction time of the signal on the receiving side is less than the second time, the image may be interrupted. Therefore, the difference between the signal acquisition time and the reproduction time on the receiving side may occur. When the value falls below the second time, the image quality of the moving image to be sent is increased or the number of dropped frames is eliminated (or reduced) on the transmitting side to prevent the image from being interrupted on the receiving side. There is.

Further, according to the communication system of the present invention, the receiving side is provided with the third detecting means, and when the third detecting means detects that the received signal is a signal excluding the moving image signal, the processing is performed as it is. By doing so, when the received signal is, for example, text data, program data, numerical data, or other binary data, the contents of these data are prevented from changing.

Further, according to the communication system of the present invention,
Further, a fourth detecting means is provided on the receiving side, and when a predetermined portion of the moving image signal of the received signal is detected by the fourth detecting means, the other portion other than the predetermined portion is discarded, thereby the buffer means. The time delay from input to output is restored to an appropriate value, and the predetermined part is processed as it is, so that the signal of the predetermined part is not discarded.

[0020]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 shows the configuration of a communication system according to an embodiment of the present invention. As shown in FIG. 1, the communication system of the present embodiment is a real-time communication system in which the difference between the time taken by the transmitting side 90 and the time taken by the receiving side 91 to capture the moving image signal is within a predetermined time. Yes, the transmitting side 90 receives an A / D conversion circuit 66 and a frame buffer 68 as a capturing means for capturing a moving image signal, and a frame capturing as a varying means for selecting the image quality of a moving image or varying the number of dropped frames in the frame buffer 68. A clock generation circuit 70,
A capture cycle designating circuit 71 as a control means for controlling the frame capture clock generating circuit 70, and a network receiving section 72 as a control signal input means for receiving a control signal from the receiving side 91 to the capture cycle designating circuit 71. It is equipped with and.

Further, the receiving side 9 of the communication system of the present embodiment.
Reference numeral 1 denotes a network receiving section 11 as a capturing means for capturing the moving image signal transmitted from the transmitting side 90, a separation circuit 12 as a reproducing means for reproducing the moving image signal, and an expansion D.
The difference between the A / A conversion circuit 19 and the time for capturing the moving image signal and the time for reproducing the moving image signal (measured value V described later) is the upper limit (V described later).
max)) first detecting means for detecting that the time is within a first time, and a second time when a difference (measured value V) between a moving image signal taking time and a reproducing time is a lower limit (Vmin described later). A buffer memory internal capacity measuring circuit 20 as a second detecting means for detecting that the video data is inside, a video data buffer memory 14 which is a buffer means for buffering the separated video data (moving image data), and signal acquisition When it is detected that the difference between the time and the reproduction time exceeds the first time (V> Vmax) or falls below the second time (V <Vmin), the signal acquisition time and the reproduction time are detected. In order to make the difference between the first time and the second time from the above, a control signal for selecting the image quality or controlling the number of frames to be transmitted within a unit time is sent to the transmitting side 90. Control signal to send Arithmetic determination circuit 81 serving as transmission means
And a network transmission unit 73.

That is, in the communication system of the present embodiment, when the transmission side 90 receives the control signal from the reception side 91, the frame side 68 changes the image quality selection of the moving picture or the number of dropped frames to change the reception side 91. It controls the amount of video data sent to.

In the communication system of this embodiment, audio data as well as moving image data is handled.

The communication system of the embodiment of the present invention can also receive signals other than audio and video signals. Therefore, the transmitting side 90 processes a data processing circuit that processes signals other than audio and video signals. In addition to providing 94, the receiving side 91 is provided with a detection circuit 25 as a third detection means for detecting that the received signal is a signal excluding audio and moving image signals.
Is further provided. The signals other than the audio and moving image signals may include, for example, text data, program data, numerical data, and other binary data. When the detection circuit 25 detects that the received signal is a signal other than a moving image signal, the reception side 91 causes the separation circuit 12 to detect only the text data and the like according to the detection output from the detection circuit 25. The data processing circuit 26 processes the separated text data and the like as they are.

Further, the detection circuit 25 on the reception side 91 of the communication system according to the embodiment of the present invention detects, from the received signal, a predetermined portion such as an important portion of video or audio as a fourth portion.
The detection output from the detection circuit 25 as the fourth detection means is used as the detection means
Send to 1. In the calculation judgment circuit 81, the audio data buffer memory 13 and the video data buffer memory 1
For 4, the read control is performed such that only the predetermined portion is processed and the other portion is discarded.

Here, first, prior to a detailed description of the communication system of the embodiment of the present invention shown in FIGS. 1 and 9 to 12,
An information processing system to which the communication system of the present invention is applied. Basic functions of an information processing apparatus compatible with multimedia data, a model of the information processing apparatus, and a so-called application programming interface (API: appli).
Cation programming interface), the positioning of the multimedia-compatible information processing device in the information processing communication system, and the evaluation of the information processing will be described in the following items.

1. Functions of multimedia-compatible information processing device in information processing communication system The goals of this information processing device are (1) There are various restrictions, but they are built on a standard personal computer or workstation and are compatible with existing systems. To maintain compatibility, (2) leave audio and video communication and specific processing to the information processing device, and the client program only controls it. (3) Media data attributes and purpose of client program Flexibly according to the following, (4) Capable of responding to changes in the load of resources such as processors and networks, (5)
The model is simple, and so on.

From these requirements, the information processing apparatus needs to have the following functions.

1.1 Media Type Media data handled by the information processing device basically includes audio and moving images. These are given temporal attributes such as the amount of data per unit time and the generation time, and these are used for processing in the information processing device. For example, in the case of voice data having a data width of 8 bits and a sampling frequency of 8 kHz, the unit time is 1/8000 (sec) and the data amount per unit time is 1 byte.

1.2 Information Processing Device and Media Device The information processing device receives data from a plurality of input devices,
Output to the output device while synchronizing in time.

The input device is (1) a hardware device such as an audio interface or a video input interface, (2) a multimedia data file such as a sound file, a moving image file or a movie file, (3) a multicast address, (4) ) Support client process etc.

Output devices include (1) devices such as audio interfaces and windows, (2) multimedia files, (3) multicast addresses, (4)
Supports client processes, etc.

There is only one information processing device in a certain host, and the device that it directly handles must be on that host. If you need the input and output devices to be on different hosts,
The client accesses each information processing device on each host and connects the information processing devices.

1.3 Transfer and Synchronization of Media Data The following can be considered as the criteria for evaluating the quality of objects provided to users as a multimedia system. For example,
(1) Allowable limit of transfer delay, (2) Allowable limit of inter-media synchronization, (3) Throughput, (4) When data loss is allowed and when it is not allowed.

In order to obtain a satisfactory quality on a given transfer path, various parameters at the time of transfer such as data amount, compression method, protocol and packet size are controlled. The information processing device performs inter-media synchronization according to the quality evaluation standard.

1.4 Network Protocol In the network protocol of multimedia data used by the information processing apparatus, it is necessary to be able to dynamically control the flow and the like while considering quality evaluation criteria. Fiber of the so-called Ethernet (Ethernet) compliant with the IEEE 802.3 standard of the current network environment and 100 Mbit / sec token passing system using optical fiber proposed in SC13 of International Standards Organization (ISO) Distributed
Data interface (FDDI: fiber-distributed
Since network devices such as data interface) share and share resources individually in a distributed environment, it is difficult to provide services that secure network resources in advance. However, considering the compatibility with the current network environment, the Internet Protocol (IP: internet
It is necessary to use the so-called transmission control protocol (TCP) as a network protocol this time.
and a so-called user datagram protocol (UDP) which is a connectionless protocol.

1.5 Data Compression / Expansion The information processing apparatus has a software / hardware compression / expansion function for audio data and moving image data. As the voice compression method, the International Governor of Telegraph and Telephone Communications Advisory Committee (CCITT) Recommendation G. 711,
G. 721, G.I. 722, G.I. Supports standards such as 728. The image compression method is the so-called JPEG (J) of the international standardization work group of the color still image coding method of the International Telegraph and Telephone Communication Advisory Committee (CCITT).
oint Picture Expert Group) and video coding recommendations for video conferencing systems. 261, MPEG (Moving Picture E) of the International Standardization Working Group on color video coding
xpert Group). Since each compression method has its own characteristics, it is necessary to use them properly depending on the application.

1.6 Resource Control The information processing apparatus has a function of processing input / output requests from a plurality of client programs. For example, when there are voice input requests from a plurality of clients even though there is only one voice input device, the following processing method can be considered. (1) Copy and distribute to all requesters. (2) Prioritize first-come-first-served requests and reject second-come-first-served requests. (3) Switch clients sequentially. This function is used by programs such as a multimedia manager, which corresponds to the window manager in the window system.

1.7 Control of Physical Device When handling multimedia data such as voices and moving images, it is necessary to connect so-called audio-visual equipment (AV equipment) such as a video camera and a VCR. The information processing apparatus supports a predetermined protocol for controlling AV equipment for these controls.

2. Model of Information Processing Device and API 2.1 Model of Information Processing Device to which AV Equipment is Connected As shown in FIG. 2, the information processing device has one execution control unit (AVobj) for each client. To generate. When the client wants to input / output multimedia data, it makes a request to the information processing device by the following procedure. First, a virtual media device (AVdev) is opened in the execution control unit (AVobj). Since the virtual media device (AVdev) is not a physical device, exclusive control and opening from multiple devices can be realized.

As shown in FIG. 3, the execution control unit (AVobj) that opens the virtual media device (AVdev) for input and the execution control unit (AVobj) that opens the media device (AVdev) for output. ) Is connected to secure a transfer path for multimedia data. The synchronization of media between devices opened in the same execution control unit (AVobj) is guaranteed.

Further, as shown in FIG. 4, an information processing device on different hosts connected by a network generates an execution control unit (AVobj), and by connecting it, multimedia data is transferred in a workstation in a distributed environment. Be seen.

A telephone can be realized by using a sound device as a device (AVdev) opened by the execution control unit (AVobj) on the transmitting and receiving sides. In addition, opening a video device enables a videophone call. When a movie file is specified as an input device and a sound device and a video device are opened as output devices, a movie player is created. As described above, by changing the input / output devices, various multimedia applications can be easily created.

2.2 API of Information Processing Device The following are prepared in the library of the information processing device, for example.

Int avs_new (char * hostname);
An execution control unit (AVobj) is generated in the information processing device activated on the host name (hostname). If an error occurs, null is returned. In the case of normal termination, the ID (identification information) of the execution control unit (AVobj) is returned. All instructions for the execution control unit (AVobj) are executed using the ID of this execution control unit (AVobj).

Int avs_open (int net, char * devname,
int mode); This opens the device (AVdev). The arguments are the ID of the execution control unit (AVobj), the device name,
Mode.

Int avs _connect (int net1, int net2);
It connects two execution control units (AVobj) point-to-point. When one of the execution control units (AVobj) connected by this becomes the transfer state by the function (avs_transfer) described below, the other execution control unit (AVobj) can receive the data and automatically receives it. Process the data. Since one reception execution control unit (AVobj) can be connected to one transmission execution control unit (AVobj), one-to-many data transfer can be processed.

Int avs_transfer (int net, int dev, i
nt length); this controls the transmission of the execution control unit (AVobj). If 0 is specified as the device ID, the execution control unit (AVobj) is valid for all the opened devices. If a positive number is specified for the length, data is transferred for that length (unit is msec). If you specify zero here, transfer is performed until the next function (av_transfer) is given. Specify a negative number to stop immediately.

Int avs_destroy (int net); This releases the execution control unit (AVobj) generated by the function (av_new).

Int avs _interval (int net, int dev, i
nt interval); This sets the transfer interval. The unit is msec. When zero is set as the device ID, it is applied to all the devices opened in the execution control unit (AVobj).

Int avs _resize (int net, int dev, int
width, int height); This requires the video device to be resized. The unit of size is pixel. When zero is set as the device ID, it is applied to all video devices opened in the execution control unit (AVobj).

Int avs _nettype (int net, char * typ
e); This sets the type of network that connects the execution control units (AVobj). Currently TCP
And UDP is supported. This is a function (avs_conn
ect) must be done before executing.

Int avs_fd (int net); This returns the file descriptor of the control connection connection between the client and the execution control unit (AVobj).

Int avs_codec (int net, char * type, i
nt quality); This specifies the compression method of the media data. Here, only the JPEG compression / expansion is supported for image data.

In addition, there are the following libraries for directly accessing the media data to the device (AVdev). For example, int avs _read (int net, int dev, int shmid, int si
ze); int avs _write (int net, int dev, int shmid, int s
ize); int avs _ioctl (int net, int dev, int request, int
shmi);

3. Mounting of Information Processing Device Positioning of the information processing device in the entire system is as shown in FIG. 3.1 Use of Threads Using threads makes the context switch time shorter than that achieved by using multiple processes, the environment such as memory can be shared between threads, and programming becomes easy. The following two methods are conceivable as methods of assigning threads to models for realizing an information processing device.

For example, each function is assigned to a thread,
There is a thread allocation for each data stream.
The former is a method of allocating threads for each function of screen input / output, audio input / output, network, etc. to form a pipeline, and the latter is a method of allocating threads for processing to perform input to output for each media data.

Even if a pipeline is formed for each function, there is no advantage unless in a multiprocessor environment, and the latter is suitable for performing schedule priority control in stream units in a plurality of streams. The method of allocating threads for each data stream was also used.

3.2 Transmission / Reception Buffer and Delay of TCP / IP In order to realize high reliability in the TCP / IP, packet ordering, checksum, timeout and retransfer are performed, resulting in a large overhead and a data transfer delay. It becomes a problem.

Therefore, a buffer is prepared here for transmission and reception. For example, in a workstation, the default is 8K (bytes).
The accumulation of data in this buffer causes a delay. For example, in the case of a screen having a resolution of 160 × 120 and a depth of 16 bits, one screen has approximately 38 Kbytes, and even if the buffers on both the transmitting side and the receiving side are combined, no frame is buffered. In this case, increasing the buffer size improves the transfer efficiency. However, when the image is compressed to a size of about 1/10, ie, about 4 Kbytes, about 4 frames are accumulated in both buffers. If transfer is performed at a speed of 5 frames per second, this alone results in a delay of about 1 second.

However, conversely, if the buffer is made too small, the transfer efficiency will deteriorate, and there is a trade-off here. The size of this send / receive buffer is set by the function (setsockop
It can be changed by t).

4. Evaluation of information processing device The evaluation when the information processing device is mounted on a workstation is as follows.

4.1 Transmission / Reception Buffer and Transfer Delay, Efficiency As an example of actually measuring the size of the transmission / reception buffer, transfer delay, and efficiency, transfer of moving image data between two workstations connected to the same network Was done. Here, the workstation is connected by Ethernet, the resolution is 160 × 120, 16-bit depth uncompressed and JPEG compressed images are transferred and measured. One frame size is uncompressed, approximately 38 Kbytes, JPEG compressed. It is about 5K bytes.

The transfer delay was measured by adjusting the clock using a function (timed (8)) and measuring the time from image capture to transfer and display on the remote side. Furthermore, as the transfer efficiency, the maximum transfer frame rate is measured. Figure 6
Shows the relationship between the transmission / reception buffer size of an image that has not been compressed, and FIG.

As shown in FIG. 6, when there is no compression and the size of one frame is large, the difference in transfer delay with respect to the size of the buffer remains almost unchanged because one frame does not fill the transmit / receive buffer. When the buffer size is 8 Kbytes, good results are obtained in terms of transfer delay and efficiency.
According to this FIG. 6, it can be understood that the transfer efficiency by TCP / IP is the most efficient in that case also from the fact that the default transmission / reception buffer size is 8 Kbytes.

On the other hand, when transferring the JPEG compressed image shown in FIG.
The maximum frame rate is small because G is compressed and expanded.
Moreover, since several frames are accumulated in the transmission / reception buffer, the delay increases as the size of the buffer increases. Therefore, when image data is sufficiently smaller than the bandwidth of the network by performing compression such as JPEG, the transfer delay of media data can be shortened by reducing the transmission / reception buffer size so that transfer efficiency does not decrease.

4.2 Screen Transfer Speed The screen size, frame rate and transfer delay are measured in the same environment. First, when Ethernet was used as the network, the results in the following table were obtained. Table 1
Is a 16-bit deep uncompressed image, Table 2 is JPEG
The maximum transfer frame rate and delay performance of compressed images are shown.

[0069]

[Table 1]

[0070]

[Table 2]

Next, an integrated service digital network (ISDN) is used as a network.
Table 3 shows the performance when using 2B) of 1B (64K). In addition, when ISDM is used, the frame rate becomes 0.2 (fps) even if the image data is not compressed and the minimum screen size (80 × 60) is not practical. Evaluation of image transfer performance was omitted.

[0072]

[Table 3]

By using such an information processing apparatus, a multimedia application can be easily created without being conscious of capturing and compressing and expanding images and sounds, and network programming.

4.4 Realization of synchronization between media In this information processing apparatus, media data is synchronized and interleaved and transferred into a single stream. Further, since TCP / IP is used as the network protocol, the packet order is guaranteed. Furthermore, if a plurality of pieces of media data are captured at the same time and interleaved to guarantee synchronization during data transmission, it can be assumed that the plurality of pieces of media data are also synchronized on the data receiving side. In fact, the synchronization of audio and video media data transferred over a single network in this way has been satisfactory.

4.5 Relationship with Delay In the above-mentioned implementation, TCP / IP which is standard in the operating system UNIX is used as the network protocol.
Is used. Here, as shown in FIG. 8, in an environment in which media data can be directly sent, the information processing apparatus can control the buffers on the transmitting side and the receiving side, and fine flow rate control can be performed.

Now, returning to FIG. 1, a communication system according to an embodiment of the present invention which specifically realizes the above will be described.

In FIG. 1, audio signals and video signals (moving image signals) from the microphone 1 and the video camera 2 are sent to the corresponding A / D conversion circuits 65 and 66, respectively.
/ D converted. The audio data from the A / D conversion circuit 65 is sent to the audio data compression circuit 67, and as described above, as the audio compression method, for example, G.264. 711, G.I. 7
21, G.I. 722, G.I. The compression is performed using any one of 728 and the like. Further, the A / D conversion circuit 66
The video data from the video data is sent to a video data compression circuit 69 via a frame buffer 68, which will be described later. 26
1 or MPEG is used for compression.

The compressed audio data and video data from the compression circuits 67 and 69 are multiplexed by the multiplexer 5, and then sent to the network transmission section 7 via the data buffer memory (eg FIFO memory) 6 and concerned. The data is sent from the network transmission unit 7 to the network reception unit 11 of the reception side 91 via the network 8 such as Ethernet.

Further, in the communication system of FIG. 1, the input terminal 89 is supplied with, for example, text data, program data, numerical data, other binary data, etc. as signals excluding the above-mentioned voice and moving images. These data are subjected to processing such as addition of an error correction code in the data processing circuit 94, and then output from the network transmission unit 7 via the multiplexer 5.

Network receiving unit 11 of the receiving side 91
The data received in 1 is separated into the compressed audio data and video data by the separation circuit 12, and the corresponding audio data buffer memory (FIFO memory) 1
3 and video data buffer memory (FIFO memory) 14
Sent to.

Further, the buffer memory internal capacity measuring circuit 20
Measures the internal capacity of the memory 14 from the write address and the read address of the video data buffer memory 14, and sends the measurement result (measured value V described later) to the arithmetic judgment circuit 81. When the operation judging circuit 81 detects from the measurement result of the internal capacity of the memory 14 from the measuring circuit 20 that the internal capacity of the video data buffer memory 14 exceeds the upper limit (Vmax described later), the fact is detected. Is transmitted to the transmission side 91 via the network transmission unit 73 as a control signal.

Further, in the arithmetic decision circuit 81, the internal capacity of the video data buffer memory 14 becomes the lower limit (Vmin described later) based on the measured result of the internal capacity of the memory 14 (measured value V described later) from the measuring circuit 20. If the drop is detected, the image may be interrupted. Therefore, a signal indicating that (lower limit has been dropped) is sent as a control signal to the transmitting side 90 via the network transmitting unit 73.

Here, in the buffer memory internal capacity measuring circuit 20 and the operation judging circuit 81, the internal capacity of the video data buffer memory 14 is measured and the control signal is generated according to the flowchart shown in FIG. I have to.

In FIG. 9, in step S11,
Measured value V of the internal capacity of the buffer memory 14 for video data
(Unit is playback time). The measured value V is calculated by the following formula. V = (buffer read address)-(buffer write address)

In the next step S2, V> Vmax is determined from the measured value V and the upper limit value Vmax of the appropriate range of the internal capacity of the buffer memory 14 for the video data. If NO is determined, the process proceeds to step S32. On the contrary, if the answer is yes, the process proceeds to step S35.

In step S35, a signal indicating that V> Vmax is sent to the transmitting side 90 as a control signal, and then the process returns to step S11.

If it is determined NO in step S12 and the process proceeds to step S32, in this step S32, the measured value V and the lower limit value Vmin of the proper range of the internal capacity of the buffer memory 14 for video data are set to V < Vmin
To judge. In step S32, if the result is NO, the process returns to step S11. If the result is YES, the process proceeds to step S36.

In step S36, a signal indicating that V <Vmin is sent to the transmitting side 90 as a control signal, and then the process returns to step S11.

When the proper value of the proper range of the internal capacity of the buffer memory 14 for the video data is Vsta, the relationship between the upper limit value Vmax, the proper value Vsta and the lower limit value Vmin is Vmin <Vsta <Vmax.

The control signal from the receiving side 91 is received by the network receiving section 72 of the transmitting side 90 and sent to the fetch period designating circuit 71. The capture period designating circuit 71 controls the frame capture clock generating circuit 70 based on the control signal. When the frame capture clock generation circuit 70 receives the signal from the capture cycle designating circuit 71, the frame capture clock generation circuit 70 generates a frame capture clock for varying the number of dropped frames in the frame buffer 68. Here, for example, the amount of video data to be transmitted can be reduced by performing frame dropping (or increasing the number of frame dropping), and the image data transmitted by not performing frame dropping or reducing the number of frame dropping can be transmitted. The amount of can be increased.

That is, the transmitting side 90 performs the processing of the flow chart as shown in FIG. 10 by the control signal from the receiving side 91.

In FIG. 10, a control signal from the receiving side 91 is received in step S41.

In the next step S42, it is judged whether or not the received control signal is a signal representing V> Vmax, and V>V>.
If it is determined that the signal represents Vmax, the process proceeds to step S43, and if it is determined to be no, step S44.
Proceed to.

In step S43, the fetch period designating circuit 71 causes the frame fetch clock generating circuit 70 to operate.
On the other hand, control is performed so that the period of the frame capture clock in the frame buffer 68 is lengthened by P. After step S43, the process returns to step S41.
The value of P is, for example, 1%.

On the other hand, if it is determined NO in step S42, it is determined in step S44 whether or not the received control signal is a signal representing V <Vmin, and it is determined that the signal is V <Vmin. If yes, go to step S45; if no, go to step S45.
Return to 41.

In step S44, the fetch period designating circuit 71 causes the frame fetch clock generating circuit 70 to operate.
On the other hand, the cycle of the frame capture clock in the frame buffer 68 is controlled to be shortened by P. After step S44, the process returns to step S41.
The value of P is, for example, 1%.

As a result, in the communication system of the present embodiment, the internal capacity of the video data buffer memory 14 on the receiving side 91 becomes large, and the video data buffer memory 14 concerned.
Even if the time delay from the input to the output and the time shift at the time exceeds the proper value, the time feed and the time shift can be restored to the proper values, and the video data buffer memory 14 Even if there is a possibility that the video content is interrupted due to a decrease in the content amount, the video interruption can be prevented.

On the receiving side 91, the audio data and the video data read from the audio data buffer memory 13 and the video data buffer memory 14 are subjected to corresponding decompression processing and D / A conversion, respectively. It is sent to the circuits 18 and 19. These expansion D / A conversion circuit 1
In Nos. 8 and 19, decompression processing corresponding to each compression processing in the A / D conversion compression circuits 3 and 4 is performed, and then D
/ A convert and output.

The audio signal from the expansion D / A conversion circuit 18 is sent to the speaker 27, and the expansion D / A conversion circuit 1 is transmitted.
The video signal from 9 is sent to the monitor display 24.

Further, in the configuration of FIG. 1 described above, by varying the number of dropped frames of the video on the transmitting side 90, the time delay from the input to the output in the video data buffer memory 14 and the time shift are set to appropriate values. Although it is designed to recover or prevent interruption of the image, for example, the transmitting side 90
The same thing can be done by selecting the image quality of the image to be sent to the receiving side 91 from.

That is, when selecting this image quality,
For example, the amount of data to be transmitted is reduced by thinning out each pixel data captured in the frame buffer 68,
On the contrary, the image quality is ensured by not performing thinning. In this case, on the receiving side 91, when the pixel data is thinned out, for example, processing such as interpolation is performed to restore the image.

Further, in the communication system of FIG.
The reception signal supplied to the separation circuit 12 on the reception side 91 is also sent to the detection circuit 25 via the separation circuit 12. The detection circuit 25 detects a signal such as text data, program data, numerical data, or other binary data as a signal excluding the audio and video signals from the received signal.

When the detection circuit 25 detects data excluding the audio and moving image signals, the detection circuit 25 outputs a detection signal to the separation circuit 12. When the detection signal from the detection circuit 25 is supplied, the separation circuit 12 separates only the data corresponding to the detection signal from the reception signal and sends it to the data processing circuit 26.

In the data processing circuit 26, the supplied data is subjected to predetermined processing and then output from the output terminal. It should be noted that examples of the predetermined processing include error correction processing and the like.

As described above, the receiving side 91 of the present embodiment has the detection circuit 25, and the separation circuit 12 removes the sound and the moving image from the reception signal according to the detection signal from the detection circuit 25. By separating the data, it is possible to perform processing according to the data excluding the audio and moving images.

Further, the separation circuit 1 on the receiving side 91 of the present embodiment.
The audio and video data separated in 2 are also sent to the detection circuit 25.

At this time, the detection circuit 25 detects the low-frequency part of the voice data in consideration of, for example, the human auditory characteristics, and is particularly important such as the human voice part. Detects various voice parts. Also,
For video data, for example, in consideration of human visual characteristics, a particularly visually important video portion is detected from the video data, or a particularly important portion in the video content is detected.
From the detection circuit 25, detection signals corresponding to the video data and the audio data are sent to the calculation determination circuit 81.

When the arithmetic decision circuit 81 receives a detection signal corresponding to the audio data from the detection circuit 25,
The audio data buffer memory 13 is controlled so that the audio portion corresponding to the detection signal is not discarded. That is, the control is performed so that the important part of the sound corresponding to the detection signal is sent from the sound data buffer memory 13 to the sound data decompression D / A conversion circuit 18 in the subsequent stage as it is. On the other hand, the audio data other than these important parts is discarded when the internal capacity of the buffer memory 13 exceeds, for example, an appropriate value, so that the internal capacity of the buffer memory 13 is optimized.

Also by this, the receiving side 91 of the communication system of this embodiment can recover the time delay from the input to the output of the buffer memory 13 to an appropriate value,
Further, since the important part of the voice is not discarded, the voice output from the speaker 27 after that can maintain high quality.

On the other hand, when the arithmetic decision circuit 81 receives the detection signal corresponding to the video data from the detection circuit 25, the arithmetic decision circuit 81 controls the video data buffer memory 14 so that the video portion corresponding to the detection signal is not discarded. I do. That is, the important part of the video corresponding to the detection signal is controlled so as to be sent from the video data buffer memory 14 to the post-stage video data decompression D / A conversion circuit 19 as it is. Further, the video data other than these important parts is discarded when the internal capacity of the buffer memory 14 exceeds an appropriate value, so that the internal capacity of the buffer memory 14 is optimized.

Also by this, the receiving side 91 of the communication system of the present embodiment can recover the time delay from the input to the output of the buffer memory 14 to an appropriate value,
In addition, since the important part of the video is not discarded, the video displayed on the monitor display 24 later is
You will be able to maintain high quality.

Furthermore, in the communication system of the present invention,
As a specific example of selecting the image quality of the image to be sent from the transmitting side 90 to the receiving side 91, for example, as shown in FIG. 11, the compression rate in the video data compression circuit 69 may be varied. In addition, in FIG. 11, in FIG.
Constituent elements similar to those described above are designated by the same reference numerals, and detailed description thereof will be omitted.

That is, in FIG. 11, the transmitter 90
Then, the control signal from the receiving unit 91 is received by the compression parameter designating circuit 85 which designates a compression parameter.
The compression parameter designation circuit 85 varies the compression rate in the video data compression circuit 69 according to the control signal. For example, by increasing the compression rate, the amount of video data sent to the receiving side 91 can be reduced, and by decreasing the compression rate, the amount of video data sent to the receiving side 91 can be increased. In other words, increasing the compression rate reduces the image quality, and decreasing the compression rate enhances the image quality. When the compression rate of the video data is changed as shown in FIG. 11, it is necessary to send the compression parameter to the receiving side 91, and the expansion D /
The A conversion circuit 19 expands the compressed video data based on this compression parameter.

Further, in the configuration of FIG. 11, if the variable compression parameters are similarly sent to the audio data compression circuit 67 for compressing the audio data, the internal capacity of the audio data buffer memory 13 of the receiving side 91 is increased. Can be controlled.

Next, the configuration of another embodiment of the present invention is shown in FIG.
Shown in.

In FIG. 12, the monitor computer 40 and the microphone 4 are provided on the transmitting side host computer.
1 and the video camera 42 are connected, and the network 4
3 is connected to the receiving side host computer.

In the transmitting host computer,
The CPU 30 controls each unit by using the program data stored in the main memory 31, and has a function as a variable unit for changing the image quality selection of the moving image or the number of dropped frames and a function as a control unit for controlling the variable unit. There is.
That is, the CPU 30 has the fetch period designating circuit 71 and the frame fetch clock generating circuit 7 shown in FIG.
0 and the same function as the data processing circuit 94.

The video signal from the video camera 42 is A /
The A / D conversion is performed by the D conversion compression processing unit 38, the compression processing for the video data as described above is performed, and the data is temporarily stored in the buffer memory 34. Further, the buffer memory 34 also operates as a capturing means for capturing the moving image signal. In the embodiment of FIG. 1, the frame buffer 68 as the capturing means captures the video data before compression, but in the example of the buffer memory 34 shown in FIG. 12, the video data after compression is captured. I am trying to capture.

On the other hand, the audio signal from the microphone 41 is A / D converted by the A / D conversion compression processing section 37, subjected to the audio data compression processing as described above, and temporarily stored in the buffer memory 33. .

In the frame memory 32, for example, C
The video frame data formed by the PU 30 and the video frame data based on the video imaged by the video camera 42 are stored. The video frame data from the frame memory 32 is sent to the monitor display 40 via the display interface 36 and displayed.

Further, the compressed audio data and video data stored in the buffer memories 33 and 34 are multiplexed by the CPU 30, for example, and then temporarily stored in the buffer memory 35 and then read. The data read from the buffer memory 35 is used by the network 4 connected to the network interface 39.
3 to the receiving host computer. The network interface 39 has the same functions as the network transmitter 7 and the network receiver 72 shown in FIG.

Next, a monitor display 61 and a speaker 62 are connected to the receiving side host computer, and further connected to the transmitting side host computer via the network 43.

In the receiving host computer,
The CPU 52 uses the program data held in the main memory 53 to control each unit and perform various calculations. In addition, the CPU 52 has a circuit 20 for measuring the internal capacity of the buffer memory and an operation judging circuit 81 shown in FIG.
It also has a function as a detection circuit 25 and a data processing circuit 26.

The data sent from the sending side host computer via the network 43 is transferred to the buffer memory 51 via the network interface 50.
It is stored once in and then read. The network interface 50 has the same functions as the network receiving unit 11 and the network transmitting unit 73 shown in FIG.

From the received data read from the buffer memory 51, for example, the compressed audio data and video data are separated by the CPU 52 and sent to the corresponding buffer memories 54 and 55, respectively.

The compressed video data sent to the buffer memory 54 is read out from the buffer memory 54 and sent to the data expansion processing section 56, where it is expanded to correspond to the compression in the transmitting side host computer. It The decompressed video data is stored in the frame memory to form a frame, and the display interface 58
It is sent to and displayed on the monitor display 61 via.

Further, the compressed audio data sent to the buffer memory 55 is read from the memory 55 and sent to the data expansion processing unit 59, where the expansion processing corresponding to the compression in the transmission side host computer is performed. Done. The decompressed audio data is converted to an analog audio signal by the D / A conversion processing unit 60 and then sent to the speaker 62.

As described above, according to each of the embodiments of the present invention, on the receiving side, the measured value V, which is the difference between the capturing time of the moving image signal sent from the transmitting side and the reproducing time, is the upper limit ( When it is detected that the value exceeds Vmax) or falls below the lower limit (Vmiv), a control signal for controlling the image quality or the number of transmission frames per unit time is transmitted so as to be between the upper limit and the lower limit. The receiver side receives this control signal and controls the image quality or the number of frames to be transmitted, so that the time delay from the input to the output and the time shift can be restored to an appropriate value, and there is also a break in the image. It is possible to prevent it.

Further, in the communication system of the present embodiment, the receiving side 91 is provided with the data detecting circuit 25, and the data detecting circuit 25 confirms that the received signal is data excluding either the audio signal or the moving image signal. When detected, the data processing circuit 26 processes the data as it is, thereby preventing the content of the data from changing when the received signal is, for example, text data.

Further, in the communication system of the present embodiment, when the detection circuit 25 detects a particularly important part in the voice or video from the received signal, the arithmetic decision circuit 81 outputs the other part except the important part. By discarding the data buffer memory 13 and the video data buffer memory 14, the time delay from the input to the output of the buffer memories 13 and 14 is restored to an appropriate value, and the above particularly important parts are processed as they are. By doing so, the signal of this part is prevented from being discarded.

[0131]

As described above, in the communication system of the present invention, at the receiving side, the difference between the capturing time of the moving image signal and the reproducing time exceeds the first time and / or the second time. When it is detected that the value is below the range, a control signal for controlling the image quality or the number of frames to be transmitted within a unit time is sent to the transmitting side between the first time and the second time,
In response to this, the transmitting side performs control to select the image quality of the moving image from the moving image signal to be transmitted or change the number of dropped frames, so that the time delay from the input to the output of the buffer means on the receiving side is set to an appropriate value. It is possible to recover, and it is possible to prevent interruption of images.

In the communication system of the present invention,
A third detecting means is provided on the receiving side, and when the third detecting means detects that the received signal is a signal excluding the moving image signal, it is processed as it is so that the received signal is, for example, text data or program data. In some cases, the contents of the data can be prevented from changing.

Further, in the communication system of the present invention, a fourth detecting means is further provided on the receiving side, and when the fourth detecting means detects a predetermined portion of the moving image signal of the received signal, the predetermined portion thereof is detected. By discarding other parts except for, the time delay from the input to the output of the buffer means is restored to an appropriate value, and the predetermined part is processed as it is, and the signal of this predetermined part is It can be prevented from being discarded.

[Brief description of drawings]

FIG. 1 is a block circuit diagram showing a schematic configuration of a transmission side and a reception side of a communication system according to an embodiment of the present invention.

FIG. 2 is a diagram showing a model of an information processing apparatus of an information processing communication system to which the communication system of the present invention is applied.

FIG. 3 is a diagram for explaining transfer of multimedia data.

FIG. 4 is a diagram for explaining transfer of multimedia data between hosts connected by a network.

FIG. 5 is a diagram for explaining the positioning of the information processing device in the entire system.

FIG. 6 is a characteristic diagram showing a relationship between a transmission / reception buffer size, transfer delay, and efficiency in the case of no compression.

FIG. 7 is a characteristic diagram showing a relationship between a transmission / reception buffer size of an image in JPEG compression, transfer delay, and efficiency.

FIG. 8 is a diagram for explaining a relationship between an information processing device and delay.

FIG. 9 is a view for explaining the measurement of the internal capacity of the video data buffer memory on the receiving side of the communication system of this embodiment and the processing of outputting a control signal indicating that the measured value exceeds the upper limit or falls below the lower limit. It is a flowchart.

FIG. 10 is a flowchart for explaining processing of image quality selection or frame dropping control on the transmission side of the communication system according to the present embodiment.

FIG. 11 is a block circuit diagram showing a specific configuration for performing image quality selection on the transmission side of the communication system of the present embodiment.

FIG. 12 is a block circuit diagram showing a schematic configuration of a transmission-side host computer and a reception-side host computer according to another embodiment of the present invention.

FIG. 13 is a block circuit diagram showing a schematic configuration of a conventional communication system including a transmitting side and a receiving side.

[Explanation of symbols]

1, 41 ... Microphone 2, 42 ... Video camera 5 ... Multiplexer 6 ... Data buffer memory 7, 73 ... Network transmitter 8 ... Network 9, 72 ... Network receiver 12 ... Separation circuit 13 ... Audio data buffer memory 14 ... Video data buffer memory 20 ... Buffer memory capacity measuring circuit 18 ... Audio data expansion D / A conversion circuit 19 ... Video Data expansion D / A conversion circuit 24, 40, 61 ... Monitor display 25 ... Detection circuit 26 ... Receiving side data processing circuit 27, 62 ... Speaker 30 ... CPU 31, 53 ... ..Main memory 32, 57 ... Frame memory 33, 34, 35, 51, 54, 55 ... Buffer memory 36, 58 ... Data Spray interface 37 ... Audio data A / D conversion compression unit 38 ... Video data A / D conversion compression unit 56 ... Audio data data expansion unit 59 ... Video data data expansion unit 60 ... Audio data D / A conversion unit 65, 66 ... A / D conversion circuit 67 ... Audio data compression circuit 68 ... Frame buffer 69 ... Video data compression circuit 70 ... Frame capture clock generation circuit 71・ ・ ・ Capturing cycle designation circuit 81 ・ ・ ・ Calculation determination circuit 90 ・ ・ ・ Sending side 91 ・ ・ ・ Reception side 94 ・ ・ ・ Data processing circuit on the transmission side

Continuation of front page (51) Int.Cl. ⁶ Identification number Office reference number FI Technical indication location H04M 11/06 8324-5K H04N 1/41 Z (72) Inventor Mitsuru Tanabe 6-7 Kita-Shinagawa, Shinagawa-ku, Tokyo No. 35 Sony Corporation

Claims (3)

[Claims] 1. In a real-time communication system in which a difference between a time for capturing a moving image signal on a transmitting side and a time for reproducing on a receiving side is within a predetermined time, the transmitting side includes a capturing means for capturing a moving image signal, and a moving image. Of the receiving side. The receiving side is provided with a changing means for selecting the image quality of the above or changing the number of dropped frames, a controlling means for controlling the changing means, and a control signal inputting means for inputting a control signal from the receiving side to the controlling means. Is a capturing means for capturing the transmitted moving image signal, a reproducing means for reproducing the moving image signal, and detecting that the difference between the capturing time of the moving image signal and the reproducing time is within a first time as an upper limit. The detection means of No. 1 and the difference between the capturing time of the moving image signal and the reproducing time are the second lower limit.
Second detection means for detecting that the time is within the time, buffer means for buffering the signal, and the difference between the signal acquisition time and the reproduction time exceeds the first time and / or When it is detected that the time falls below the second time, the transmission side is configured to make a difference between the signal acquisition time and the reproduction time between the first time and the second time. On the other hand, a communication system comprising a control signal transmitting means for transmitting a control signal for controlling the image quality or the number of frames to be transmitted within a unit time. 2. The receiving side is provided with a third detecting means for detecting that the received signal is a signal excluding the moving image signal, and the receiving side also receives the signal excluding the moving image signal,
2. The communication system according to claim 1, wherein when the detection means detects that the received signal is a signal excluding the moving image signal, the signal excluding the moving image signal is processed as it is. 3. The receiving side is provided with a fourth detecting means for detecting a predetermined portion of the moving image signal, and the receiving side detects the predetermined portion of the moving image signal by the fourth detecting means. The communication system according to claim 1 or 2, wherein only the predetermined portion is processed as it is and the other portion is discarded.