JPH10336644A - Image/sound synchronization processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image/sound synchronization processor capable of allocating the time of an image processing and a sound processing so as to reproduce sound without interruption and performing the control of switching it at a high speed without changing the compression/expansion program of images/sound. SOLUTION: Sound signals are compression-encoded by a sound compression means 9, a compression control means 7 calculates the time allocated to the compression of image signals from the time required for the compression of the sound signals, and when the time is the sufficient time capable of compressing the images, an image compression means 8 is called and the image signals obtained from a device control means 2 are compression-encoded. Image data and sound data compression-encoded in such a manner are respectively inputted to a code synthesis means 10 through the compression control means 7 and the device control means 2, synthesized into one and transmitted through a transmission means 11 and a modem 12 through the device control means 2. In such a manner, by preferentially compressing the sound between the sound and the images, the interruption of the sound is prevented.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image / audio synchronous processing apparatus, and more particularly to an image / audio synchronous processing apparatus for synchronizing images / audio and compressing / expanding them in real time.

[0002]

2. Description of the Related Art When transmitting and receiving images and voices using a telephone line or a digital line like a videophone, the data is compressed and transmitted according to the transmission capacity of the line, and the data is expanded on the receiving side. The method is common. At this time, the image is cut out of one piece of data from continuous data and compressed in units of one frame, and the audio is compressed using data divided in unit time as one sample. By repeating such compression of the image and the audio, data of the audio that is continuous with the compressed moving image is created and transmitted to the other party through the transmission path. The data received on the receiving side is expanded for each minimum unit.

A conventional image / sound synchronization processing apparatus for synchronizing image / sound transmission in real time, transmitting it, and decompressing it on the receiving side is configured as shown in FIGS. 15 and 16, for example. FIG. 15 is a block diagram showing an example of a compression device in a conventional video / audio synchronous processing device. In the figure, a device control unit 42 is executed by a CPU 41, and an image signal and an audio signal are input to an image input unit 43 and an audio input unit 45 from a camera 44 and a microphone 46, respectively.

[0004] The image signal input to the image input means 43 is input to the image compression means 47 via the device control means 42, is converted into a compression code, and is returned to the device control means 42. Similarly, the audio signal input to the audio input unit 45 is returned to the device control unit 42 after being converted into a compression code by the audio compression unit 48 via the device control unit 42.
The image data and the audio data obtained by the compression are converted into a data format for transmission by the code synthesizing unit 49 and transmitted by the transmitting unit 50 through the modem 51.

FIG. 16 is a block diagram showing an example of a decompression device in a conventional video / audio synchronous processing device. CPU 61
Starts the device control device 62, and the receiving device 70 receives the composite code transmitted from the compression device of FIG. The received composite code is input to the code separation unit 69 via the device control unit 62 and is separated into the compressed image code and audio code.

[0006] The separated image code and audio code are input to an image expansion unit 67 and an audio expansion unit 68 via the device control unit 62, respectively, and are converted into image data and audio data before compression. The output image data of the image expansion means 67 is input to the image output means 63 via the device control means 62, and further output to the monitor 64 for display. On the other hand, the output sound data of the sound expansion means 68 is input to the sound output means 65 via the device control means 62, and further output to the speaker 66 for sounding.

In order to perform the above-described compression / decompression processing while performing communication, the entire apparatus needs to operate in real time. However, in order to perform processing with a large amount of computation in real time, such as image compression / decompression, it is necessary to effectively use a large-sized register.
In addition, in order to synchronize and compress the image and the sound and compress and expand the image and the sound, it is necessary to operate the image and the sound in parallel, and switch between the image processing and the sound processing at regular time intervals to perform the processing alternately.

However, since the value of the register being used must be stored when the process is switched, a process for register storage is performed every time the process is switched. In particular, when switching occurs when a large-sized register is used, processing for saving is further required, which hinders real-time operation. In order to avoid this, it is necessary to switch after one of the image processing and the audio processing ends.

As a method of synchronizing an image and a sound, a method of transferring image and sound data to a buffer memory is known (Japanese Patent Laid-Open No. 7-64730). This is a method for always securing a certain amount or more of data in the buffer memory so that the reproduction data does not break. Also, a device that detects the delay time between the image expansion process and the audio expansion process and interpolates the audio data or continuously displays the video signal for the delay time to realize the synchronization between the video and the audio has been conventionally known. (Japanese Patent Laid-Open No. 7-75059). Further, there is also known a conventional moving picture / audio synchronization control apparatus in which a frame to be processed next is obtained from the extension processing time of the current frame, and synchronization control is performed by expanding audio accompanying the frame. (JP-A-7-110
756).

[0010]

In the above-described conventional image / sound synchronization processing apparatus, the sound is interrupted for synchronization, or the image expansion processing and the audio expansion processing operate in parallel. , Register saving processing at the time of switching occurs frequently, audio expansion processing can not be in time,
There is a problem that the sound is interrupted. Also, Japanese Patent Application Laid-Open
In the conventional device described in Japanese Patent Application Laid-Open No. -64730, only the data transfer method is described, but no description is given of the synchronous control of image and sound. Also, there is no description about switching between operations of image processing and audio processing.

In the invention described in Japanese Patent Application Laid-Open No. 7-75059, when image processing is delayed, the sound is interpolated (reproduced in slow) and synchronization is achieved. Being late, away from real-time processing.
No means for solving this is disclosed. Also,
It does not describe switching between the operations of image processing and audio processing. Further, in the invention described in Japanese Patent Application Laid-Open No. H7-110756, since the timing is set from the processing of the image, the image processing cannot be performed in time, and when the interval to the next processing frame becomes large, the sound is attached to the image. However, there is a problem that the sound is interrupted because the sound is reproduced. Further, there is no description on switching between image processing and audio processing operations.

The present invention has been made in view of the above points,
An image / audio synchronous processing device that allocates time for image processing and audio processing so that audio can be played back without interruption, and can control switching between them at high speed without changing the image / audio compression / expansion program. The purpose is to provide.

[0013]

In order to achieve the above object, the present invention provides an input means for inputting an image signal and an audio signal in real time, an image compression means for compressing an input image signal, and Audio compression means for compressing an audio signal; and audio compression such that one of the audio signals for one sample is preferentially compressed within a specified unit time among the image compression by the image compression means and the audio compression by the audio compression means. Control means, measure the time required for audio compression, and when the time obtained by subtracting the measured audio compression time from the specified unit time is longer than the time required for image compression by the image compression means, 1 Compression control means for compressing an image signal corresponding to the audio signal for the sample; output compression-encoded audio data of the audio compression means and output compression-encoded image data of the image compression means; Data and a is obtained by a configuration in which an output means respectively synthesized and output.

Here, the compression control means of the present invention, when the time obtained by subtracting the measured voice compression time from the specified unit time is shorter than the time required for image compression by the image compression means, performs the image compression by the image compression means. Instead, the audio compression means is made to perform audio compression for the next one sample.

In the present invention, since audio compression is performed with priority, audio signals can be compressed and output without interruption, and audio compression and image compression can be performed alternately. Further, in the present invention, when the time obtained by subtracting the measured audio compression time from the specified unit time is shorter than the time required for image compression by the image compression means, the audio compression processing is performed ahead of time, , The allocation time of the next sequence can be increased.

According to another aspect of the present invention, there is provided a receiving means for receiving data obtained by code-synthesizing compressed and encoded audio data and compressed and encoded image data; Separating means for separating the compressed and coded image data, image decompressing means for decompressing the compressed and coded image data, sound decompressing means for decompressing the compressed and coded audio data, and image decompression and sound decompression by the image decompression means Controlling the audio decompression means so as to give priority to the decompression of the encoded audio data for one sample within the designated unit time, and measuring the time required for the audio decompression, When the time obtained by subtracting the measured sound expansion time from the specified unit time is longer than the time required for image expansion by the image expansion means, the image expansion means A configuration having expansion control means for expanding compression-encoded image data corresponding to compression-encoded audio data, and synchronous output means for synchronously outputting output audio data of the audio expansion means and output image data of the image expansion means, respectively; It was done.

Here, when the time obtained by subtracting the measured voice expansion time from the designated unit time is shorter than the time required for the image expansion by the image expansion means, the expansion control means of the present invention performs the image expansion by the image expansion means. Instead, the sound expansion unit performs the sound expansion for the next one sample.

In the present invention, since the audio expansion is performed with priority, the audio signal can be expanded and output without interruption, and the audio expansion and the image expansion can be performed alternately. Further, in the present invention, when the time obtained by subtracting the measured audio expansion time from the specified unit time is shorter than the time required for image expansion by the image expansion unit, the audio expansion process is performed ahead of time, , The allocation time of the next sequence can be increased.

Furthermore, since the synchronous processing is realized by the timing and the number of times of calling the image and audio compression means and expansion means, the processing can be realized without changing the compression means and expansion means.

[0020]

Next, embodiments of the present invention will be described with reference to the drawings. FIG. 1 shows an image according to the present invention.
FIG. 2 is a block diagram of an embodiment of a compression device in the audio synchronization processing device. The compression device includes a central processing unit (CPU) 1 for operating a program, a device control unit 2 executed by the CPU 1 to control the operation of the entire device, a camera 4 for inputting image signals, and an image input unit. 3
A microphone 6 for inputting an audio signal, an audio input unit 5, an image compression unit 8 for compressing and encoding an image signal,
An audio compression means 9 for compressing and encoding an audio signal, a compression control means 7 for controlling operations of the image compression means 8 and the audio compression means 9, a code synthesizing means 10 for synthesizing compressed image data and audio data, It comprises a transmitting means 11 for transmitting the combined code and a modem 12.

In the compression apparatus shown in FIG. 1, an image signal input from the camera 4 is transmitted to the apparatus control means 2 by the image input means 3.
Passed to. At the same time, the audio signal input from the microphone 6 is passed to the device control means 2 by the audio input means 5. The compression control means 7 calls the audio compression means 9 and
The audio signal obtained from the device control means 2 is compression-encoded, and the time allocated to the compression of the image signal is calculated from the time required to compress the audio signal.

The compression control means 7 calls the image compression means 8 if the time allocated to the above-mentioned compression of the image signal is a time sufficient for performing image compression, and converts the image signal obtained from the apparatus control means 2. Perform compression encoding. The image data and the audio data thus compression-encoded are input to the code synthesizing unit 10 via the compression control unit 7 and the device control unit 2 and synthesized into one. The synthesized code data is converted into a predetermined signal form by the transmission means 11 via the device control means 2 and then modulated by the modem 12 before being transmitted.

FIG. 2 is a block diagram showing an embodiment of a decompression device in the image / audio synchronous processing device of the present invention. The decompression device shown in FIG.
1, a device control means 22 which is executed by the CPU 21 to control the operation of the entire apparatus, a modem 32 for receiving codes, a receiving means 31, and separates the received codes into image codes and audio codes. Code separation means 30, image expansion means 28 for expanding an image, audio expansion means 29 for expanding audio, expansion control means 27 for controlling the operations of image expansion means 28 and audio expansion means 29, and expanded image , An image output unit 23 for displaying an image, an audio output unit 25 for outputting an audio, and a speaker 26.

In the decompression device shown in FIG. 2, the composite code data input via the modem 32 and the receiving means 31 is input to the code separating means 30 via the device control means 22 and the image data compressed and coded. After being separated into audio data,
It is passed to the device control means 22.

The apparatus control means 22 inputs the compressed and encoded image data and audio data to the decompression control means 27. The expansion control unit 27 calls the audio expansion unit 29 and expands the input compressed and encoded audio data, and calculates the time allocated to the expansion of the image data from the time required for the expansion. The decompression control means 27 calls the image decompression means 28 if the time allocated for decompression of the compressed and coded image data is a sufficient time to perform decompression, and obtains the compressed and coded image obtained from the apparatus control means 22. Decompress and decode the data.

The image signal of the image signal and the audio signal thus expanded and decoded is input to the image output means 23 through the expansion control means 27 and the apparatus control means 22.
The audio signal is input to the audio output unit 25 via the expansion control unit 27 and the device control unit 22. Image output means 23
The output image signal is output to the monitor 24 and displayed as an image. The audio signal output from the audio output means 25 is output to the speaker 26, is subjected to electro-acoustic conversion, and is generated as sound.

Next, the operation of the embodiment of the present invention will be described.
This will be described with reference to the flowcharts of FIGS. FIG. 3 is a flowchart for explaining the operation of the compression apparatus in the apparatus of the present invention. In FIG. 3, when communication is started, compression and decompression capabilities of images and sounds are exchanged between the communicating terminals (step Sl). The compression device needs a function of receiving and analyzing decompression capability data from the decompression device, but is not shown in FIG. Similarly,
The decompression device also needs a function of transmitting data indicating its own decompression capability to the compression device, but is not shown in FIG.

In this capacity exchange, a frame rate of an image and a reference value of a processing time per frame are set which are optimal for each processing capacity (Step S2). Images and sounds are input based on the set reference values (step S
3) The compression of one sample of the sound and the accompanying image is performed (step S4). In the compression apparatus shown in FIG. 1, the audio signal is compression-encoded by the audio compression unit 9, and the compression control unit 7 calculates the time allocated to the compression of the image signal from the time required for the compression of the audio signal. If the time is long enough to perform the image compression, the image compression means 8 is called and the image signal obtained from the apparatus control means 2 is compression-encoded.

The image data and the audio data thus compressed and encoded are supplied to the compression control unit 7 and the device control unit 2.
After being input to the code synthesizing unit 10 via the device control unit 2 and synthesized into one, it is written into the transmission buffer in the transmission unit 11 via the device control unit 2, read out, and further modulated by the modem 12. (Step S
5). Step S3 to step S until transmission is completed
6 is repeated (step S6).

FIG. 4 is a flowchart for explaining the operation of an example of the compression control section. This compression control section comprises the compression control means 7, the image compression means 8 and the audio compression means 9 of FIG.
The compression operation will be described in more detail with reference to FIG. 4. First, the audio signal is compressed by one sample by the audio compression means 9 for the input image signal and audio signal (step S11).

The compression control means 7 measures this compression processing time and updates the reference value of the time required for audio compression (step S12). Further, the time allocated to the image compression processing is obtained from the difference between the actual time of the data used for the audio compression and the compression processing time (step S13), and the allocated time is compared with the reference value of the current image compression processing time. Then, it is determined whether the compression of the image signal of one frame is possible (step S14). If the image compression of one frame is possible, the image is compressed by one frame by the image compression means 8 (step S15). If the time for performing image compression has not been allocated, the compression process ends.

After performing the compression processing of one frame, the compression control means 7 measures the processing time of the image compression and updates the reference value of the time required for the image compression (step S16). Subsequently, it is determined whether or not there is image data of the next frame that has not been compressed yet from the currently set frame rate (step S17). If there is, the process returns to step S13 and returns to step S13. Repeat the compression process. If there is no image data to be compressed, the compression processing ends.

FIG. 5 is a flowchart for explaining the operation of the decompression device in the device of the present invention. In FIG. 5, when communication is started, compression and decompression capabilities of images and sounds are exchanged between the communicating terminals (step S21). In this capacity exchange, the frame rate of the image and the reference value of the processing time per frame that are optimal for the processing capacity of each terminal are set (step S22). When the decompression device receives the compression-encoded image data and the compression-encoded audio data (step S23), one sample of the compression-encoded audio data and the accompanying compressed and encoded image data are decompressed (step S23). S24).

The expansion control unit 27 calls the audio expansion unit 29 to expand the input compressed and encoded audio data, calculates the time allocated to the expansion of the image data from the time required for the expansion, and calculates the time. If it is sufficient time to perform the above, the image expansion unit 28 is called, and the compression-encoded image data obtained from the device control unit 22 is expanded and decoded.

When the decompression is completed, an image of the decompressed and decoded image signal is displayed on the monitor 24 shown in FIG. 2, and the decompressed and decoded audio signal is output by the speaker 26 (step S25). The processing of steps S23 to S26 is repeated until the reception ends (step S26).

FIG. 6 is a flowchart for explaining the operation of an example of the extension control section. This expansion control section comprises the expansion control means 27, the image expansion means 28 and the audio expansion means 29 of FIG. The expansion operation of this embodiment will be described in more detail with reference to FIG. 6. First, the compression-encoded audio data is expanded by one sample with respect to the input compression-encoded image data and compression-encoded audio data (step S1). S31).

The expansion control means 27 measures the audio expansion processing time and updates the reference value of the time required for audio expansion (step S32). The expansion control means 27 obtains the time allocated to the image expansion processing from the difference between the actual time of the data used for audio expansion and the expansion processing time (step S33), and calculates the allocated time and the current image expansion processing. By comparing the time reference values, it is determined whether or not one frame of image can be expanded (step S34). If one frame of image can be expanded, the compression-encoded image data is expanded by one frame (step S35). If the time for performing image expansion is not assigned, the expansion processing is terminated.

After performing the decompression processing of one frame of the compressed and encoded image data, the decompression control means 27 measures the image decompression processing time and updates the reference value of the time required for the image decompression (step S36). Subsequently, it is checked whether or not the image code of the next frame exists (step S37). If there is an image code that has not been expanded yet, the process returns to step S33 and the image expansion processing of steps S33 to S36 is repeated. If there is no image data to be decompressed, the decompression process ends.

[0039]

Next, an embodiment of the present invention will be described with reference to the drawings. When video signals and audio signals require real-time compression and decompression, such as videophones, and when audio must be reproduced without interruption, image processing and audio processing can be performed at high speed while prioritizing audio processing. Need to switch.

FIG. 7 shows the relationship between the current CPU processing capacity and
FIG. 2 is a block diagram of an embodiment of a compression control device that performs processing by switching between audio compression and image compression so that audio signals are not interrupted. In the figure, a compression control device 80 is an embodiment of the compression control means 7 and has at least an image buffer 81 and an audio buffer 82. The image compression device 83 and the audio compression device 84 are embodiments of the image compression means 8 and the audio compression means 9.

The compression control device 80 stores the input image data and audio data in the image buffer 81 and the audio buffer 82, respectively.
3. Transfer to the audio compression device 84. The image compression device 83 and the audio compression device 84 compress and encode the input image data and audio data in designated units, and output the obtained compression-encoded data to the compression control device 80.

FIG. 9 is a diagram showing an example of the operation of the compression control device 80. 7, the same components as those in FIG. 7 are denoted by the same reference numerals. Here, a description will be given of a case where the processing capacity of the CPU is exchanged between the terminals and the data compression processing capacity per second is set to one sample of audio and three frames of image (hereinafter, a group of this data is defined as one sequence). .

First, in order to perform synchronous compression so that audio is not interrupted, input audio data a1 and audio data a2 are input.
And the image data p1 to p6 are stored in the audio buffer 8 respectively.
2. Save in the image buffer 81. While storing the audio data a3 and the image data p7 to p9, the image data p1 to p3 and the audio data a1 are compressed. Since it is necessary to perform compression so that the sound is not interrupted, first, the sound data a1 is compressed.

When the compression of the audio data of one sample is completed, the time required for compressing the audio of one sample is measured. Now, the compression of one sample of audio data a1 is 0.4
If it takes seconds, it is determined whether the image compression of one frame is completed in the remaining 0.6 seconds. The reference value of the image compression time used for the determination of the first frame uses an initial value set at the time of capability exchange between terminals. This time, this initial value is 0.2
If it is a second, compression is performed because it is determined that image compression of one frame is possible.

When the image compression of the image data p1 of one frame is completed, the time required for the image compression of one frame is measured. In order to use the measured compression time as a reference value for the compression time of the next frame, the reference value of the image compression time is updated. Assuming that it takes 0.2 seconds to compress one frame of image data p1, it is determined whether image compression of one frame can be performed in the remaining 0.4 seconds. By repeating the image compression process in this manner, in the above example, the compression of the audio data a1 of one sample and the image data p1 to p3 of three frames are performed.
Can be performed as shown in FIG.

Since switching to image compression is performed after audio compression for one sample is completed, no overhead is generated for storing a large register used in the compression processing, and high-speed operation can be realized. Also, since the audio compression processing of one sample is completed before the input of the next one sequence, the audio can be compressed without interruption.

However, when an application having a large amount of processing is operated at the same time, the set processing capacity may not be obtained by the capacity exchange performed at the time of starting. The processing in such a case will be described with reference to FIG. In FIG. 10, it is assumed that the load on the CPU increases due to the operation of another application at the time of compressing the nth sequence. Assuming that the audio compression of the n-th sample takes 0.5 seconds, the processing time allocated to image compression is 0.5 seconds. At present, it is set that image compression can be performed in 0.2 seconds per frame. Therefore, it is determined that compression of one frame can be performed in the time allocated for image compression, and thus image compression processing is called, and m The image compression of the frame is performed.

When the image compression of the first frame (m-th frame) is completed and the compression time is measured, if it takes 0.3 seconds, the time allocated to the image compression of the (m + 1) -th frame is 0.2 seconds. It is determined that the image compression cannot be performed as compared with 0.3 seconds which is a reference value of the time required for the newly set image compression. Therefore, the compression of the image ends here, and the audio compression of the (n + 1) th sample is performed. However, in this case, 1.2 seconds are assigned to the compression of the image and the sound, which is 0.2 seconds of the carry-over time from the compression of the previous sequence and 1 second of the reference assignment time.

Assuming that the audio compression of the (n + 1) -th sample is completed in 0.5 seconds, the time allocated to the image compression of the (m + 3) -th frame is 0.7 (= 1.2−0.5) seconds. Since it is longer than 0.3 seconds, which is the reference value of the compression time, it is determined that image compression is possible, and image compression is performed. Assuming that the image compression of the (m + 3) th frame is completed in 0.3 seconds, 0.4 (= 0.7−0.3) seconds, which is the time allocated to the image compression of the (m + 4) th frame, is a reference value of the image compression time. Therefore, it is determined that image compression is possible, and image compression of the (m + 4) th frame is executed.

Assuming that the image compression of the (m + 4) -th frame is completed in 0.3 seconds, 0.1 (= 0.4−0.3) seconds, which is the time allocated to the image compression of the (m + 5) -th frame, is equal to the image compression. Since it is shorter than 0.3 seconds, which is the reference value of time, it is determined that image compression cannot be performed. for that reason,
The compression of the image ends here, and the audio compression of the (n + 2) th sample is performed. However, in this case, 1.1 seconds are assigned to the compression of the image and sound, which is 0.1 seconds of the carry-over time from the compression of the previous sequence and 1 second of the reference assignment time. Hereinafter, the same operation as described above is repeated.

Even if the load further increases, the allocation time required for one sequence can be reduced by not performing image compression, so that the next compression processing is brought forward as in the case of FIG. Thus, it is possible to increase the allocation time of the next sequence. FIG. 11 shows an example of this.

In FIG. 11, it is assumed that the reference value of the image compression time of the i-th sequence is 0.4 seconds. Assuming that the audio compression of the i-th sample takes 0.7 seconds as shown in FIG. 11, the time allocated to the image compression of the j-th frame is 0.3 (= 1.0−0.7) seconds. Since the reference value of the image compression time is shorter than 0.4 seconds, the compression of the i-th sequence is terminated without executing the image compression.

In the compression of the (i + 1) th sequence, the compression processing time allocated to this sequence is 1.3 seconds, which is a total of 0.3 seconds of the carry-over time from the compression of the previous sequence and 1 second of the reference allocation time. And
If it took 0.7 seconds to compress the audio, then 0.
Six seconds are allocated. Since 0.6 seconds allocated to this image compression is longer than 0.4 seconds which is the current reference value of the processing time of image compression, as shown in FIG.
The compression processing of the image of the (j + 3) th frame which is the first image of the first sequence is executed.

By starting the compression processing after storing the data in the image buffer 81 and the audio buffer 82 in this way, not only can the delay of the compression processing be absorbed, but also the compression of the audio is not interrupted because it can be advanced. Even if there is a sequence in which image compression is not performed, a compression device that is updated so that image compression can be performed in a few sequences thereafter can be realized.

Next, according to the current processing capacity of the CPU,
An embodiment of a decompression control device that performs processing by switching between audio decompression and image decompression so that audio is not interrupted will be described. FIG. 8 shows a block diagram of one embodiment of the decompression control device. In the figure, a decompression control device 90 is an embodiment of the decompression control means 27 and has at least an image code buffer 91.
And a voice code buffer 92. In addition, the image decompression device 93 and the audio decompression device 94
And an embodiment of the voice expansion means 29.

The decompression control device 90 stores the input compressed and coded image data and compressed and coded audio data in the image code buffer 91 and the voice code buffer 92, respectively. Pass to. The image decompression device 93 and the audio decompression device 94 decompress the input compression-encoded image data and compression-encoded audio data in designated units, and output the obtained data to the decompression control device 90.

FIG. 12 is a diagram showing an example of the operation of the decompression control device 90. 8, the same components as those in FIG. 8 are denoted by the same reference numerals. Here, a description will be given of a case where data per second is one sample of audio and three frames of images as a result of the exchange of the processing capacity of the CPU between the terminals. First, the input audio code a1 and image codes p1 to p3 are stored in an audio code buffer 92 and an image code buffer 91, respectively, in order to perform synchronous expansion so that audio is not interrupted.

While the audio code a2 and the image codes p4 to p6 are stored, the image code p1 to p3 and the audio code a1 are expanded. Since it is necessary to perform expansion so that the voice is not interrupted, first, the voice code a1 is expanded. At the time when the decompression of the voice code a1 of one sample is completed, the time required for voice decompression of this sample is measured. Assuming that the extension of the audio code a1 takes 0.4 seconds, it is determined whether the image extension of one frame is completed in the remaining 0.6 seconds.

The reference value of the image expansion time used for the determination of the first frame uses an initial value set at the time of capability exchange between terminals. In this case, if the initial value is 0.2 seconds, the time allocated to image expansion is shorter than 0.6 seconds, and it is determined that image expansion of one frame is possible, so image expansion is performed. When the expansion of the image code p1 of one frame is completed, the time required for image expansion of this frame is measured. In order to use the measured extension time as a reference value for the extension time of the next frame, the reference value of the image extension time is updated.

FIG. 12 shows the expansion of the image code p1 of the frame 1.
Assuming that it took 0.2 seconds as shown in
It is determined whether the image code p2 of the next one frame can be expanded in seconds. By repeating the image expansion process in this way,
In the above example, one voice sample and three image frames can be expanded in each sequence.

Since switching to image decompression is performed after audio decompression is completed for one sample, high-speed operation can be realized without generating overhead for storing a large-sized register used in decompression processing. In addition, since the audio decompression process of one sample is completed before the input of the next one sequence, the audio can be extended without interruption.

However, when applications with a large amount of processing are simultaneously operated, a case may occur in which the set processing capacity cannot be obtained by the capacity exchange performed at the time of startup. The processing in such a case will be described with reference to FIG.

In FIG. 13, it is assumed that the load on the CPU has increased due to the operation of another application during the expansion of the n-th sequence. If it takes 0.5 seconds for the audio expansion of the n-th sample, the processing time allocated to image expansion is 0.5 (= 1.0-0.5) seconds. At present, it is set that image expansion can be performed in 0.2 seconds per frame, and since this time is shorter than the processing time 0.5 seconds allocated to image expansion described above, it is allocated to image expansion. It is determined that one frame can be expanded in time, the image expansion process is called, and the image expansion of the m-th frame is performed.

When the image expansion of the m-th frame is completed and the expansion time is measured, it takes 0.3 seconds.
The time allocated to the image expansion of the (+1) th frame is 0.
2 (= 0.5−0.3) seconds, and it is determined that image expansion cannot be performed as compared with 0.3 seconds, which is a reference value of the time required for newly set image expansion. Therefore, image expansion ends here, and audio expansion of the (n + 1) th sequence is performed. However, in this case, 1.2 seconds are assigned to the extension of the image / sound, including 0.2 seconds of the carry-over time from the extension of the previous sequence and 1 second of the reference assignment time.

Assuming that the audio expansion of the (n + 1) -th sample is completed in 0.5 seconds, as shown in FIG. 13, the audio is allocated to the (m + 3) -th frame, which is the compression-encoded image data of the first frame of the (n + 1) -th sequence. The time taken is 0.7 (= 1.2−0.5) seconds, which is longer than the reference value of 0.3 seconds which is the reference value of the image expansion time. Be executed.

Assuming that the image expansion of the (m + 3) th frame is completed in 0.3 seconds, the time allocated to the image expansion of the (m + 4) th frame is 0.4 (= 0.7−0.3).
Since this is longer than the reference value of 0.3 seconds which is the reference value of the image expansion time, it is determined that the image can be expanded, and the image expansion of the next (m + 4) th frame is executed. Hereinafter, the same operation as described above is repeated.

Even if the load further increases, the time required for one sequence can be reduced by not performing image decompression, so that the next decompression processing is brought forward as in the case of FIG. Thus, it is possible to increase the allocation time of the next sequence. FIG. 14 shows an example of this.

In FIG. 14, it is assumed that the reference value of the image expansion time of the i-th sequence is 0.4 seconds. As shown in FIG. 14, assuming that the audio expansion of the i-th sample takes 0.7 seconds, the time allocated to the image expansion of the j-th frame, which is the expansion encoded image data of the first frame of the i-th sequence, is 0. .3 (= 1.0-0.7) seconds,
Since it is shorter than the reference value of 0.4 seconds,
The extension of the i-th sequence is completed without performing the image extension.

In the decompression of the (i + 1) th sequence, the decompression processing time allocated to this sequence is 1.3 seconds in total, which is 0.3 seconds of the carry-over time from the decompression of the previous sequence and 1 second of the reference allocation time. And
If it takes 0.7 seconds to expand the audio, 0.
Six seconds are allocated. Since 0.6 seconds allocated to this image decompression is longer than 0.4 seconds, which is the reference value of the current processing time of image decompression, as shown in FIG.
J which is the first compression-encoded image data of the first sequence
Decompression processing of the +3 frame compressed image is executed.

As described above, by starting the decompression process after storing data in the image code buffer 91 and the audio code buffer 92, not only can the delay of the decompression process be absorbed, but also the decompression of the sound is interrupted because the data can be moved forward. That is to say, even if there is a sequence in which image expansion is not performed, an expansion device that is updated so that image expansion can be performed in a few sequences thereafter can be realized.

[0071]

As described above, according to the present invention,
Since audio compression / expansion is performed with priority, audio signals can be compressed / expanded and output without interruption.Also, audio compression and image compression are alternately performed, so when switching between audio and image The processing of register storage that occurs can be minimized.

According to the present invention, when the time obtained by subtracting the measured voice compression / expansion time from the designated unit time is shorter than the time required for the image compression / decompression by the image compression / decompression means, the voice compression / decompression is performed. By performing the decompression processing ahead of time, the allocation time of the next sequence of the image signal is increased, so that the image signal can be compressed and decompressed with the minimum missing information.

Further, according to the present invention, since the synchronous processing is realized by the timing and the number of times of calling the image and audio compression means and expansion means, the processing can be realized without changing the compression means and expansion means. it can.

[Brief description of the drawings]

FIG. 1 is a block diagram of an embodiment of a compression device in an image / sound synchronization processing device according to the present invention.

FIG. 2 is a block diagram of an embodiment of a decompression device in the image / sound synchronization processing device according to the present invention.

FIG. 3 is a flowchart for explaining the operation of FIG. 1;

FIG. 4 is a flowchart illustrating the operation of a compression control unit in FIG. 1;

FIG. 5 is a flowchart for explaining the operation of FIG. 2;

FIG. 6 is a flowchart for explaining the operation of the decompression control means in FIG. 2;

FIG. 7 is a block diagram of an embodiment of a compression control device according to the present invention.

FIG. 8 is a block diagram of an embodiment of a decompression control device according to the present invention.

FIG. 9 is an operation explanatory diagram of an example of FIG. 7;

FIG. 10 is an operation explanatory diagram of another example of FIG. 7;

FIG. 11 is an operation explanatory diagram of still another example of FIG. 7;

FIG. 12 is a diagram illustrating an operation of the example of FIG. 8;

FIG. 13 is an operation explanatory diagram of another example of FIG. 8;

FIG. 14 is an operation explanatory diagram of still another example of FIG. 8;

FIG. 15 is a block diagram of an example of a compression device in a conventional image / sound synchronization processing device.

FIG. 16 is a block diagram of an example of a decompression device in a conventional image / sound synchronization processing device.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1, 21 Central processing unit (CPU) 2, 22 Device control means 3 Image input means 4 Camera 5 Audio input means 6 Microphone 7 Compression control means 8 Image compression means 9 Audio compression means 10 Code synthesis means 11 Transmission means 12, 32 Modem Reference Signs List 23 image output means 24 monitor 25 audio output means 26 speaker 27 expansion control means 28 image expansion means 29 audio expansion means 30 code separation means 31 reception means 80 compression control device 81 image buffer 82 audio buffer 90 expansion control device 91 image code buffer 92 Voice code buffer

Claims (6)

[Claims] An input unit for inputting an image signal and an audio signal in real time; an image compression unit for compressing the input image signal; an audio compression unit for compressing the input audio signal; Means for controlling the audio compression means so as to give priority to compression of an audio signal for one sample within a specified unit time among image compression by the means and audio compression by the audio compression means. When the time required is measured and the time obtained by subtracting the measured audio compression time from the specified unit time is longer than the time required for image compression by the image compression means, the sound for one sample is output by the image compression means. Compression control means for image-compressing an image signal corresponding to a signal; output compression-encoded audio data of the audio compression means; and output compression-encoding of the image compression means Image and sound synchronization processing apparatus characterized by an output means for outputting the image data respectively synthesized and. 2. The image compression device according to claim 2, wherein the time obtained by subtracting the measured audio compression time from the designated unit time is shorter than the time required for image compression by the image compression device. 2. The image / sound synchronization processing apparatus according to claim 1, wherein said sound compression means causes said sound compression means to perform sound compression for the next one sample instead of the sound compression. 3. The image / audio synchronous processing apparatus according to claim 1, wherein the time required for image compression by said image compression means is updated to the time required for previous image compression. 4. Receiving means for receiving data obtained by code-synthesizing the compression-encoded audio data and the compression-encoded image data, and separating the compression-encoded audio data and the compression-encoded image data from the code-combined data. Code decompressing means, image decompressing means for decompressing the compressed and coded image data, sound decompressing means for decompressing the compressed and coded audio data, and image decompression by the image decompression means and audio decompression by the sound decompression means. The audio expansion unit is controlled so that expansion of the compression-encoded audio data for one sample is preferentially performed within a specified unit time, and the time required for the audio expansion is measured. When the time obtained by subtracting the measured sound expansion time from the unit time is longer than the time required for image expansion by the image expansion means, Decompression control means for image decompression of the compressed and encoded image data corresponding to the compressed and encoded audio data for the sample, and a synchronous output for synchronously outputting the output audio data of the audio decompression means and the output image data of the image decompression means, respectively. And an image / sound synchronization processing device. 5. The image expansion device according to claim 1, wherein when the time obtained by subtracting the measured voice expansion time from the specified unit time is shorter than the time required for image expansion by the image expansion device, the image expansion device performs image expansion by the image expansion device. 5. The image / sound synchronization processing device according to claim 4, wherein the sound expansion unit causes the sound expansion unit to perform sound expansion for the next one sample. 6. The image / audio synchronization processing apparatus according to claim 4, wherein the time required for image expansion by said image expansion means is updated to the time required for the previous image expansion.