JP2003235041A - Real time picture encoding apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To control the coding quantity depending on the processing situation at the terminal when decoding or compositing a plurality of pictures or sounds simultaneously. <P>SOLUTION: The decoding apparatus of this invention comprises: a reception control unit 11 for receiving the information; a separation unit 12 for analyzing and separating the received information; a priority decision unit 14 for determining the priority of processing of the pictures separated in the separation unit 12; a picture expanding unit 18 for expanding the pictures according to the determined priority; a picture compositing unit 19 for compositing the pictures on the basis of the expanded pictures; a composition result accumulating unit 22 for accumulating the composited pictures, a reproduction time control unit 23 for controlling the time for starting reproduction; and an output unit 24 for delivering the result of composition according to the information of the reproduction time control unit 23. <P>COPYRIGHT: (C)2003,JPO

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a real-time image
The present invention relates to an encoding device. 2. Description of the Related Art Heretofore, it has been difficult to determine whether or not an image of
Extract a person image, for example, and
The sent person image and the pre-stored partner
Superimposes the image of the virtual space to be displayed in common
To show that the other party is in front of you.
One that satisfies the presence and aims for realistic video communication
(Japanese Patent Publication No. 4-24914, "Hypermedia
ASystem Personal Communication System "
(Fukuda, K., Tahara, T., Miyoshi, T.: "Hypermedia
Personal Computer Communication System: Fujitsu H
abitat ", FUJITSU Sci. Tech. J., 26, 3, pp.197-206
(October 1990).), Nakamura: "Network-enabled virtual reality
Distributed Collaborative Work Support by Reality ", Information Processing Society of Japan
Research Group on Ovisual Composite Information Processing (1993)). Special
In addition, in the conventional technology, speedup for performing image synthesis,
Inventions have been made on methods for reducing
For example, Japanese Patent Publication No. Hei 5-46592: Image Compositing Apparatus,
105226: image synthesizing device). [0003] However, the conventional
In technology, 2D still images and 3D CG data are synthesized
Was proposed, but multiple video
And audio are simultaneously decoded (decompressed), synthesized and displayed.
It did not describe how to implement the system.
In particular, it can decode, synthesize, and display multiple video and audio simultaneously.
Terminal equipment lacks capacity or changes in processing capacity.
Video and audio playback methods that do not break down
Was not stated. In addition, depending on the billing situation
It describes how to decode, combine and display multiple videos.
Had not been. Specifically, (1) information of a plurality of images and sounds, and
Those who manage the information describing the relationship and the information of the processing result
Law. (2) Multiple images when the processing state of the terminal is overloaded
For determining the priority of decoding, synthesis, and display of
And billing methods. [0005] Further, a plurality of videos and sounds are simultaneously decoded and combined.
In the environment that can be created and displayed, the state of the receiving terminal and the receiving terminal
Image compression method according to decoding, synthesis, and display priority
And consider how to control the amount of coding.
Not. SUMMARY OF THE INVENTION The present invention provides a conventional
To simultaneously decode and combine multiple video and audio
When performing coding, the amount of coding is controlled according to the processing status of the terminal.
Control of multiple video and audio depending on the billing status.
Video and audio decoding device that can control the
Providing an image / speech coding apparatus and an information transmission system
The purpose is. The present invention is limited to only two-dimensional image synthesis.
Not. Table combining two-dimensional and three-dimensional images
The current format may be used, or a wide-field image (panoramic image)
Combining multiple images adjacent to each other
Methods may also be included. The communication form targeted by the present invention is a wired form.
Not only bidirectional CATV and B-ISDN. example
For example, video and audio transmission from the center terminal to the home terminal
Transmission is by radio wave (for example, VHF band, UHF band), satellite broadcasting
Information transmission from the home terminal to the center terminal is
It may be a telephone line or N-ISDN for logs (video,
Voice and data need not necessarily be multiplexed.
No). In addition, IrDA, PHS (Personal Handy
Communication type using wireless such as -phone) and wireless LAN
It may be in a state. The target terminal is a portable information terminal.
Even a portable terminal like this, a set-top box,
A desktop terminal like a personal computer
Is also good. Specifically, the present invention according to claim 1
Akira has one or more image input means for manually inputting images, and
Image input management means for managing the control state of the image input means
And other terminal control request management to manage the receiving status of the receiving terminal
Means and at least the reception status of the managed receiving terminal
Or, according to the control state of the image input means,
Encoding processing determining means for determining an encoding method;
Encoding the input image in accordance with a result determined by the logic determining means
Image encoding means for outputting the encoded image.
Real-time image, comprising:
An image encoding device. In this application, the following items 1 to 19
There is technology disclosure. 1. Image coding for coding image information
Means and transmitting the encoded various information
An image encoding device having transmission management means for recording, and a code
Management means for receiving various types of information, and its reception
Decoding means for decoding decoded various information, and decoding thereof
Image synthesizing means for synthesizing one or more images thus obtained, and
Decoding having output means for outputting a combined image of images
Image decoding coding apparatus, comprising:
Place. 2. Speech coding for coding speech information
Means and transmitting the encoded various information
Speech coding apparatus having transmission management means for recording, and code
Management means for receiving various types of information, and its reception
Decoding means for decoding decoded various information, and decoding thereof
Voice synthesizing means for synthesizing one or more generated voices, and
Decoding having output means for outputting synthesized speech
Coding apparatus characterized by comprising a coding device
Place. 3. Item 1 of the image decoding / encoding device
And the speech decoding / encoding device according to item 2,
The image encoding device and / or the audio encoding device,
Predetermine the priority of processing when the coded information is overloaded
The encoded information and the decision
Priority adding means for associating the assigned priority,
The image decoding device and / or the audio decoding device receive
Processing of various types of information according to the priority at the time of overload.
Characterized by having priority determining means for determining a method.
Video / audio encoding / decoding device. [0015] 4. The priority adding means and the priority
The degree determining means includes at least an image coding method and an image
Size, contrast, image composition ratio, quantization
Group, frame number, number of frames, interframe coding and
Difference in intra-frame encoding, display position, display time, sound interval
Based on one or more of the differences between
Decoding, synthesis, display order, presence / absence of encoded images and audio
To determine how to handle priorities,
A feature that determines the priority to be processed
4. The video / audio coding / decoding device according to item 3. 5. The priority adding means and the priority
The degree determination means calculates the time taken for decoding when encoding the image.
And the encoded information based on the time it took to encode.
To determine the priority to be added to the
Item 3 characterized by determining the priority to be set as the elephant
Video / audio coding / decoding device. 6. The priority adding means and the priority
The degree determining means performs the processing of decoding, synthesizing, and displaying the image.
Define an implementation rate that stipulates the number of rows, and based on that implementation rate
To determine the priority to be added to the encoded information,
Determining the priority to be processed at the time of decryption
Item 3. The video / audio coding / decoding device according to Item 3, which is characterized by the following. 7. At least the intra-frame coding frame
Or the first or last frame
In other words, the processing when the scene change frame is overloaded is excellent.
An image according to item 4, wherein the priority is set high.
Audio encoding / decoding device. 8. Inter-coded images are the same
Of the item 4, characterized by assigning the priority of
Video / audio coding / decoding device. [9] Multiple in intra-coded images
Item 4 is characterized by assigning priority levels
Video audio encoding / decoding device. 10. The image decoding means is configured to output one frame
Specially, perform image decoding in smaller units.
Item 3. The image encoding / decoding device according to Item 1. 11. Item 1. The image decoding / encoding apparatus according to Item 1.
And the speech decoding / encoding device according to item 2.
At least information on billing and service content
Information, password, user code, country code,
Information indicating the order of generation and display, information indicating the order of decoding,
One or more of user's instruction, terminal processing capacity, and playback time
Image and audio to be decoded, synthesized, and displayed based on the
The order, presence / absence, and playback method
Video / audio coding / decoding device. 12. Item 1. The image decoding / encoding apparatus according to Item 1.
And the speech decoding / encoding device according to item 2.
In addition, the reception management means may include an image information among the various information.
Information describing the relationship between reports and audio information
Treat it independently as image information and audio information
A video / audio coding / decoding device characterized by the above-mentioned. 13. The image information, the audio information
To identify the description method for describing the relationship between
An item characterized by a description method identified by an identifier
13. The video / audio coding / decoding device according to claim 12. 14. Item 1. The image decoding / encoding apparatus according to Item 1.
And the speech decoding / encoding device according to item 2.
The image synthesizing means or the voice synthesizing means,
Until the sending side instructs to discard the decryption result,
Video and audio characterized by holding, managing and using results
Encoding / decoding device. 15. Item 1. The image decoding / encoding apparatus according to Item 1.
And the speech decoding / encoding device according to item 2.
Information that describes the relationship between image information and audio information
Decoding required when synthesizing images and audio based on
Cannot be synthesized because there is no prepared image or sound
It is especially important to show users that images and sounds are present.
A video / audio coding / decoding device to be referred to. 16. One or more image input
Force means and an image for managing the control state of the image input means
Input management means and other terminals for managing the reception status of the receiving terminal
Control request management means and at least the managed receiving end
Response to the current reception status or the control status of the image input means.
Encoding processing determining means for determining an image encoding method
According to the determination result of the code processing determining means.
Image encoding means for encoding the force image, and the encoded
And output means for outputting an image.
Real-time image coding device. 17. The code processing determining means is configured to
Encode at least according to the control state of the image input means
Priority, priority information of overload processing, coding method, amount
Substep value, number of frames, size of image to be encoded
And decide whether or not to encode or not.
Item 16. The real-time image encoding device according to Item 16. 18. Item 1. The image decoding / encoding apparatus according to Item 1.
And a small number of the audio decoding / encoding devices according to item 2.
At least one of them as a receiving terminal, and
3. A coding and encoding apparatus and the speech decoding and coding according to item 2.
Device and the real-time image encoding device of item 16
At least one of them as the sending terminal
An information transmission system in which communication is
At least the load of the receiving terminal, the priority of the receiving terminal
The encoded information to be processed, determined by the determination means
Information on the priority of the
Transmitting any of the conditions of the kip to the transmitting terminal;
In the transmitting terminal, the image or audio code
Coding, coding priority, coding method, coding
Image size, quantization step value, number of frames, reception
One of the processing priorities for terminal overload
An information transmission system characterized by the following. 19. Item 1. The image decoding / encoding apparatus according to Item 1.
The speech decoding / encoding device according to item 2 and
And the image decoding / encoding device of item 1 and item 2
17. The speech decoding / encoding device according to item 16, and
The real-time image coding device is used as the transmitting terminal,
An information transmission system in which the ends are connected by a communication path.
Image transmission is performed by a retransmission method, and audio transmission is performed by retransmission.
Use a transmission method that does not perform transmission, and at least retransmit the image.
Number, error rate of received information, and discard rate
By transmitting any of the information to the transmitting terminal,
The encoding process determining means includes an encoding method and a quantization step.
Value, number of frames, size of image to be encoded,
Presence / absence and low priority of processing when receiving terminal is overloaded
Information transmission system characterized by determining either
Tem. DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the embodiments.
Will be described with reference to the drawings showing "Image" used in the present invention
The meaning of "image" includes both still images and moving images. In addition,
The image to be created is of computer graphics (CG).
Consisting of two-dimensional images and wireframe models
Is a mixture of three-dimensional image data
You may. In this case, the relationship between the images is wireframe
Equivalent to model. Script language for writing
Are Java (registered trademark) and VRML.
You. FIGS. 1 and 2 show an embodiment of the present invention.
1 is a schematic configuration diagram of an image decoding / encoding device in the present embodiment. FIG.
FIG. 2 shows a configuration in the case where there is no audio playback function.
Is a configuration having a function of reproducing images and sounds. This
Needless to say, the same configuration can be applied to the case of only voice. The present apparatus shown in FIG. 1 or FIG.
And a decoding device, the coding device in the case of FIG.
Sets the priority of processing when the encoded image is overloaded in advance.
It is determined based on the determined standard, and the encoded image and priority are determined.
Priority assigning unit 101 for associating images to be encoded
The encoding unit 102 transmits the encoded information with the priority added.
Alternatively, the transmission management unit 103 for recording, and the encoded
It comprises a reception management unit 104 for receiving information.
In addition, the encoding device in the case of FIG.
The audio encoding unit 105 is provided. On the other hand, in the decoding device, information is received.
The reception management unit 11 for transmitting information and the transmission management unit 13 for transmitting information
Transmits information on coaxial cable, CATV, LAN, modem, etc.
Means to send. The terminal connection form is a TV phone
Video information between terminals like a TV or TV conference system
Transmission and reception, satellite broadcasting, CATV, Internet
Broadcast-type (one-way) video broadcasting on the Internet
It is. In the present invention, such connection forms of terminals
Take into account. The separation unit 12 receives the encoded (compressed)
It is a means to analyze and separate information
Becomes a multiplexing unit by the inverse operation). For example, MPEG1
And MPEG2, H.264. 320 terminals (using N-ISDN
TV phone / conference device rules). 221 is H.
324 terminals (TV telephone / meeting using analog telephone line)
H. 223 is video / audio / data
Is a protocol for multiplexing and demultiplexing. The present invention complies with the rules.
May be realized with a configuration that does not conform to the rules.
May be. H. 323 or on the Internet
Video and audio are in separate streams,
It may be transmitted independently. The priority determining section 14 obtains the
Information (eg video, audio, management information)
Decoding when the terminal is overloaded (hereinafter referred to as "decompression")
) And expand image and audio
(The method of determining the priority of the process is determined in advance by the receiving terminal device.
May be negotiated, or the transmitting terminal (encoding device)
The following method is used for recording media and transmitted packets.
Transmission and recording with the information on the priority determined in
It may be added as a format. Regarding priority
Are expressed as priority "large", "medium", "small"
Expressions that are not quantified and numbers such as 1, 2, 3
Valued expression may be used). Composed of a plurality of images or audio frames
For handling data on a stream-by-stream basis
The data transmission / reception process between the sender and the receiver using the identifier
Process to manage the buffer on the receiving side and the data on the transmitting side.
Data transmission can be scheduled. That is,
Specify the identifier of the stream sent from the sender if necessary.
Notify the receiving status of the receiver and
Notification of the stream identifier to the receiving terminal, from the receiving side
It is possible to request a necessary stream. Priority of processing when coded information is overloaded
Degree is determined based on the criteria described above, and the
Priority adding means for image coding
Of various information received in the device or speech coding device.
Priority to determine processing method according to priority at overload
Image frame or audio of priority to be processed
Is determined, and decoding and combination processing are performed. In addition, in the image frame
About the frame so that the frame can be skipped.
Periodically insert encoded (I-frame) frames
There is a need to. The unit to which the priority is added is video or sound.
Voice frame units (priority ratio between frames)
Comparison), in units of streams composed of multiple frames
Yes (comparison of priorities between streams). As a method focusing on the characteristics of an image, an image
Compression format (for example, H.263 and run-length
Length), the size of the image (for example, C
IF and QCIF, give priority to QCIF), control
Last (for example, prioritize bright contrast
), The image composition ratio (for example,
Priority), quantization step (for example, quantization step
Priority is given to small values of
And intra-frame coding (eg, intra-frame coding
Priority), the display position (for example, if the display position is
Give priority to things. If the image is a three-dimensional image,
When placed in the back, the priority is lower and displayed in the front.
Higher priority), the frame number (number
Scenes where the first and last frames have higher priority
Change frame priority, etc.) and number of frames
(For example, images with a small number of frames to be played
Higher. The frame number is H. 263, tempo
Rally reference (TR) and changes in TR value
Judgment based on the conversion to sound), sound and silence sections,
Method based on display time (PTS) and decoding time (DTS)
Is mentioned. In addition, the interframe encoded P frame
And the B frame are assigned the same priority. Also,
Assigning multiple levels of priority to intra-frame coded images
By doing so, the frequency of skipping can be controlled. As an example focusing on the difference between media,
Is a method that gives priority to audio decompression over image decompression.
No. This allows audio to be streamed without interruption
Can be played. Further, the reproduction of the reproduction managed by the receiving terminal is performed.
Determination of information (image, sound) to be expanded based on permission information
May be performed, or a retransmission may be sent from the transmitting side as control information.
Select the information to be expanded based on the raw permission information
Is also good. The reproduction permission information specifically relates to billing.
Information (for example, if no billing has taken place,
Do not perform processing of display and display. On the receiving terminal side,
Information may be managed, or charging information may be managed on the transmitting side.
Information) indicating the content of the service (for example,
If the terminal does not have permission to play in the broadcast
Does not perform length, composition, or display processing. Playback permission is on the receiving side
It may be managed at the terminal or at the sending terminal.
Password) (e.g. password for certain programs)
If no is input, decompression, synthesis, and display are not performed. path
The word may be managed at the receiving terminal,
May be managed), user code (eg,
If it is not a user who has been expanded, decompression, synthesis and display are performed
Absent. The user code may be managed by the receiving terminal,
May be managed by the sending device), country code (e.g.,
Depending on the country, images and sounds to be expanded, synthesized, displayed, and playback
Change the way. The country code may be managed by the sender.
Alternatively, it may be managed on the receiving side. Play by country code
The scramble can be realized by changing. Information on billing and contents of service
Image and audio such as information, password, user code
Reproduction methods with restrictions on playback permission include
When creating or displaying images, the position or pixels may be shifted
Image scaling, image sampling (eg,
, Pixel inversion, contrast change, etc.
Update, change color palette, skip frames
And the like. The playback method of these images (image
Stretching, compositing, and display)
Good. Alternatively, H.264, which is one of image compression, is used. Set at 263
Smaller than one frame as defined and processed independently
GOB (Group Of Bloc)
k) Restrict image expansion, synthesis, and display methods in units
This allows the entire screen to be
Control that is more flexible than the method that disturbs the That is, G
By processing in OB units, only a part of the screen is scanned.
Because you can crumble,
Reputation for interactive software
Value becomes possible. Similarly, as a method of reproducing the sound,
Change the sound direction, change the sound frequency,
Change, change the sampling of the sound,
There is a method of inserting images and sounds (any method
Also, processing in advance on the sending side and processing in the receiving side
Method). Image and sound reproduction methods include image and sound
Out of sync. Combination, display order
Information to be displayed (The display order is determined in advance by the receiving terminal.
For example, priority is given to CIF and still images,
On the side, the order displayed in the transmission information and the information about the priority
The extension order is also indicated.)
Information (Decompression order is determined in advance by the receiving terminal.
For example, QCIF or intra-frame encoded image data
Priority is given to conversation sounds over BGM
Longer. Similarly, on the sending side, send information
May be added to the display order).
User instructions (for example, extension,
Select the image and audio information to be displayed or displayed, or
Decompress, combine and display based on the information selected accordingly
Determine the image and audio information), the processing power of the device (e.g.
Occupy the processing of CPU for a certain period of time now or in the past
By measuring the time, images that are likely to take
Suppress expansion, synthesis, and display of images and sounds. Estimate processing time
The method is to use local decoding when performing compression.
The time taken and the time taken to compress
Information and manage it in conjunction with
Configuration, whether to display, priority can be determined),
Playback time (for example, images, audio information
(Decompression, synthesis, and display are stopped) and decoding time.
The priority and presence or absence of the image or sound to be lengthened may be determined. In addition, only specific images and sounds have priority.
As a way to prevent stretching and display,
Information on the rate of voice expansion, synthesis, and display processing
The order and presence or absence of images to be decompressed, synthesized and displayed based on
Can be determined. For example, ten times of stretching
One of them is the receiving terminal that expands the CIF size image once.
Set on the sending side, or decompress or
And the implementation rate of the display
A method of transmitting audio information is conceivable. Implementation rate is specific
Of the I frame (frame encoded within the frame)
It can be defined by the insertion interval. This allows for specific images and audio
Only objects are not stretched, composited or displayed
Become. Priority for controlling these expansion, synthesis, and display
Information about the relay is added not only to the sending device, but also to the relay device.
May be added and controlled by a device that performs In addition, the receiving terminal
Regarding the priority determined by the priority determining unit 14 of the decoding device
Transmitting information to a destination via the transmission management unit 13;
The video and audio transmission according to the determination status of the priority determination unit 14
(Image objects that are difficult to select)
By sending the project ID to the sender,
No longer). If the receiving terminal is overloaded,
Information indicating the priority of the
May be transmitted as a transmission format.
Recording media such as CD-ROMs and hard disks.
Of MPEG2 as a format for recording
Transport streams may be expanded or standardized
Transmission and recording format not considered
No. Also, for each media (video, audio,
Multiplexed as separate streams into
The transmission and recording may be performed without performing. The image decompression unit 18 as image decoding means
A means for performing image decompression processing (hereinafter referred to as the encoding device
If the encoding means), the image format handled by the image decompression unit 18
MPEG1, MPEG2, H.264 261,
H. 263 and the like. Image expansion is single frame
H. GOB unit specified in 263
May be performed. When processing one frame at a time
When performing inter-frame coding, the
The state needs to be stored in the image decompression unit 18. GOB
When images are expanded in units, the order of image expansion
Is no longer a problem. Therefore, decompression processing is performed in GOB units.
When performing, it is necessary to have a plurality of image decompression units 18 in the receiving device.
Rather, one video decompression unit 18 decompresses multiple videos.
It becomes possible. On the other hand, it is necessary to store the growth results
There is. Audio decompression unit 2 as audio decoding means in FIG.
0 is a means for expanding the audio,
The audio format is G. 721 and G. 723 etc.
Is mentioned. Processing methods include DSP and general
Software processing and dedicated hardware by CPU
Processing. In the case of realizing by software, images and
And audio decompression processing are performed by one process or one process.
Images and audio to be expanded at the same time are managed in units of red
If there are multiple processes, the number of processes or
Process in a time-sharing manner with threads. The image decompression management unit 15 checks the state of image decompression.
It is a means to manage. Also, the audio decompression management unit 16
This is a means for managing the state of extension. For example, these management
When the unit is realized by software, the separation unit 12
Obtained compressed information is determined according to a defined procedure (for example,
Executed by the audio decompression unit 20 and then executed by the image decompression unit 18
Is delivered to the image decompression unit 18 and the audio decompression unit 20
And monitor the extension status. When all elongation is completed
For example, the image synthesizing unit 19 or the voice synthesizing unit 21
Hand over the information. In software, shared memory
By using semaphores, you can limit the information to be passed,
It knows that the decompression process has been completed (details will be described later).
). The time information management unit 17 stores time information.
It is a means to manage. For example, if the system is
When implementing on a computer, time information is personal
What is necessary is just to realize using the timer of a computer. The image synthesizing section 19 expands the image data.
Image synthesis is performed based on. A place to combine multiple images
Image combination based on the composition ratio (α value) of each image.
Performing For example, when combining two images, the foreground
When the composition ratio of the image is α, the RGB value of the background image is set to 1−
α, foreground images are mixed at a ratio of α. The image to be expanded
The image is displayed by managing the processing in units of one frame.
When synthesizing multiple images using the indicated time,
Configuration and implementation can be simplified. Also, if the image synthesizing unit 19
Or the speech synthesizer 21 discards the decompression result from the transmitting side.
Until instructions are received, manage and use the decompression results.
With the sending side, the same pattern information is repeatedly transmitted from the transmitting side.
Need to be eliminated. Information describing the relationship between images and between sounds
Decoding required when synthesizing images and audio based on
Cannot be synthesized because there is no prepared image or sound
By presenting the presence of images and sounds, users can
You can know the composition status. So you need a user
Do not select an image quality that is
Which instructions will allow you to
Can be achieved. In addition, the decoded image and sound
As a method of storing and managing data in a buffer,
Delete images in order of arrival,
Look at the script that describes the relationship between the voices, as a whole
Check the usage status of the decoded image and audio data
There is a way to leave. The audio decompression management unit 16 includes at least one
Manages the expanded state of the audio expander 20 that expands the above audio
I do. The voice synthesizer 21 uses the expanded information
Means for synthesizing the speech, and the synthesis result accumulation unit 22
The image synthesized by the image synthesis unit 19 and the voice synthesis unit 21 are combined.
This is a means for storing generated speech. The reproduction time management section 23 should start reproduction
This is a means for reproducing the synthesized image or sound at the time. The output unit 24 outputs a composite result (example:
For example, display, printer, etc.), input unit 2
5 means for inputting information (for example, keyboard, mouse
, Cameras, videos, etc.). Terminal control unit 2
Reference numeral 6 denotes a unit for managing these units. FIG. 3 shows the priorities in communication and recording formats.
FIG. 9 is a diagram for explaining an example of a case in which information about information is added. The example shown in FIG. 3 (a) shows that all media
A) In this example, video (audio, control information) is multiplexed.
You. The control information is used to determine the overload processing.
Indicates the priority (priority pointed out in the present invention) and the display order
Priority is indicated. The control information includes
Relationship between images, between voices, and relationship between images and voices (temporal, location
May be described. FIG.
In the example of (a), for example, multiplexing of MPEG1 / 2,
H. Control information and data (video, audio) like 223
Suitable for application of mixed packet multiplexing. In addition,
Priority of loading process is frame unit or stream
Add in units. In the example of FIG. 3B, a large amount of information is
This is an example of overlapping. In this example, control information, image information
Information and voice information are transmitted from different communication ports. image
Information on the relationship between voices, between voices, and between
Sent as control information from a communication port separate from images and audio
do it. H. Multiple like 323 or internet
For applications where communication ports can be established simultaneously.
Therefore, the multiplexing process can be simplified as compared with FIG.
Thus, the load on the terminal can be reduced. As a method of describing images and audios, J
It can be supported by description languages such as AVA and VRML.
It seems that there is a unique scripting language specification
It is possible that the situation is not fixed. So between images, audio
Relationships (eg, positional information, temporal information (table
Period))).
Support multiple types of description methods by providing an identifier for
can do. Knowledge to identify how information is described
As a method of adding a separate element, for example, in MPEG2
Is a program that manages MPEG2-TS streams.
Provided in the system map table or written a script
This can be handled by providing it in the stream. Overload handling
Priority and information describing the correspondence between images and audio
(Control information). In MPEG2
Is MPEG2-TS (transport stream)
Related video and audio streams
So that it can be managed with the program map table
In addition, structural information and strikes for associating images with audio
If a stream is defined and managed, MPEG2
Can be transmitted independently. FIG. 4 shows a tree structure constructed by software.
It is a figure explaining the example of the case. Multitasking operation
The invention on operating systems capable of
Is realized, the processes described in FIG. 1 and FIG.
Software execution modules such as processes and threads
Divided into units, shared memory between each process and thread
The information is exchanged by the semaphore (in the example of FIG. 4,
The part shown by the solid line corresponds to the semaphore).
Exclusive control of the stored information is performed. The following describes each process and process.
The function of Red will be described. The DEMUX thread 31 is connected to a network or
Information (video, audio, control information) multiplexed from the disc
Information), audio, video and the correspondence between audio and video
Surveillance table describing relationships and information about playback time
(Details will be described later). DEMUX thread
Reference numeral 31 corresponds to the separation unit 12 described above. DEMUX Thread
The information separated in the buffer 31 is stored in the ring buffer 3 for audio.
2. Ring buffer 33 for video, ring buffer for monitoring
To each other. If it is audio information,
The information sent to the ring buffer 32 is
Expanded by a thread 35 (corresponding to the above-described audio expansion unit 20)
Lengthened. If it is video information, the
The transmitted information is decompressed by the decoding process 36.
You. Regarding the monitoring table, the ring buffer
To determine the order for decompressing the video.
Monitoring thread 37 (the terminal control unit 26 described above,
(Corresponding to the image expansion management unit 15 and the audio expansion management unit 16)
Used. Also, the same monitoring table is used for image synthesis.
This is used in the image composition thread 39. Monitoring thread
The surveillance table used at 37 is for all audio and video
When the image has been expanded, the next table is
Read from the file 34. Decoding process 36 (described above)
The image information expanded by the image expansion unit 18)
The data is sent to the video single buffer 38. Sent out
When the image information is complete, the image synthesizing thread 39 (described above)
In the monitoring table)
Image synthesis is performed using the managed image synthesis ratio. Combination
The synthesis result is stored in the synthesis buffer 41 (the synthesis result accumulation unit described above).
22 corresponding to the display monitoring thread 42
Wait until the display time comes and wait for the display
(Corresponds to the playback time management unit 23). FIG. 5 shows the structure of information used in the structure of FIG.
It is a figure explaining structure. In the example of FIG.
Or the information received from the network is 188 bytes
e is a fixed length (B). DEMUX thread 31 minutes
The structure of the released audio information is a code for packet synchronization,
Playback time, frame length indicating the length of audio to be played, sound
It consists of voice data (C). The structure of video information is packet
Code for synchronization, frame number to identify the image,
Frame length indicating the size of image information,
(D). The present invention needs to be performed in units of one frame.
Not a small block unit such as a macro block unit.
It is permissible to perform processing at the order. The structure of the monitoring table is as follows when an image is displayed.
The number of images to be displayed (combined) in one frame, each image
ID, frame number, priority for decompression and display, frame
Identifiers (I-pictures, P-pictures,
B picture), display horizontal position, display vertical position, composition
(E). What
Change the image composition ratio and audio composition ratio
You may let it. For example, two types of images are sound
In the case of two types, the image composition ratio is α: 1−α.
In some cases, the corresponding voice synthesis ratio is also α: 1-α
May be attached. Not only the relationship between image information, but also audio
The relationship between them may also be described (eg, direction, type (B
GM, conversation sounds)). FIG. 6 shows the operation of the DEMUX thread 31.
FIG. File or network
Read fixed-length data of 188 bytes from the
-1). Analyzes the read data,
Set to the type of image and structure of the monitoring table (5-
2). If writing to the ring buffer is possible,
Ring buffer for voice, video and monitoring tables
Write to. Image object ID and multiple images
Correspondence with the image expansion means is taken. In the example,
Shared memo of young ring buffer number from object ID
(5-3). Of the written buffer
The write pointer is updated (5-4). Monitoring table
Monitoring thread after writing video and audio information for one
The counter of the control semaphore is advanced (5-5). this
As described above, the monitoring thread is controlled by the DEMUX. FIG. 7 shows the operation of the monitoring thread 37.
FIG. Read monitoring table and read
The pointer is advanced (6-1). Overloaded objects
Check the priority of
Check (6-2). The contents of the monitoring table are
Give it to Red (6-3). Monitoring tape from DEMUX
Wait for creation of data for one bull (6-4). Processing excellence
Decode the frame numbers of the images to be displayed in descending order of priority.
(6-5) Write the current time and display
Compare the times, and if it is not in time,
Skip only PB frames without skipping
(6-6). Allow execution of the corresponding decoding process
(6-7), and waits until the processing is completed (6-8). FIG. 8 shows the operation of the decoding process 36.
FIG. From the monitoring thread 37
Wait for permission (7-1). Change the state of the input image
Check the image serial number, input frame
Checks whether the image should be skipped (7-2). De
Until image data to be coded accumulates in the ring buffer
Wait (7-3). The image thread specified by the monitoring thread
If there is no image data corresponding to the real number, decode
Is skipped and the read pointer is advanced (7-4). Entering
If not skip skipping image, execute decoding process
Then, the read pointer is advanced (7-5). Decoding result
The result is output (7-6), and the processing ends in the monitoring thread 37.
Is notified (7-7). The same process (may be a thread).
If the hardware is a different processor)
When decompressing image objects of any type,
The frame number of the image decompressed in the past in the process 36
By associating and managing images before being expanded
Need to create and use many processes at the same time
Is lost (at least, only information about the previous frame
Is also good. Also, different types such as I, P, B
If frame images exist, the order and output to be managed
Since the order to be performed is different, the decoding process 36
Such management is necessary). FIG. 9 shows the operation of the image synthesizing thread 39.
FIG. The monitoring thread 37
Wait for a cable (8-1). Check the priority of the image to process.
(8-2). Decoding results in descending order of priority
Wait for an image (8-3). Combining images according to the display position
(8-4). Write the synthesis result to the synthesis buffer 41
(8-5). The selection of image information to be displayed is
This can be performed by image expansion means or image synthesis means.
Skip image object IDs that should not be displayed
The decompression result is not output to the image synthesis means.
Need to be notified. Audio to be played for audio
Selection of information is performed by voice expansion means or voice synthesis means
be able to. FIG. 10 shows the operation of the display monitoring thread 42.
FIG. That the composite image is written
Wait (9-1). Start display if this is the first display
(9-2), and the time to display
Manage correspondence. If the display time has not been reached,
Wait for the time that is not done and delay the display of the composite image
(9-3). Referring to FIG. 11, a user of the image synthesizing apparatus of the present invention will be described.
The user interface will be described. In the example of FIG. 11, a foreground image is added to a background image.
Buildings that are synthesized and are located far away are semi-transparent at a synthesis ratio of 0.5
The image is clearly synthesized. As shown in FIG.
The image to be processed need not be a two-dimensional image. 3D drawing in foreground
Helicopter and balloon are two-dimensional images
It is combined with the scenery. The helicopter and balloon in the foreground
Need not always be a three-dimensional image. Far away
If located (the size displayed as two-dimensional on the screen
Should be defined in For example, 20 dots x 20 dots
If it is smaller than the size of the object, the object is far away
If it is defined, it is expressed in two dimensions,
If they are located in different places, they may be expressed in three dimensions. Also,
Map to 3D image wireframe model
The moving image may be not only a still image but also a moving image.
As for the image quality, the image quality in the central part is high and goes to the peripheral part
Moderately rough, giving priority to the information required by the user
Can be selected and transmitted (in this way, the image
Responds by changing the image quality according to the position where
Can be expected to improve). In the case of a three-dimensional image,
Lower priority for images displayed far away and closer
The priority of the image to be performed may be set higher. In addition, image quality
By changing the quantization step
realizable. FIG. 12 is a graph showing the variation in the capability of the receiving terminal.
FIG. 4 is a diagram for explaining a method of performing image transmission. Next
In addition, the number of transmitted images increases,
In order to prevent overloading of the process,
Then, how to manage and control is described. For example, Her
MPEG2-based video implemented in hardware
In an on-demand system, the transmitting terminal is
Device performance (for example, the method and size of image compression,
Communication protocol) before transmitting or receiving video information.
Check with each other. For this reason, the transmitting terminal
Since the processing capacity at the receiving end is almost fixed,
There is no need to monitor the signal status and playback status sequentially. On the other hand, image compression and decompression are performed by hardware.
If realizing, the number of images that can be compressed and decompressed by the terminal
Is fixed. However, compression and decompression of images with software
If you want to achieve long, you can compress and decompress the image on the terminal
The number can be dynamically changed. In addition, multi
When compressing and decompressing images in a task environment,
And quantization parameters for image compression,
Image (intra-frame or inter-frame encoding, shooting
Greatly affected by the
Image size that can be processed (compressed, decompressed), and can be processed simultaneously
The number of images changes over time. Also sent along with this
In the receiving terminal, the receiving status of the receiving terminal (for example,
Receive buffer capacity, video playback priority,
Image compression method (image compression method,
Whether image compression, quantization step, compression priority, compression
Priority when receiving terminal is overloaded
If you do not consider the decision of the
It comes down to failure. For example, as shown in FIG.
If the receiving buffer capacity of the side terminal exceeds 80%,
Notify the receiving side that the receive buffer is about to overflow.
Knows the image compression method (for example,
To reduce the amount of compressed images sent)
Whether image compression is performed (image compression and transmission
Change the compression priority (the process to be compressed
If there are, reduce the priority for compression and
The compressed image transmission amount), and change the image size (C
Change the size to be compressed from IF to QCIF
To reduce the amount of compressed images sent) and change the quantization step
(Reduce the amount of compressed images sent by changing the image quality)
To limit the amount of data to be transmitted, and to adjust the number of frames
(Reduce the number of frames to be processed), the receiving terminal is overloaded
Select and combine methods for determining the priority of time as appropriate
carry out. As a result, the receiving buffer of the receiving terminal is
Avoid bar flow. Similarly, if the capacity of the receiving buffer on the receiving side is 2
If the value falls below 0%, the receiving terminal receives the data from the receiving terminal.
Notifies that the buffer is about to underflow
Then, in the opposite way to the above, the image compression
Method, image compression, image compression priority, image size
, Quantization step and number of frames as appropriate
It will be implemented together. How to increase the amount of transmission in this way
By performing this, the reception buffer of the receiving terminal is unencrypted.
Underflow can be avoided. In addition to monitoring the status of the reception buffer,
The playback capability on the local terminal is limited,
If there is more than one, the image to be played back preferentially on the receiving terminal
Is explicitly determined by the user or the terminal
It is necessary to automatically determine the image to be generated (the
Which image should be played preferentially by the user?
It must be registered in the receiving terminal as a rule. example
For example, if the image size is small,
Images displayed as images have slow playback intervals
It may be.) For example, the load (
For example, the CPU occupation time required for playback)
By notifying the end, it can be easily realized. The reproduction load of the receiving terminal depends on the processing capability of the terminal.
If the power exceeds 80%, the receiving terminal is overloaded.
Is notified to the sender, and the sender
However, in the same manner as described above, the negative
To reduce the load, the image compression method (for example, MPE
Change from G1 to run length to reduce processing amount),
Whether image compression is performed (temporarily suspend image transmission
Change compression priority (for images with low importance)
In order to reduce the priority for compression,
Compress and send images with priority), change image size
(Change the size to be compressed from CIF to QCIF
To reduce the load on the playback side), and change the quantization step
(Reduce the amount of compressed images sent by changing the image quality)
Method, method of adjusting the number of frames, priority of overload processing
Select or combine methods based on degree
To reduce the amount of processing at the receiving terminal.
To reduce. On the other hand, when the load is 20% of the processing capability of the receiving terminal.
%, There is a margin in the processing capacity of the receiving terminal.
In some cases, the sending terminal
, The image compression method, the presence or absence of image compression, the priority of image compression
Image quality, image size, quantization step and number of frames.
High quality by selecting, combining and implementing
Then, an image with a short frame interval is transmitted to the receiving terminal.
This enables image transmission utilizing the capabilities of the receiving terminal.
It will work. Finally, a method of knowing the processing status of the receiving terminal
As a response to the reception confirmation from the receiving image combining device
You can know by time. For example, the sending terminal
When image data is sent from the
The terminal receives image data, decodes, combines,
When a response is made to the transmitting terminal that the
For example, if the response time of the
In this case, the response time becomes
5 seconds, etc. (The normal value is
It may be measured once or periodically during communication.
Yes, or the user may instruct. In addition, measurement of response time
May be performed periodically, or the terminal load or previous response
The measurement interval may be changed in relation to the time result.
No). Due to this change in response time, the above-described image compression
Method, image compression, image compression priority, image size
And quantization steps as appropriate.
This reduces the load on the receiving terminal.
Therefore, the response time can be reduced (see FIG. 16).
Case 1). Playback time or decryption at the receiving terminal
The same processing as described above may be performed by receiving the time. A method considering the state of the terminal on the receiving side and
Then, the capacity of the receiving buffer of the receiving
How to measure the load on the receiving terminal and the response time of the receiving terminal
Method is not used independently,
May be used in combination.
Method is applicable). In addition, the priority information is
Information about images and sounds processed based on
Received when image stream and audio stream exist
The image and audio streams actually processed by the
Stream, and the played image stream is
Information about the number of frames)
By transmitting to the transmission destination, the transmission side
The amount that image data transmission exceeds the processing amount of the receiving terminal
Can be prevented beforehand (the case of FIG. 16).
2 See, to know about image data actually processed
Information on the quantization parameters, image size, etc. on the transmitting side
The amount can be adjusted. Note that in this example,
Processing feedback is returned on a frame-by-frame basis.
As described, for example, 263 is like GOB
Image units that can be handled independently of each other). Less than
The above method can be applied to speech as well. FIG. 13 shows an image pressure according to an embodiment of the present invention.
It is a figure explaining a compression device. Note that the present embodiment
Describes an example for images, but for audio compression
Also applicable to: In the example of FIG.
The quantization step can be changed for each
The receiving status at the receiving terminal by controlling the stage 1207
If the situation changes, follow the quantization step and change
To reduce the increase in the amount of generated compressed images.
Is to try. The image compression device in FIG.
Step management that manages information about quantization steps
Unit 1201 manages the control state of image input unit 1207
Image input management unit 1202, the receiving buffer of the receiving terminal device.
Other terminal control request management unit 1203 for monitoring the status of the
Operation management unit 120 that records and manages the transition of control over time
4. Image compression unit 1205, which is a means for performing image compression,
An output unit 1206 that outputs a compression result to a communication path or a storage device;
Image input means 1207 for inputting an image and each of these units
Image processing decision control system that manages and controls
It consists of a stage 1208. The image compression method includes JPEG,
MPEG1 / 2, H.264. 261, H .; Standard like 263
Or wavelet or flux
It may be a non-standardized method such as a barrel.
Even if the image input means 1207 is a camera,
It can be a recording device like a optical disc
No. The method of using this image compression apparatus is as follows.
When the image input unit 1207 is a camera,
When the camera of the sending terminal is operated or
When camera operation is performed, image quality changes greatly
In addition, the transmitted coding amount varies. For example, a camera
When the contrast is increased, the image becomes easier to see, but
The amount of coding to be output increases. So, the direction of the contrast
To reduce the amount of coding as described above with the above
Image compression method, presence or absence of image compression, priority of image compression
Image quality, image size, quantization step and number of frames.
The amount of coding by selecting, combining and implementing
Can be suppressed. The camera operation described here refers to the camera operation.
Direction (pan, tilt, zoom), contra
Strike, focus, camera position (e.g.
The camera downwards when shooting a person
Is horizontal). Change the image compression method
If you point the camera downwards,
It is determined that the image is being shot, and the image is run-length.
Transmit the image and if the camera is facing horizontally,
Assuming that the state of an object's face is being photographed, 261
There is a method of transmitting an image by photographing. This
Transmission of unnecessary information can be reduced.
You. Also, if there are a plurality of cameras and a plurality of
Communication when it is necessary to transmit the video obtained from
If the capacity is limited, the camera
The image quality and the number of frames of the
I will reduce the image quality and the number of frames of the camera I do not see
There is a method. Image obtained from the camera of interest
By manipulating the image quality and the number of frames, the amount of information
From the cameras that are not paying attention accordingly
It is necessary to adjust the amount of generated information by limiting the
You. One way to adjust the amount of information that
To adjust noise, quantization step value, number of frames, etc.
Law. In addition, wide-field image using multiple cameras
An example in the case of creating a file will be described later with reference to FIG.
You. FIG. 14 is managed by the operation management unit 1204.
It is an example of information. In the example of FIG.
Control, control request of other terminals, quantization step, not shown
The number of frames is managed. These management information
The receiving buffer of the receiving terminal overflows
The relationship between the quantization step and the camera operation so that
By recording and managing it as history information,
Restrictions can be placed on the user. Also, quantization
Automatically change steps, image size, number of frames, etc.
The receiving terminal's receiving buffer associated with the camera operation.
To prevent overflow and underflow
Can be. FIG. 15 shows the image compression apparatus having a wide-field image.
Here is an example in which the present invention is applied to the purpose of creating. In the example of FIG.
Images input from a plurality of cameras are captured by the input unit 1407.
Get. The obtained plural images are transmitted to the receiving terminal 1408 side.
When joining (combining) seamlessly with the
If terminal 8 is overloaded, terminal will fail, so prevent it
Therefore, processing at the time of overload in the receiving terminal 1408 is performed.
The priority defining the order of the images to be added is added to the images.
This prevents the receiving terminal 1408 from being overloaded.
Can be prevented. The image compression apparatus shown in FIG.
Input unit 1407 provided with an input unit 14 (N units)
Add priority to each image obtained in 07
The priority determination control unit 1401 to be performed and the user
Command to operate the camera
The operation history management unit 1402 for managing the history and the image quality of the image
An image quality control unit 1403 for controlling
An image synthesizing unit 1404 that synthesizes images based on priorities (excellent
It is not necessary to combine low-priority images.)
Output unit 1405 to output and compression to control each unit
And a control unit 1406. The output unit 1405
It is connected to the receiving terminal 1408 via a communication path. The output destination of the output unit 1405 is a recording device.
Or a communication path. Also, be sure to combine images
It does not need to be performed at the transmitting terminal. Priority is added
The transmitted image is transmitted to the receiving terminal through the
You may synthesize | combine at the terminal side. Sending the obtained images
When combining on the receiving terminal and playing back on the receiving terminal,
The required image is needed at the receiving terminal on the sending side (for display)
Combine images in descending order of priority, and use the transmission path to combine images.
Transmit to the receiving terminal device. As a method of adding the priority, the user instructs
Image obtained by a camera that has been instructed
From the image obtained in the order of higher priority, higher image quality (
For example, increase the number of frames and increase the resolution).
(Necessary to make high priority images high quality)
Not necessary). As a result, the degree of user attention is high
Images are displayed with high quality and priority. Attached to the image
Image transmission from the sending terminal was controlled according to the priority
Control the image expansion and display on the receiving terminal.
More responsiveness of the terminal for the user.
Wear. In addition, priority, high quality image, frame
In order from the image with the largest number of images,
And gradually reduce the priority and image quality (priority management
The processing may be managed by the transmitting terminal or the receiving terminal.
May be managed at the end). As a method of determining priority
Is not necessarily based on the operation history of the camera.
Good. As mentioned earlier, the local
-Priority may be determined based on decoding time.
Images with high priority, high image quality, and many frames
In order from the image, the number of times of processing is
May be defined. In addition, regarding audio
Also, microphones are provided for each of a plurality of cameras to enable audio compression.
By controlling the absence, the image in the direction of the user's attention
Can be synthesized only. As described above, the transmitting terminal and the receiving terminal
Quantization step with reference to the response time between the
Or the number of frames may be determined. Also, if the receiving terminal
For images processed based on priority information at the time of load
Sending information by sending it to the destination via a communication channel
Processing of image data transmission from receiver to receiver terminal by receiver terminal
It can be prevented from exceeding the amount
You. Also, transmit the frame skip status at the receiving terminal
The data amount according to the status.
Can be adjusted. Further, the image is transmitted by a retransmission method.
The voice is transmitted by a transmission method that does not perform retransmission, and
The terminal determines the number of image retransmissions, the error rate of the received
Transmits any information about the drop rate to the sender terminal
Configuration. Then the image compression method at the sending terminal
Expression, quantization step value, number of frames, image to be compressed
The size of the image and whether or not to use image compression
And reduce the delay of audio transmission without disturbing the image.
Control can be performed. For example, TCP / IP
In the communication used, image transmission is TCP, and audio transmission is
This can be achieved by using UDP (video and audio are physically
It may or may not be on the same transmission path). In addition,
The communication method is not limited to TCP / IP. This method
When transmitting multiple video and audio simultaneously,
Define the discard rate and error rate for each audio
The compression method and the transmission method may be controlled. Finally, usually, an analog telephone line was used.
Low bit rate image transmission and large fluctuations in image content
Image, large block noises and cracks occur in the image
I do. In such a case, the image quality can be maintained only by the compression process.
One is difficult. Therefore, monitor the output side of the image
Filter that transmits only the signals of the
Low-pass filter or physical polarization filter
Filter makes the image look blurry
However, you can get an image that does not bother noise or
You. As apparent from the above description, the present invention
The invention is intended for decoding and synthesizing a plurality of video and audio simultaneously.
The processing amount based on the priority according to the load status of the terminal.
Has the advantage that it can be controlled. Further, according to the present invention, a plurality of
There is an advantage that video and audio can be synthesized.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a schematic configuration diagram of an image decoding / encoding device according to an embodiment of the present invention. FIG. 2 is a schematic configuration diagram of a video / audio decoding / encoding device showing another example of the embodiment. FIG. 3 is a diagram illustrating an example of a case where information regarding priority is added in a communication and recording format. FIG. 4 is a diagram illustrating an example in which the configuration of the present invention is configured by software. FIG. 5 is a diagram illustrating a structure of information. FIG. 6 is a diagram illustrating the operation of a DEMUX thread. FIG. 7 is a diagram illustrating an operation of a monitoring thread. FIG. 8 is a diagram illustrating the operation of a decoding process. FIG. 9 is a diagram illustrating an operation of an image composition thread. FIG. 10 is a diagram illustrating an operation of a display monitoring thread. FIG. 11 is a diagram illustrating a user interface of the image composition device. FIG. 12 is a diagram illustrating a method of performing image transmission according to a change in the capability of the receiving terminal. FIG. 13 is a diagram illustrating an image compression device according to an embodiment of the present invention. FIG. 14 is a diagram illustrating information managed by an operation management unit. FIG. 15 is a diagram illustrating an image compression device when a wide-field image is created. FIG. 16 is a diagram illustrating a response situation between a transmitting terminal and a receiving terminal. [Description of Signs] 11 Reception management unit 12 Separation unit 13 Transmission management unit 14 Priority determination unit 17 Time information management unit 18 Image decompression unit 19 Image synthesis unit 20 Voice decompression unit 21 Voice synthesis unit 31 DEMUX thread 36 Decoding process 37 Monitoring thread 39 Image composition thread 42 Display monitoring thread 1204 Operation management section 1205 Image compression section 1208 Image processing decision control section 1401 Priority decision control section 1402 Operation history management section 1404 Image composition section 1407 Input section

   ────────────────────────────────────────────────── ─── Continuation of front page    F term (reference) 5C059 KK35 MA00 MA05 MA14 MA24                       MA43 MC11 PP01 PP05 PP06                       PP07 PP19 SS07 SS08 TA07                       TA17 TA46 TC20 TC25 UA02                       UA05                 5J064 AA00 BA00 BB10 BB12 BC02                       BC29 BD02

Claims (1)

Claims: 1. An image input means for manually inputting an image, an image input management means for managing a control state of the image input means, and another terminal for managing a reception state of a reception terminal. Control request management means, at least according to the reception status of the managed receiving terminal or the control state of the image input means,
Encoding processing determining means for determining an image encoding method, image encoding means for encoding the input image according to the result of the encoding processing determining means, and output means for outputting the encoded image. A real-time image encoding device, comprising: