JP4521931B2 - Image coding apparatus, image decoding apparatus, control method therefor, and image communication system - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an encoding apparatus and method for encoding, for example, a digital moving image signal and the like, and a decoding apparatus and method for an encoded image encoded by the encoding apparatus.
[0002]
[Prior art]
A conventional encoding and decoding system will be described with reference to FIGS.
[0003]
In FIG. 7, each frame is composed of one or more mutually independent object images. In FIG. 7, one frame is composed of N object images.
[0004]
In the example shown in FIG. 7, the object images constituting each frame are input to the encoder group 701 under the control of the encoding control unit 708, and the dedicated encoders 1 to N are used for each object image. It is programmed so that a predetermined encoding process is performed.
[0005]
The encoding format of each of the encoders 701-1 to 701 -N constituting the coder group 701 is arbitrary, but the compression format in the encoders 701-1 to 701 -N is assumed to be MPEG4 or the like, and its internal The structure is as shown in FIG.
[0006]
The encoder shown in FIG. 8 is representative of an inter-frame predictive encoding method typified by MPEG. An object image input to the encoder group 701 is received by a motion compensation unit 801 in FIG. A motion vector is detected, coding mode (inter-P or B / intra I) determination is performed using the accuracy of the detected motion vector (prediction error magnitude, etc.) as a parameter, and the coding mode is determined to be interframe. If the prediction error is determined to be an intra frame based on the detected motion vector, each pixel value of the input object image is in units of a certain size (block) in the horizontal and vertical directions. Input to the orthogonal transform unit 702.
[0007]
The block-unit prediction error (or pixel value) input to the orthogonal transform unit 702 is converted into an orthogonal transform coefficient and input to the quantization unit 803.
[0008]
The quantization unit 803 performs data compression by dividing the orthogonal transform coefficient by a predetermined quantization coefficient and replacing values other than the significant coefficient (integer) with zero.
[0009]
Each object image (encoded image data) that has been subjected to a predetermined encoding process in the encoder group 701 is temporarily stored in the buffer memory 702 under the control of the encoding control unit 708 and then to the multiplexer 703. Input, predetermined multiplexing processing is performed for each frame, and output to a transmission path, storage media, or the like. The multiplexing process is programmed in the encoding control unit 708 so as to comply with a protocol suitable for the output destination.
[0010]
The quantization coefficient is programmed to be controlled (changed) as needed under the control of the encoding control unit 708 using the use efficiency of the buffer memory 702 or an arbitrarily set code amount as a parameter. FIG. 9 shows the procedure. First, the free capacity of the output buffer is detected and it is determined whether or not it is greater than or equal to a predetermined value. If it is greater than or equal to the predetermined value, normal encoding is performed, otherwise the quantization table is changed and encoded.
[0011]
The above processing is sequential processing with respect to the input image signal (object), and processing from each processing block 701 to 703 takes into account external control or requests, etc., except for the encoding control unit 708. Not.
[0012]
Next, the output of the multiplexer 703 or the encoded image data subjected to the same encoding process as the output of the multiplexer 703 is temporarily stored in the buffer memory 704 under the control of the decoding control unit 709, and then the splitter 705. Are separated into units of objects constituting the frame.
[0013]
The encoded object image separated by the splitter 705 is input to the decoder group 706 under the control of the decoding control unit 709 and subjected to a predetermined decoding process.
[0014]
Each object image decoded by the decoder group 706 is input to the video memory 707.
[0015]
Each decoded object image input to the video memory 707 is arranged (stored) to form a predetermined frame image under the control of the decoding control unit 709.
[0016]
The encoding and decoding processes described above may be a series of real-time processes or non-real-time processes in which encoding and decoding are performed independently.
[0017]
Therefore, in a system in which encoding / decoding proceeds simultaneously in real time, if the amount of data to be decoded (code amount) exceeds the capacity of the decoder, as shown in the flowchart of FIG. When it is detected that the value is less than a certain value, the coded data in the frame is discarded in units of objects or in units of frames until the free space in the buffer 704 becomes equal to or greater than the certain value, thereby avoiding the failure of the decoder. It is customary to do this.
[0018]
Here, the free capacity of the buffer being less than a certain value means that the amount of code input with respect to the decoding performance of the decoder is excessive.
[0019]
As described above, in the conventional encoding and decoding system, when the amount of code to be decoded exceeds the capacity of the decoder, object thinning and frame thinning are performed regardless of the intention of the coding side (or user). Therefore, it is assumed that an important object as the meaning of the image is missing or the whole scene is missing.
[0020]
The present invention has been made in view of the above-described conventional example. A priority is assigned to each image object, and an image object having a high priority is decoded by encoding and decoding the image according to the priority. It is an object of the present invention to provide an image encoding device, an image decoding device, a control method thereof, and an image communication system that are prevented from being lost due to time thinning.
[0021]
[Means for Solving the Problems]
In order to achieve the above object, the present invention comprises the following arrangement.
[0022]
An image encoding device that encodes an image frame having an object image as a component,
Encoding means for encoding an object image;
Receiving means for receiving a request from an external device;
Determining means for determining the type of request from the external device;
Wherein it is determined by the determining means requests, if the designated priority for encoding and decoding of the object image, and assigning means for assigning the priority to the object image,
And means for multiplexing the encoded image data corresponding to said priority given to coded object image and the object image by the encoding means to said image,
Said encoding means, when said judged by determining means requests an overemphasis request code amount specified and the object of the priority, by controlling the code amount of each object image, priority is specified code amount of the object, and so that the coding of more than the code amount of the object priority is not specified,
It said applying means, wherein, when there is no request from the external device, the object images included in the frame, and automatically generated to impart the said priorities based on the occupancy area and its position.
[0023]
More preferably, the information processing apparatus further includes notification means for notifying whether or not the request can be approved to the external device when the request determined by the determination means is a designation of reduction or increase in encoding amount.
[0024]
More preferably, when the request determined by the determination unit is to specify a reduction or increase in the encoding amount , the adding unit sets the position and the occupation area of the object image included in the frame. Based on this, the priority is automatically generated and assigned.
[0034]
DETAILED DESCRIPTION OF THE INVENTION
The image encoding / combining system according to the first embodiment of the present invention will be described with reference to FIGS.
[0035]
<Encoding device>
FIG. 1 is a block diagram of an encoding apparatus (encoder) 1 according to the present invention.
[0036]
The input data of the encoder (FIG. 1) in the first embodiment of the present invention is digital moving image data. The input digital moving image data is composed of a plurality of object images in which each frame is independent of each other.
[0037]
The object images constituting each frame are input to the encoder group 101 in units of frames and subjected to a predetermined encoding (compression) process.
[0038]
In parallel with the encoding processing of the image object, the encoding control unit 105 monitors the availability of the buffer memory 102 that accumulates the output of the encoder group, and always in a range where the buffer 102 does not overflow or underflow. The quantization coefficient (compression rate) is controlled for each encoder constituting the encoder group 101, that is, for each object image constituting the frame, so as to achieve the maximum utilization efficiency.
[0039]
The operation of the encoding control unit 105 will be described using the flowchart of FIG. In the encoding process, the encoding control unit 105 first determines whether a request has been received from the decoder side or from the operator of the encoding / decoding system (step S301). If a request is received, the content is determined (step S302). If the request is a code amount reduction request, the process proceeds to step S305 to determine whether the request can be approved. If approval is possible, an approval signal indicating approval is sent to the decoder side.
[0040]
After sending the approval signal to the decoder, the encoding control unit 105 applies the quantum unit to each of the encoders constituting the encoder group 101 in units of frames or an appropriate unit of encoding processing such as a macro block. A normal encoding process is performed by controlling the encoding coefficient so that n (n is an integer) steps larger than the pre-encoded frame or the pre-encoded macro block is selected and encoded (step S306). ). Here, the value of n is determined according to how much the code amount is reduced. The value of n is determined according to the value of the code amount, for example, if there is a code amount reduction request from the decoder side. Then, the priority of each image object is set according to the procedure shown in FIG. 4 (step S307).
[0041]
If it is determined in step S305 that the code amount reduction request cannot be approved, normal encoding processing is performed (step S303), and the priority of each image object is determined by the procedure of FIG. 4 (step S304). ).
[0042]
On the other hand, if it is determined in step S302 that the request from the decoder side is a request for priority designation and code amount weighting for a specific object, an object image in which the code amount of the priority designated object image is not designated. The encoding process is performed by controlling the code amount of the specified object deviation (step S310) so that the number of the objects is larger than that in step S310. Then, the priority of the object image is designated or changed based on the priority designation (step S311). Note that an object image without a priority designation request is set in accordance with a priority automatic setting means described later. In this case, as a result, the image quality of the designated object is improved at the expense of the image quality of the objects excluding the designated object.
[0043]
Even when the distributed code amount allocation is performed for the specified object image, there are restrictions on the transmission path capacity and the processing capacity of the decoder. Therefore, as a whole, the capacity of the buffer 102 is exceeded, the allowable code quantity of the decoder is exceeded, or the transmission path The code amount control is performed within a range that does not cause the failure of excess capacity.
[0044]
If it is determined in step S302 that the request is a priority only request for the specified object image, the encoding process is performed as usual (step S308), and the priority setting for each object image constituting the frame is set as a request. The setting is made accordingly (step S309).
[0045]
In this way, the code amount is controlled according to the request, and a priority is given to the encoded object.
[0046]
The object image that has been subjected to predetermined encoding processing (compression) by the encoder group 101 is temporarily stored in the buffer memory 102. The object images stored in the buffer memory 102 are sequentially read for each frame and input to the priority order setting unit 103.
[0047]
The priority order setting unit 103 performs priority order setting (numbering) for each object image constituting each frame under the control of the encoding control unit 105. The priority order set here is the priority order given to each object in the procedure of FIG.
[0048]
The priority order setting in the priority order setting unit 103 is a priority order for each object image according to the specification of the request when a request via the user interface on the encoder side or a request from the reception (decoder) side is detected. If there is no request, the priority order for each object image is similarly set by automatic setting (default) means using an arbitrary function. In this embodiment, the computer 106 shown in FIG. 1 is used as the encoder user interface.
[0049]
Each object image constituting the frame thus set with the priority order is input to the multiplexing unit (multiplexer) 104 together with the corresponding priority order, and a stream format suitable for the transmission path and the storage medium as the output destination. Are multiplexed and output. When multiplexing, a header including mapping information such as the position (coordinates) of each object image constituting the encoded frame and the context between the object images is added and stored for each frame.
[0050]
Next, an example of automatic priority setting processing for each object image constituting a moving image frame will be described with reference to FIG.
[0051]
When automatic setting is selected for the priority of each object image that composes the frame,
Step S401: The area ratio occupied by each object image or the occupation ratio with respect to the entire frame is calculated. Further, the position of each object image is detected.
[0052]
Step S402: Classify each object by an arbitrary method using the obtained occupancy and position as parameters.
[0053]
Step S403: Priorities are set according to the class of each object calculated in step S402 (class 1 is priority level 1).
[0054]
The following method can be considered as an example of prioritization according to the above procedure. As a parameter,
(A) Object position: The weight of the center of the frame is set to 100, and the weight is reduced in inverse proportion to the distance from the center in the radial direction.
(B) Occupancy rate: The object occupancy rate with respect to the area of the entire frame is used, and c = a + b, and the object having the largest value of c is class 1, that is, priority level 1.
[0055]
In this embodiment, the maximum number of object images constituting a frame and the number of encoders constituting the encoder group 101 are set to the same number, and encoding processing for one frame is performed in parallel at a time. However, if the processing time satisfies the required specifications, the object images constituting each frame are repeatedly encoded with a single or less than the maximum number of the object images. You may take the form which makes it.
[0056]
<Decoding device>
Next, the configuration of a decoding device (decoder) for decoding a moving image encoded by the above-described encoding device will be described with reference to FIGS.
[0057]
FIG. 2 is a block diagram of the decoding device (decoder) 2. FIG. 11 is a flowchart of a control procedure by the decoding control unit 206.
[0058]
In FIG. 2, when a bit stream (encoded data) is input to the decoding device 2, the encoded data is first temporarily stored in the buffer memory 201. The procedure of FIG. 11 shows the subsequent procedure.
[0059]
Next, the decoding control unit 206 reads the header information of the encoded data stored in the buffer memory 201, and from there, the position (coordinates) of each object image constituting the encoded frame and between each object image Mapping information such as the context is extracted (step S1101).
[0060]
The encoded data stored in the buffer memory 201 is sequentially read according to the mapping information, input to the splitter 202, and separated for each data attribute such as video, audio, and other various data (step S1102). ).
[0061]
The encoded object image separated by the splitter 202 is input to the priority detection unit 203, and the priority is detected for each frame (step S1103). The detected priority order is sent to the decoding control unit 206 for each frame.
[0062]
The encoded image data that has passed through the priority order detection unit 203 is input to the decoder group 204.
[0063]
The encoded image data input to the decoder group 204 is input to each of the decoders 1 to N in units of object images constituting the frame for each frame, and the object images constituting the frame are decoded in parallel. Is done.
[0064]
At this time, it is determined whether the code amount exceeds the processing capability (step S1104), and the code amount of the encoded image data input to the decoder group 204 exceeds the processing capability of the decoder group 204, or the code There is a risk that the code amount of one or more object images constituting the decoded image data exceeds the processing capability of each decoder constituting the decoder group 204, or the buffer memory 201 and the video memory 205 overflow. Is detected by the decoding control unit 206, the decoding control unit 206 transmits a code amount reduction request signal to an encoded image data source such as an encoding device (step S1106). If the encoded image data source side accepts the code amount reduction request (an approval signal is returned), the decoding control unit 206 continues the normal decoding process unless the encoding process fails (step S1109).
[0065]
On the other hand, when the code amount reduction request is rejected to the encoded image data source side (a non-approval signal is returned) (step S1107-NO), or when reproducing from the storage medium (step S1105-YES), The decoding control unit 206 performs thinning processing of the object images constituting the frames in order of the set priority (step S1108), and performs decoding processing (step S1109).
[0066]
Here, when the encoder and the decoder are connected by a communication line or the like and the line capacity is sufficient, the decoding control unit 206 detects the risk of underflow with respect to the buffer memory 201. In this case, the request signal for increasing the code amount can be transmitted in the same manner as in the case of overflow.
[0067]
Each object image that has been decoded is controlled by the decoding control unit 206, and the frame of each object image that has been decoded is extracted from the header information of the encoded frame based on the mapping information regarding all object images that constitute the frame. Are written in the video memory 205 in a form reflecting the positional relationship and the front-rear relationship.
[0068]
The decoded frames written in the video memory 205 are sequentially output to a monitor or the like.
[0069]
In the decryption system according to the present invention, a user interface (UF) connected to the decryption control unit 206 as shown in the computer 207 of FIG. 2 is provided.
[0070]
When a user who is viewing an image decoded on the decoding side designates a desired object image with a mouse (pointer) and selects priority designation or image quality improvement from a pull-down menu or the like, a UF is selected. A request signal corresponding to is transmitted to the encoding side. As described in the description of the encoding device, the encoding side that has received the request changes the setting of the object degree of each object image or weights the code amount for the specified object image as appropriate according to the request. Perform processing such as.
[0071]
The processing blocks 202 to 204 in FIG. 2 can also be realized by causing a computer to execute a program.
[0072]
Here, it is assumed that the priority of each object image of the image composed of object images shown in FIG. 6 (1) is set in advance as shown in FIG. 6 (2). The operation of the decoding apparatus 2 when the user designates the priority of each object image as shown in FIG. 6 (3) for this image will be described with reference to the flowchart of FIG.
[0073]
Step S501: It is determined whether the priority is designated by the user.
[0074]
Step S502: If specified by the user, the priority information for the object image specified by the user is transmitted to the encoding side.
[0075]
Step S503: The presence / absence of a consent signal from the encoding side is detected for the information transmitted in step S502.
[0076]
Step S504: If the encoding side rejects priority designation or the decoder is stand-alone (package media playback, etc.), the decoder side is forced by the control of the decoding control unit 206. The priority of the object image specified automatically is changed to the specified value.
[0077]
Step S505: If the encoding side accepts the priority specification, the encoder sets the priority of the specified object image to the specified priority and performs the normal encoding process. The device also performs normal decoding.
[0078]
In the case of this example, the priority specified by the user may be only the object of priority 1 (the highest priority), and the priority of other objects in that case is re-set according to the default priority. Although it is set, there is no problem even if the encoder and decoder are reset by any means (algorithm or the like).
[0079]
Note that, in the present embodiment, it has been described that when a priority order of an object image, a coding amount reduction designation, or a weighting of a coding amount is designated, these are designated from the decoding device. You may specify on the side.
[0080]
In addition, when performing classification according to a request, the user is allowed to specify an object included in an image before encoding, the class of the object, or a weight for classification. When the class is specified, the specified object is classified into the specified class, and when the weight is specified, the specified weight is added to the above-described calculation formula example “c = b + a”. And weighting by multiplying or multiplying. This makes it possible to weight the object.
[0081]
Thus, an identifier indicating the priority order for each object, a function of assigning the identifier to the encoding side, and a function capable of changing the priority order (contents of the identifier) set on the encoding side on the decoding side And a back channel for information communication between the encoder and decoder, the priority of the object image can be set at the user level independently on the encoding side and the decoding side. On the encoding side, the intention of the encoding user on the object image can be reflected in the code amount distribution, so that it is possible to maintain the image quality of the image that is most desired to be displayed (or transmitted).
[0082]
Even on the decoding side, the priority of each object image is requested regardless of the setting on the encoding side, and the information is fed back to the encoding side so that the image quality of the object image of interest of the decoding side user can be kept good. I can do it.
[0083]
Even when there is no back channel or when the encoding side does not return acceptance of the request, the priority on the decoding side is less than the amount of code to be decoded and the object image is discarded. In such a situation, the object image with the lowest priority is discarded from the lowest object image. Therefore, an image with a high priority (high attention) is unlikely to be lost to the end.
[0084]
As described above, since the user interface is provided on the encoding side and the decoding side, the priority of each object image constituting the frame can be arbitrarily specified. By changing the code amount distribution of the object image, the image quality can be improved only for the desired object image.
[0085]
When encoding / decoding is real-time processing, the priority set on the encoding side is changed on the decoding side, and the desired image quality of the object image is maintained and improved on the decoding side. Is possible.
[0086]
Similarly, when encoding / decoding is not real-time processing, that is, when the decoder is stand-alone and the code amount exceeds the processing capability of the decoder, by specifying the priority on the decoding side, It is possible to make it difficult for an object image (highest priority) of a desired image to be lost (or dropped).
[0087]
[Other Embodiments]
Note that the present invention can be applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, and a printer), and a device (for example, a copying machine and a facsimile device) including a single device. You may apply to.
[0088]
Another object of the present invention is to provide a storage medium (or recording medium) on which a program code of software that realizes the functions of the above-described embodiments, for example, a program for executing the procedures of FIGS. 3 to 5 and 11 is recorded. It is also achieved by supplying to a system or apparatus and reading and executing the program code stored in the storage medium by the computer (or CPU or MPU) of the system or apparatus. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. A case where part or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing is also included.
[0089]
Furthermore, after the program code read from the storage medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is determined based on the instruction of the program code. The case where the CPU of the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing is also included.
[0090]
【The invention's effect】
As described above, according to the present invention, the priority of each object image constituting the image is specified in the encoder, or the code amount distribution of the desired object image is changed, and the desired object image is limited. Image quality can be improved.
[0091]
Also, when encoding / decoding is real-time processing, the priority set on the encoder side is changed on the decoder side, and the image quality of the desired object image is maintained and improved on the decoding side. It becomes possible.
[0092]
Even when encoding / decoding is not real-time processing, if the code amount exceeds the processing capability of the decoder, the priority is designated on the decoding side so that a desired image or an object in the image can be obtained. It is possible to make it difficult to lose an image.
[Brief description of the drawings]
FIG. 1 is a block diagram of an encoding system.
FIG. 2 is a block diagram of a decoding system.
FIG. 3 is a control flowchart of the encoding control device in the encoding system of FIG. 1;
4 is a flowchart of an object image priority automatic setting program in the encoding system of FIG. 1; FIG.
FIG. 5 is a processing flowchart according to priority designation in the decoding system of FIG. 2;
FIG. 6 is a diagram illustrating a priority setting example of each object image constituting a frame.
FIG. 7 is a block diagram of a conventional encoding / decoding system.
FIG. 8 is a block diagram showing the internal configuration of the encoder in FIG. 7;
FIG. 9 is a control flow of an encoding control device in a conventional encoding system.
FIG. 10 is a control flow of a decoding control apparatus in a conventional decoding system.
FIG. 11 is a control flowchart of the decoding control apparatus in FIG. 2;

Claims (5)

An image encoding device that encodes an image frame having an object image as a component,
Encoding means for encoding an object image;
Receiving means for receiving a request from an external device;
Determining means for determining the type of request from the external device;
Wherein it is determined by the determining means requests, if the designated priority for encoding and decoding of the object image, and assigning means for assigning the priority to the object image,
And means for multiplexing the encoded image data corresponding to said priority given to coded object image and the object image by the encoding means to said image,
Said encoding means, when said judged by determining means requests an overemphasis request code amount specified and the object of the priority, by controlling the code amount of each object image, priority is specified code amount of the object, and so that the coding of more than the code amount of the object priority is not specified,
It said applying means, characterized in the absence request from the external device, the object image included in the frame, that is automatically generated to impart the said priorities based on the occupancy area and its location An image encoding device.   The information processing apparatus according to claim 1, further comprising: a notification unit configured to notify the external device whether or not the request can be approved when the request determined by the determination unit is specified to reduce or increase an encoding amount. 2. The image encoding device according to 1.   When the request determined by the determination unit is to specify a reduction or increase in encoding amount, the assigning unit gives priority to the object image included in the frame based on the position and the occupation area. 2. The image encoding apparatus according to claim 1, wherein the ranking is automatically generated and assigned. An image encoding method for encoding an image frame having an object image as a component,
An encoding process for encoding the object image;
A receiving process for receiving a request from an external device;
A determination step of determining the type of request from the external device;
When the request determined in the determination step is a priority order for encoding and decoding the object image,
A step of applying said priority to said object image,
And a step of multiplexing said priority given to coded object image and the object image by the encoding process as coded image data corresponding to the image,
In the encoding step, when the request determined in the determination step is the priority designation and the code amount deviation request of the object, each object image is controlled , and the priority of the object for which the priority is designated code amount, and so that the coding of more than the code amount of the object priority is not specified,
The applying step, wherein if there is no request from the external device, the object images included in the frame, and automatically generated to impart the said priorities based on the occupancy area and its location An image encoding method.   A computer-readable storage medium storing a computer program for causing a computer to execute the image encoding method according to claim 4.

Images