JPH11205791A - Image coding method and image encoder - Google Patents

Abstract

PROBLEM TO BE SOLVED: To smooth the movement of a reproduced image at a receiving end by estimating the occupancy amount of a buffer memory at the next coding start timing and changing a target frame rate when the occupancy amount becomes larger than a threshold. SOLUTION: A target frame rate setting part 23 of a controlling part reads setting information made by an operating part 5, monitors occupancy amount of a buffer memory 3, estimates occupancy amount of the memory 3 at the next coding start timing based on setting information and the occupancy amount and switches a target frame rate when the occupancy amount exceeds the threshold. The part 23 sets a target frame rate based on the bit rate of a transmission line and an image mode, and designates a coded frame in a coding processing part 2 according to the target frame rate. Also, it changes a target frame rate according to monitoring results by an occupancy monitoring part 24. Also, a quantization step setting part 22 sets a quantization step of DCT coefficients in a quantization part 12 of the part 2.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image encoding method and an image encoding apparatus for encoding, decoding, and reproducing image information so that the motion of the reproduced image is smooth. 2. Description of the Related Art In a video conference system and a system for transmitting other images, there is a demand for compression-encoding, as well as a small deterioration in image quality of an image reproduced and decoded, and a smooth motion for a moving image.

[0002]

2. Description of the Related Art FIG. 16 is an explanatory view of a conventional image encoding apparatus, in which 51 is a frame memory, 52 is an encoding processing unit, 53 is a buffer memory, 54 is a control unit, and 55 is a DC.
A T operation unit, 56 is a quantization unit, and 57 is a variable length coding unit. A digitized input image signal from a television camera or the like is input to the frame memory 51.

[0003] The encoding processing unit 52 is already known in various configurations. For example, as shown in FIG.
, A DCT (discrete cosine transform) operation is performed for each block of 8 × 8 pixels, and the quantization unit 56
, And is converted into a code having a predetermined code length in accordance with the occurrence probability by the variable length coding unit 57, and is transmitted as an output image signal to the transmission path via the buffer memory 53 at a constant speed. Note that an encoding processing unit including a configuration for obtaining an inter-frame difference and a configuration for performing motion compensation is also known.

The control unit 54 monitors the occupancy of the buffer memory 53, reads the image signal corresponding to the frame from the frame memory 51 in the encoding processing unit 52, and controls the quantization unit in accordance with the occupancy. The quantization step at 56 is set to prevent underflow and overflow of the buffer memory 53.

FIG. 17 is a flow chart of a conventional image encoding process, in which the control unit 54 and the encoding unit 52 described above are used.
The control unit 54 performs a reading process of the set value set by the operation unit (L1), and the previously read set value is equal to the currently read set value. (L2), and if not equal, a bit rate setting process (L3), a frame rate setting process (L4), and a buffer threshold setting process (L5) are performed in accordance with the set values read this time. Perform sequentially. Then, it is determined whether the encoding of one frame is completed (L5). If the previous set value is equal to the current set value in step (L2), the process proceeds to step (L5).

When the encoding of one frame is completed, a buffer capacity reading process is performed (L7), an encoded frame setting process is performed (L8), and an encoding process is performed (L
9). If the encoding of one frame is not completed in step (L6), the process proceeds to step (L9) to perform the encoding process.

FIG. 18 is an explanatory diagram of a frame dropping process of a conventional example. FIG. 18A shows an image signal in frame units of frame numbers n, n + 1, n + 2,.
A case where encoding is performed on 15 frames of a 30-frame image signal per second using an even-numbered frame number as an encoding target frame, and the amount of information generated by encoding the frame of frame number n at time t1 during encoding processing is as follows. The amount of generated information is large for the frames of frame numbers n + 2 and n + 4 at times t2 and t3 corresponding to a small but large motion screen or the like, which increases the occupancy of the buffer memory 53 and exceeds the threshold. , The encoding process is in a wait state. For example, the frame number n +
The coding of the 6, n + 8, n + 10 frames is stopped. That is, a drop occurs.

Then, at time t4, the buffer memory 53
, The encoding starts from the frame of frame number n + 11 and the amount of generated information at that time is medium, so that the encoding of the frame of frame number n + 13 is stopped, and the next frame at time t5 Number n + 14
Since the amount of information generated for this frame is also medium, the occupation amount of the buffer memory 53 does not exceed the threshold, and the encoding of the next frame at time t6 is performed. Will be That is, encoding is performed in frame units indicated by the arrow of the frame shown in (b).

[0009]

In a conventional image encoding method or image encoding apparatus, a buffer memory 53 is used.
Buffer memory 5 so that overflow of
The occupation amount of No. 3 is monitored, and if the occupancy exceeds the threshold, the frame coding is stopped to change to the state of dropped frames. Such dropped frames occur discontinuously, and when the image is close to a still image, the image quality of the decoded reproduction image on the receiving side does not deteriorate, but in the case of a moving image, for example, As shown in FIG. 18, encoding of a number of consecutive frames is stopped. That is, there is a problem that the decoded reproduced image on the receiving side shows an unnatural movement because the dropped frames become discontinuous. SUMMARY OF THE INVENTION It is an object of the present invention to smoothly move a decoded and reproduced image on a receiving side.

[0010]

The image encoding method according to the present invention is: (1) an image encoding method for encoding an input image signal and transmitting the same through a buffer memory, wherein the occupancy of the buffer memory is And predicts the occupancy of the buffer memory at the next encoding start timing based on the occupancy and the setting information. When the predicted occupancy is out of the range of the threshold, the target frame rate is switched. And a process of performing frame-by-frame encoding in accordance with the target frame rate.

(2) The method may include a step of switching the target frame rate according to whether the set image quality mode is the motion emphasis mode or the image quality emphasis mode.

The method may further include (3) setting a video rate, and determining the target frame rate according to the video rate.

The image encoding apparatus according to the present invention comprises: (4) a frame memory 1 for adding an input image signal;
, A buffer memory 3, a control unit 4, and an operation unit 5, wherein the control unit 4 reads setting information from the operation unit 5 and monitors the occupancy of the buffer memory 3. Then, based on the setting information and the occupancy, the occupancy of the buffer memory 3 at the next encoding start timing is predicted, and when the predicted occupancy exceeds a threshold, the target frame for switching the target frame rate is calculated. It has a rate setting unit 23.

[0014]

FIG. 1 is an explanatory diagram of an image coding apparatus according to an embodiment of the present invention.
a, 1b, 1c, 1d are image signal storage areas in frame units, 2 is an encoding processing unit, 3 is a buffer memory, 4 is a control unit, 5 is an operation unit, 11 is a DCT operation unit, and 12 is a quantization unit. , 13 are a variable length encoding unit, 21 is a setting information reading unit,
22, a quantization step setting unit; 23, a target frame rate setting unit; and 24, an occupancy monitoring unit.

The input image signal is sequentially written into the storage areas 1a, 1b, 1c, 1d of the frame memory 1 on a frame basis. For example, the image signal of the next frame after writing the image signal in the storage area 1d is written in the storage area 1a, and the image signal of the next frame is written in the storage area 1b.

The encoding unit 2 includes a DCT operation unit 11
And a configuration including a quantization unit 12 and a variable-length encoding unit 13, and reads an image signal in frame units from a storage area of the frame memory 1 in accordance with an instruction of an encoded frame from the control unit 4. , A DCT operation is performed in a DCT operation unit 11, a DCT coefficient is quantized in a quantization unit 12, a variable length encoding is performed in a variable length encoding unit 13, and stored in the buffer memory 3 for transmission. The data is read out and transmitted at a constant speed according to the transmission speed of the path. In addition, the encoding processing unit 2
Can be configured to include processes such as inter-frame coding and motion compensation.

The control unit 4 also transmits the bit rate of the transmission path set by the operation unit 5 and the image quality mode to the setting information reading unit 21.
And the occupancy monitoring unit 24 monitors the occupancy of the buffer memory 3 and monitors whether it is outside the threshold range.
This threshold is a first threshold for monitoring overflow,
A second threshold value for monitoring underflow can be set.

The target frame setting section 23 sets a target frame rate based on the bit rate of the transmission path and the image quality mode, and instructs an encoded frame in the encoding processing section 2 according to the target frame rate. Further, the target frame rate is changed according to the monitoring result by the occupancy monitoring unit 24. The quantization step setting unit 22 sets the quantization step of the DCT coefficient in the quantization unit 12 of the encoding processing unit 2. Each unit of the control unit 4 described above can also be realized by calculation and processing functions by a processor or the like.

FIG. 2 is an explanatory diagram of the frame rate. The frame rate when encoding all 30 frames per second for frame numbers n, n + 1, n + 2,... Is 30 frames / sec. . Also, the frames indicated by the dotted lines are shown as unencoded frames,
The frame rate when encoding is performed every other frame is 15 frames / sec. When encoding is performed every two frames, the frame rate is 10 frames / sec.
The frame rate when encoding is performed every three frames is 7.5 frames / sec.

FIG. 3 is an explanatory diagram showing the relationship between the quantization step, generation information, image quality mode, and target frame rate.
(A) shows an outline of the relationship between the quantization step and the amount of generated information (buffer occupancy). When the quantization step is increased, that is, when the reproduced image is made coarse, the amount of generated information is reduced. If the quantization step is made small, that is, if the reproduced image has a high image quality, the amount of generated information increases.

FIG. 2B shows an outline of the relationship between the image quality mode and the target frame rate. In the case of a mode that emphasizes motion as the image quality mode, the target frame rate is increased to reproduce smooth motion. On the other hand, in the case of the mode in which the image quality is emphasized, for example, in the case where the image quality close to a still image is emphasized, the target frame rate may be reduced.

FIG. 4 is a diagram for explaining the operation of the first embodiment of the present invention. FIG. 4A shows frame numbers n, n + 1, n +.
.., For example, if the target frame rate is 15 frames / sec, encoding is performed every other frame as shown in the second row of FIG. At the target frame rate of 15 frames / sec, as shown in FIG.
1, the buffer occupancy at the next encoding start timing is small, and the time t of the next encoding frame number n + 2
If the buffer occupancy at the next encoding start timing in Step 2 is large and is expected to exceed the first threshold, the target frame rate is changed from 15 frames / sec to 10 frames / sec.

Thus, after the frame number n + 2, the frame of the frame number n + 5 at the time t3 is coded. If the amount of generated information of the frame number n + 5 is large, but the amount of generated information for the frame of the frame number n + 8 at the next time t4 is medium and the buffer occupancy does not exceed the first threshold, the target Since the frame rate is not changed, the encoding of the frame of the frame number n + 11 at the next time t5 is performed.

If the buffer occupancy at the start of encoding of the frame of frame number n + 14 at the next time t6 is less than the second threshold, the target frame rate is changed and the original frame rate is changed from 10 frames / sec. 15 frames / sec. Therefore, at the next time t7, the encoding of the frame of the frame number n + 16 is performed.

Therefore, as shown in FIG. 4B, the relationship between the coded frame indicated by the arrow and the uncoded frame indicated by the dotted line is the same even when the amount of generated information is the same as in the conventional example shown in FIG. Since the number of frames corresponding to dropped frames can be prevented from increasing rapidly, the movement of the decoded reproduced image on the receiving side becomes smooth.

FIG. 5 is a flow chart of the first embodiment of the present invention, in which the setting values such as the transmission speed of the transmission line and the image quality mode set in the operation unit 5 (see FIG. 1) are set in the control unit 4.
Is read (A1), it is determined whether or not the previously read set value is equal to the currently read set value (A2). If not, the bit rate setting process (A3) and the frame rate setting process ( A4), a buffer threshold setting process (A5) is sequentially performed.

If the previous set value is equal to the current set value in step (A2), the bit rate setting process, the frame rate setting process, and the buffer threshold setting process can be omitted. ), It is determined whether the encoding of one frame is completed or not (A).
6) When the processing is completed, a buffer capacity reading process (A7), a frame rate updating process (A8), and an encoded frame setting process (A9) are sequentially performed for the encoding process of the next frame. Then, if the encoding of one frame is not completed in step (A6) and after the processing of step (A9) is completed, the encoding processing is performed (A10).

FIG. 6 is a flow chart of the frame rate setting process, showing an outline of the process of step (A4) in the flow chart of FIG. 5. First, the bit rate is read out (B1), and this bit rate is smaller than 128 kbps. (B2), and if smaller, 128k
A frame rate for less than bps is set (B3). That is, the minimum frame rate is set. If the bit rate is not smaller than 128 kbps,
It is determined whether or not it is equal to s (B4).
A frame rate for 28 kbps is set (B5). That is, a frame rate one step higher than the minimum frame rate is set.

When the bit rate is not equal to 128 kbps, that is, when the bit rate is larger than 128 kbps, 1
It is determined whether or not it is equal to 92 kbps (B6). If it is, a frame rate for 192 kbps is set (B7). If it is not equal to 192 kbps, that is,
If the bit rate is greater than 192 kbps, 256
kbps is determined (B8), and if so, a frame rate for 256 kbps is set (B8).
9). If the bit rate is not equal to 256 kbps, that is, if the bit rate is larger than 256 kbps,
It is determined whether or not the frame rate is greater than ps (B10).
1) If not, set a frame rate for 320 kbps (B12). That is, the frame rate is set according to the bit rate.

FIG. 7 is a flowchart of the frame rate updating process, and shows an outline of the process of step (A8) in the flowchart of FIG. First, the expected buffer capacity at the end of the next encoding is calculated (C1). The next buffer capacity, that is, the expected buffer capacity calculated in step (C1) is compared with the UPPER threshold (C2). This UPP
The ER threshold and the LOWER threshold in the next step are:
This corresponds to a first threshold for monitoring an overflow set in step (A5) in the flowchart of FIG. 5 and a second threshold for monitoring an underflow.

Then, in step (C2), the UP
When the next buffer capacity is larger than the PER threshold, it is determined whether or not the current frame rate is equal to the target frame rate, that is, whether the target frame rate is equal to the target maximum frame rate (C4). Since the rate has been reached, the process proceeds to step (C8). If not equal, an update process for reducing the next frame rate is performed (C5), and the process proceeds to the next step (C8).

In step (C2), UPPER
If the next buffer capacity is not larger than the threshold, LOW
The R threshold is compared with the next buffer capacity (C3), and LOW
If the next buffer capacity is smaller than the R threshold, it is determined whether the current frame rate is equal to 7.5 frames / sec (C6). If equal, the update processing is performed because the minimum frame rate has already been reached. Without performing, the process proceeds to step (C8), and if they are not the same, an update process for increasing the next frame rate is performed, and the process proceeds to step (C8). If it is determined in step (C3) that the next buffer capacity is not smaller than the LOWER threshold, the process proceeds to step (C8). In this step (C8), the target frame rate of the next frame is set.

FIG. 8 is a flowchart for calculating the expected buffer capacity at the end of the next encoding, and shows an outline of the processing of step (C1) in the flowchart of FIG. The transmittable information amount of one frame is determined (D1). The amount of transmittable information during one frame can be obtained from the transmission speed. Then, the next frame transmittable information amount is determined (D2), and the buffer capacity of the next frame is determined (D2).
3). That is, the buffer occupancy at the next encoding start timing is a value obtained by adding the difference value between the expected occurrence information amount and the transmittable information amount of the next frame to the current buffer occupancy amount.

FIG. 9 is a flowchart of the buffer threshold setting process, and shows an outline of the process of step (A5) in the flowchart of FIG. First, the bit rate is read (E1), and it is determined whether or not this bit rate is smaller than 64 kbps (E2).
A buffer threshold for less than 8 kbps is set (E3). If the bit rate is not smaller than 64 kbps, it is determined whether or not the bit rate is equal to 128 kbps (E4). If so, a buffer threshold for 128 kbps is set (E5).

If the bit rate is not equal to 128 kbps, it is determined whether or not the bit rate is equal to 192 kbps (E6). If the bit rate is equal, a buffer threshold for 192 kbps is set (E7). If not equal, 256k
bps is determined (E8), and if so, a 256 kbps buffer threshold is set (E8).
9).

If the bit rate is not equal to 256 kbps, it is determined whether or not the bit rate is higher than 320 kbps (E10). If the bit rate is higher, a buffer threshold for 320 kbps or more is set (E11). Set the buffer threshold (E1
2).

For example, when the above-mentioned bit rate is set as the video rate, the first threshold value at 64 kbps is compared with the first threshold value at 128 kbps.
Since the bit rate transmitted from the buffer memory 3 (see FIG. 1) is low, the occupation amount of the buffer memory 3 tends to increase. Therefore, the first threshold value as the buffer threshold value in this case is set to a large value. Also, the target frame rate can be set according to the first threshold,
For example, if the video rate is 128 kbps compared to 64 kbps,
The target frame rate with kbps can be increased. As a result, in moving image coding transmission,
By setting it as a video rate, transmission can be performed without causing significant dropout of frames.

FIG. 10 is a flowchart of the second embodiment of the present invention. Steps (F1) to (F5) correspond to steps (A1) to (A5) of the flowchart of the first embodiment shown in FIG. In this embodiment, the same image quality mode setting processing (F6) is performed.
Is provided. Then, the subsequent steps (F
Steps (7) to (F11) perform the same processing as steps (A6) to (A10) in the flowchart of the first embodiment shown in FIG. 5, and a duplicate description will be omitted.

FIG. 11 is a flow chart of a frame rate changing process corresponding to the image quality mode. FIG. 11 shows a process corresponding to the image quality mode set in step (F6) in the flowchart of FIG. The process is performed (G1), and it is determined whether the image quality mode is important for motion (G2). If not important for motion, it is determined whether or not image quality is important (G3).

If the image quality mode emphasizes motion, it is determined whether the frame rate is 30 frames / sec (G
4) If 30 frames / sec, the process ends.
If the frame rate is not 30 frames / sec, the motion-oriented frame rate is set (G5), and the process ends.

If the image quality mode emphasizes motion, it is determined whether the frame rate is 7.5 frames / sec (G
6), if 7.5 frames / sec, the process ends. If the frame rate is not 7.5 frames / sec, the image quality-oriented frame rate is set (G7), and the process is terminated.

FIG. 12 is a flowchart of the buffer threshold value changing process in the image quality mode. The buffer threshold value set by the process in step (F5) in the flowchart of FIG. 10 is read (H1), and whether the image quality mode is motion-oriented is determined. Is determined (H2), and in the case of emphasis on motion, a buffer threshold for motion emphasis is set (H6). If the movement is not important, it is determined whether or not the image quality is important (H3). If the image quality is important, a buffer threshold for the image quality is set (H5). If neither movement nor image quality is important, a standard buffer threshold is set (H4). The video rate can be set as the image quality mode. In the case of this video rate setting, it is general that the emphasis is on motion, and the buffer threshold value setting processing based on the above-mentioned emphasis on motion can be performed.

FIG. 13 is a flowchart of the third embodiment of the present invention. This embodiment has a step (I6) of the delay control setting process, and other steps (I1) to (I5). Is the step (A1) in FIG.
To (A5) and steps (F1) to (F1) in FIG.
The same processing as in step 5) is performed.
7) to (I11) correspond to steps (A6) to FIG.
(A10) and steps (F7) to (F7) in FIG.
This performs the same processing as 11). The duplicate description of these similar processing steps will be omitted.

FIG. 14 is a flow chart of the buffer threshold changing process by the delay control, showing the outline of the process of step (I6) in FIG. 13, reading the buffer threshold (J1), and controlling the delay control with a low delay. Is determined (J2), and if the delay is set to low, a buffer threshold for low delay is set (J3), and the process ends. If the control is not the low-delay control, the process ends. That is, if the buffer threshold value for determining whether or not the buffer memory 3 (see FIG. 1) overflows is increased, the amount of accumulated information can be increased, and the transmission of encoded data is delayed. . Therefore, when the delay is set to be low, the buffer threshold is set so that the delay does not increase.

FIG. 15 is a flowchart of the fourth embodiment of the present invention. This embodiment is a case where the frame rate is updated not for each frame but for each set number of frames. K1) to (K6)
Steps (A1) to (A) in the flowchart of FIG.
This is the same as 6), and a duplicate description will be omitted.

If it is determined in step (K6) that the encoding of one frame is completed, the frame end counter is counted up (K7), and it is determined whether or not the count value of the frame end counter is equal to the set value (K8). ), If not equal, proceed to step (K12); if equal,
The frame end counter is cleared (K9), a buffer capacity reading process is performed (K10), and a frame rate updating process is performed according to the buffer capacity (K11).

If the encoding process for one frame is continued, the process shifts to step (K13) to continue the encoding process (K13). Step (K1
The setting process of the coded frame is performed according to 2), and then the process proceeds to the coding process (K13). Therefore, it is determined whether or not to perform the frame rate update process for each frame set in advance, and the process becomes easier as compared with the case of determining whether or not to perform the update process for each frame.

[0048]

As described above, according to the present invention, the occupancy of the buffer memory 3 at the next encoding start timing is determined based on the occupation amount of the buffer memory 3 and the setting information such as the transmission speed and the image quality mode. When the predicted occupancy becomes larger than the threshold, the target frame rate is switched. By switching the target frame rate,
The number of dropped frames will not increase extremely,
Therefore, there is an advantage that the movement of the reproduced image on the receiving side becomes smooth.

[Brief description of the drawings]

FIG. 1 is an explanatory diagram of an image encoding device according to an embodiment of the present invention.

FIG. 2 is an explanatory diagram of a frame rate.

FIG. 3 is an explanatory diagram showing a relationship among a quantization step, an amount of generated information, an image quality mode, and a target frame rate.

FIG. 4 is an operation explanatory diagram of the first embodiment of the present invention.

FIG. 5 is a flowchart according to the first embodiment of the present invention.

FIG. 6 is a flowchart of a frame rate setting process.

FIG. 7 is a flowchart of a frame rate update process.

FIG. 8 is a flowchart for calculating an expected buffer capacity at the end of the next encoding.

FIG. 9 is a flowchart of a buffer threshold setting process.

FIG. 10 is a flowchart according to the second embodiment of the present invention.

FIG. 11 is a flowchart of a frame rate changing process corresponding to an image quality mode.

FIG. 12 is a flowchart of a buffer threshold value changing process in an image quality mode.

FIG. 13 is a flowchart of the third embodiment of the present invention.

FIG. 14 is a flowchart of a buffer threshold changing process by delay control.

FIG. 15 is a flowchart of a fourth embodiment of the present invention.

FIG. 16 is an explanatory diagram of a conventional image encoding device.

FIG. 17 is a flowchart of a conventional image encoding process.

FIG. 18 is an explanatory diagram of a frame dropping process of a conventional example.

[Explanation of symbols]

 Reference Signs List 1 frame memory 2 coding processing unit 3 buffer memory 4 control unit 5 operation unit 11 DCT calculation unit 12 quantization unit 13 variable length coding unit 21 setting information reading unit 22 quantization step setting unit 23 target frame setting unit 24 occupation amount Monitoring unit

Claims (4)

[Claims] 1. An image encoding method for encoding an input image signal and transmitting the encoded image signal through a buffer memory, wherein an occupancy of the buffer memory is monitored, and a next occupancy is monitored based on the occupancy and setting information. Of the buffer memory at the encoding start timing, and when the estimated occupancy is out of the range of the threshold value, the target frame rate is switched, and encoding is performed in frame units according to the target frame rate. An image encoding method, comprising the steps of: 2. The image encoding method according to claim 1, further comprising the step of switching the target frame rate according to whether the set image quality mode is a motion-oriented mode or an image-oriented mode. 3. The method according to claim 1, further comprising the step of setting a video rate and determining the target frame rate according to the video rate. 4. An image encoding apparatus having a frame memory for adding an input image signal, an encoding processing unit, a buffer memory, a control unit, and an operation unit, wherein the control unit is configured to control the operation unit The setting information is read, and the occupancy of the buffer memory is monitored. Based on the setting information and the occupancy, the occupancy of the buffer memory at the next encoding start timing is predicted. An image encoding device comprising a target frame rate setting unit that switches a target frame rate when the amount exceeds a threshold.