JP2011233991A - Encoding apparatus and encoding method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an encoding apparatus that, when encoding a moving image, broadens a region to which an intra-encoded block is applied, while maintaining a target encoding amount, by setting the width of the region on the basis of a characteristic value and target encoding amount of the moving image.SOLUTION: An encoding apparatus encodes a moving image while switching between in-frame encoding and interframe encoding. The encoding apparatus includes a refresh control unit for controlling a refresh operation that performs concealment compensation for errors occurring in an initial frame through the last frame by performing in-frame encoding using a partial region in the frame as a refresh region. The refresh control unit controls the refresh operation so as to set a position and range of the refresh region in each frame.

Description

  The present invention relates to an encoding apparatus and an encoding method.

  In a moving image that has been compression-encoded using motion compensation prediction, if a loss or error occurs in data on the transmission path, an error occurs in motion compensation on the decoding side. In order to recover the generated motion compensation error, there is a method of performing refresh by periodic intra-frame coding, that is, intra coding.

  Japanese Patent Application Laid-Open No. 2004-228688 proposes a method of performing refresh by arranging intra coded blocks in the vertical direction in the screen and moving the intra coded blocks in the horizontal direction as the frame advances.

JP 7-322266 A

  However, in the conventional method, since the region of the intra-coded block having a certain width circulates, it takes time until the intra-coded block makes a round in the screen. Therefore, there is a problem that it takes a long time for the entire screen to recover after an error occurs in the transmission path. If it takes time until the entire screen returns, there is a concern that the time during which the screen is not displayed becomes longer in moving image transmission, and the convenience for the user is reduced.

  On the other hand, in order to shorten the refresh time, if the width of the region to which the intra-coded block is applied is widened, intra-frame coding requires more code amount than inter-frame coding, so that There is a concern that the typical code amount increases. An increase in the local code amount causes local variations in the generated code amount. For example, there is a problem that a delay time is increased in transmission at a constant rate.

  Therefore, in the present invention, when the encoding device encodes a moving image, the target code amount is protected by setting the width of the region to which the intra-coded block is applied from the feature amount of the moving image and the target code amount. However, the area to which the intra-coded block is applied is expanded.

  In the encoding apparatus according to the present invention, if the code amount does not increase even if the area to which the intra-coded block is applied is expanded by setting the application area of the intra-coded block, the intra-coded block goes around the screen. Can be shortened, that is, the refresh time. As a result, even when an error occurs temporarily in the transmission line, the refresh operation can be completed in a shorter time and the entire screen can be restored as compared with the conventional method.

It is an example of the block diagram of an encoding apparatus. It is an example of a figure showing basic operation of refresh FIG. 3 is an example of a diagram illustrating that refresh is performed with an appropriate refresh width set according to the first exemplary embodiment. FIG. 10 is an example of a diagram illustrating that an image is divided into a plurality of horizontal regions and refresh is performed by setting a refresh width for each region in the second embodiment. FIG. 10 is a diagram illustrating an example in which an image is divided into a plurality of horizontal regions and refresh is performed by setting a refresh start position for each region in the third embodiment. FIG. 10 is an example of a diagram illustrating that an image is divided into a plurality of vertical regions and a refresh operation is performed for each region in the fourth embodiment. FIG. 10 is an example of a diagram illustrating that an image is divided into two vertical regions, refreshing from the left region from the left, and refreshing from the right region from the right according to the fifth embodiment. FIG. 3 is an example of a flowchart illustrating an operation flowchart of refresh control in the first embodiment.

  Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is an example of a configuration diagram of an encoding device.

  The image input unit 101 inputs an image to be encoded and outputs it as an encoding target image. The image is processed by being divided into blocks of a predetermined size, that is, macroblocks. The subtracter 102 outputs the difference between the image input 101 and the predicted image. The switch 103 selects and outputs one of the output from the image input 101 and the output from the subtractor 102. The DCT / quantization unit 104 performs orthogonal transform on the output from the switch 103, quantizes the orthogonal transform coefficient, and outputs the result. The variable length coding unit 105 performs variable length coding on the output of the DCT / quantization unit 104 and outputs the result.

  The inverse quantization / IDCT unit 106 dequantizes the output of the DCT / quantization unit 104, performs inverse orthogonal transformation, and outputs the result. The buffer memory 107 temporarily stores the output of the inverse quantization / IDCT unit 106. The motion prediction unit 108 predicts and outputs a motion vector from the image input 101 and the buffer memory 107. The motion compensation unit 109 outputs a predicted image obtained by performing motion compensation on the image in the buffer memory 107 from the motion vector predicted by the motion prediction unit 108.

  The encoding control unit 110 compares the output of the image input 101 and the output of the subtractor 102, determines which is higher in encoding efficiency, and outputs the determination result. The refresh control unit 111 compares the parameter stored in the memory 113, determines whether the image input 101 satisfies the refresh condition, and outputs a determination result. The OR circuit 112 outputs an OR of outputs from the encoding control unit 110 and the refresh control unit 111. The switch 103 switches the switch according to the output of the OR circuit 112.

  If the encoding control unit 110 determines that the output of the image input 101 is higher in encoding efficiency, the encoding control unit 110 determines to encode the macroblock as an intra macroblock. Conversely, if the encoding control unit 110 determines that the output of the subtracter 102 has higher encoding efficiency, the encoding control unit 110 determines to encode the macroblock as an inter macroblock.

  When it is determined that the refresh condition is satisfied, the refresh control unit 111 determines to encode the macroblock as an intra macroblock. Conversely, if the refresh control unit 111 determines that the refresh condition is not satisfied, the refresh control unit 111 determines to encode the macroblock as an inter macroblock.

  When the output of the image input 101 is selected by switching the switch (that is, when the switch 103 is connected to the upper terminal in FIG. 1), the macroblock is called an intra macroblock. When the output of the subtracter 102 is selected (that is, when the switch 103 is connected to the lower terminal in FIG. 1), it is called an inter macro block. The output of the OR circuit 112 is obtained by switching the switch 103 to the image input 101 side when the encoding control unit 110 or the refresh control unit 111 determines to encode the macro block as an intra macro block. When both the refresh control units 111 determine that the macroblock is to be encoded as an inter macroblock, the switch 103 is switched to the subtractor 102 side.

  Image input unit 101, subtractor 102, switch 103, DCT / quantization unit 104, variable length encoding unit 105, inverse quantization / IDCT unit 106, buffer memory 107, motion prediction unit 108, motion compensation unit 109, encoding The control unit 110, the refresh control unit 111, the OR circuit 112, and the memory 113 are realized by hardware such as an integrated circuit.

  Further, the processing of the processing unit and the like can be realized by software. In this case, the image input program 101, the subtractor program 102, the switch program 103, the DCT / quantization program 104, the variable length encoding program 105, the inverse quantization / IDCT program 106, the buffer memory program 107, which are stored in the memory. Various processes such as a motion prediction program 108, a motion compensation program 109, an encoding control program 110, a refresh control program 111, an OR circuit program 112, and a memory program 113 are executed by a processor (not shown) to perform each process.

Moreover, a part of the processing unit can be realized by hardware such as an integrated circuit, and a part can be realized by software. Next, the basic operation of the refresh control unit 111 will be described with reference to FIG.
In the first frame, the refresh control unit 111 has the horizontal position of the processing macroblock on the right side of the refresh start position (P (1)) and on the left side of the position obtained by adding the refresh width (W) to P (1). In some cases, the macro block is treated as an intra macro block, and a determination result that satisfies the refresh condition is output. Conversely, when the horizontal position of the processing macroblock is on the left side of P (1) or on the right side of the position obtained by adding W to P (1), a determination result that the refresh condition is not satisfied is output.

  The refresh control unit 111 finishes processing the first frame and updates P (1) when processing the second frame. For example, by adding W to P (1) and setting it to P (2), the region to be refreshed next can be shifted to the right by W. Also in the second frame, as in the first frame, the horizontal position of the processing macroblock is on the right side of the refresh start position (P (2)) and from the position where the refresh width (W) is added to P (2). When it is on the left side, the macro block is treated as an intra macro block, and a determination result that satisfies the refresh condition is output. Conversely, when the horizontal position of the processing macroblock is on the left side of P (2) or on the right side of the position obtained by adding W to P (2), a determination result indicating that the refresh condition is not satisfied is output.

  The area on the left side of P (2) in the second frame is the refresh area of the first frame, and is treated as refreshed in the second frame. Since the refreshed area is encoded as an intra macroblock, even if a transmission error occurs before the first frame, no erroneous decoding is performed. Thereafter, the frame advances, and when all the areas have been refreshed or become the refresh area in the Nth frame, the refresh is completed. The above is the basic operation of the refresh process.

  The refresh processing time is the time until all frames are refreshed, counting from the frame immediately after the transmission error has occurred. For example, assume that the input image width is L, the refresh width is W, and the refresh position P (i + 1) is P (i + 1) = P (i) + W. When the refresh position of the frame immediately after the occurrence of the transmission error is at the left end of the screen, that is, P = 0, the refresh processing time T (a) is minimum, and T (a) = L / W. In addition, when the refresh position of the frame immediately after the occurrence of the transmission error is right by W from the left end of the screen, that is, P = W, the refresh processing time becomes maximum, and T (a) = (2L−W) / W. .

  Processing performed by the refresh control unit 111 of this embodiment to shorten the refresh processing time will be described with reference to FIG.

  In the first frame, the refresh control unit 111 has a horizontal position of the processing macroblock on the right side of the refresh start position (P (1)) and a position obtained by adding a refresh width (W (1)) to P (1). When it is on the left side, the macro block is treated as an intra macro block, and a determination result that satisfies the refresh condition is output. Conversely, when the horizontal position of the processing macroblock is on the left side of P (1) or on the right side of the position obtained by adding W (1) to P (1), a determination result indicating that the refresh condition is not satisfied is output. To do.

  The refresh control unit 111 finishes processing the first frame and updates P (1) when processing the second frame. For example, by adding W (1) to P (1) to be P (2), the area to be refreshed next can be shifted to the right by W (1). Further, W (2) is updated. For example, a feature amount is extracted from pixel information of the input image, a predetermined calculation is performed, and an appropriate W (2) is set. As a calculation method of W (2), there is a method of calculating the code amount of the frame to be equal to or less than the target code amount even if the refresh area of the second frame is processed as an intra macroblock.

  Parameters such as the target code amount are stored in the memory 113, and the refresh control unit 111 obtains them and compares them with the calculated frame code amount. Alternatively, the parameters are input from the user by an input unit (not shown).

  For example, when the variance value of the pixel value of the first frame is acquired as the feature amount and this variance value is lower than the target variance value that is a parameter stored in the memory 113, the refresh control unit 111 displays a simple image. Therefore, it is assumed that the amount of data required for encoding is small, and W (2) is widened. On the contrary, when the variance value of the pixel value of the first frame is higher than the target variance value that is a parameter stored in the memory 113, the refresh control unit 111 is a complex image, and therefore the amount of data required for encoding Therefore, W (2) is narrowed. As a result, W (2) can be expanded within the range of the target code amount.

  An operation flowchart of the refresh control unit 111 will be described with reference to FIG. When the i-th frame image is input (S802), the refresh control unit 111 sets the i-th frame refresh width W (i) (S803), and sets the i-th frame refresh start position P (i). (S804). Next, the refresh control unit 111 determines whether or not the currently processed macroblock is in the refresh area (S805), and if it is in the refresh area, outputs a determination result indicating that it is an intra macroblock. (S806). If it is not in the refresh area, a determination result indicating that it is an inter macro block is output (S807).

  Thereafter, the refresh control unit 111 determines whether or not the macroblock is the end of the frame (S808). If the macroblock is not the end of the frame, the next macroblock is selected (S809), and the process returns to S805. If the macroblock is at the end of the frame, the refresh control unit 111 selects the next frame (S810), and returns the process to S802.

According to this method, even if the macroblock is an intra macroblock, for example, when the image is simple and the amount of data required for encoding is small, the refresh width can be increased. As a result, the refresh time can be shortened.
On the contrary, if the macro block is an intra macro block, for example, when the image is complicated and a large amount of data is required for encoding, the refresh width can be reduced. As a result, the set bit rate can be maintained.

In this case, the refresh width can be set to a minimum value or a maximum value. By setting the minimum value, the refresh can always be completed within a predetermined time. By setting the maximum value, it is possible to control the delay amount while suppressing local variations in the code amount. The minimum value and the maximum value of the refresh width are stored in the memory, or an input unit (not shown) receives value input from the user.
Thereafter, the frame advances, and when all the areas have been refreshed or become the refresh area in the Nth frame, the refresh is completed.

  According to the method of this embodiment, the refresh processing time T (b) is L / ((W (1) + W (2) +... + W (N)) / N). Therefore, when the average of the refresh width W (X) in the sections 1 to N is larger than W, T (b) <T (a) is satisfied, so that the refresh time can be shortened.

  On the contrary, when the average of the refresh width W (X) in the sections 1 to N is smaller than W, T (b)> T (a) is satisfied, so that the refresh time is increased. This can be prevented by setting the minimum value of. For example, when the minimum value of the refresh width is set to W, the average of the refresh width W (X) in the sections 1 to N is the same as or larger than W. Therefore, T (b) <= T (a) Therefore, the refresh time does not increase beyond T (a).

According to the present embodiment, the refresh time can be expanded to shorten the refresh time and promptly return the screen.
For example, when the intra-frame pixel correlation of the input image is high, the target code amount can be satisfied even if encoding is performed with an intra macroblock. In such a case, the refresh area is widened. Conversely, when the intra-frame pixel correlation of the input image is low, the refresh area is narrowed. By setting the refresh area for each frame depending on the nature of the input image, the refresh time can be shortened while satisfying the target code amount.

In this embodiment, the refresh direction has been described as an example from the left to the right, but the refresh may be performed from the right to the left.
In this embodiment, the refresh area shape is a set of vertically long macroblocks, but the set of horizontally long macroblocks may be refreshed by moving vertically.

Embodiments of the present invention will be described below with reference to the drawings. In addition, description is abbreviate | omitted about the part same as Example 1, and a different part is demonstrated in detail.
Processing performed by the refresh control unit 111 of this embodiment to shorten the refresh processing time will be described with reference to FIG.

  The refresh control unit 111 sets the refresh area by dividing the input image in the horizontal direction. For example, the input image is divided into four in the horizontal direction, and the first area, the second area, the third area, and the fourth area are sequentially arranged from the top. In the first frame, a refresh width is set for each region. For example, the refresh control unit 111 extracts a feature amount for each region, performs a predetermined calculation, and sets an appropriate refresh width.

  When the feature amount is different for each region, a different refresh width is set. Therefore, in each processing macroblock, the refresh width varies depending on the vertical position of the processing macroblock. For example, in the first frame of FIG. 4, the first region and the fourth region have a narrow refresh width, the second region is wider, and the third region is the widest.

  Even after the second frame, the refresh width can be set for each region. Since the refresh width differs from region to region, only some regions may complete the refresh first. For example, in the third frame, although the first area and the fourth area have unrefreshed areas, the second area and the third area are all refreshed or refresh areas. This is because even if the second area and the third area are encoded with a wide refresh width, the target code amount can be maintained. It shows that the refresh time can be shortened compared to the area and the fourth area.

  According to this embodiment, the refresh time can be shortened for each region by setting the refresh width for each region. For example, even if the intra-frame pixel correlation is low when viewed in the entire input image, the intra-frame pixel correlation is high when there are areas where the intra-frame pixel correlation is high and the intra-frame pixel correlation is low when viewed for each area. As for the area, the refresh width can be widened to shorten the refresh time as much as possible. In this embodiment, the refresh width can be changed for each frame in combination with the first embodiment. By combining the present embodiment and the first embodiment, more precise refresh control can be realized.

    In the present embodiment, the number of divisions is four, but the number of divisions may be other than this.

Embodiments of the present invention will be described below with reference to the drawings. In addition, description is abbreviate | omitted about the part same as Example 1, and a different part is demonstrated in detail.
Processing performed by the refresh control unit 111 of this embodiment to shorten the refresh processing time will be described with reference to FIG.

  As in the second embodiment, the refresh control unit 111 divides the input image in the horizontal direction and sets a refresh area. For example, the input image is divided into four in the horizontal direction, and the first area, the second area, the third area, and the fourth area are sequentially arranged from the top. The refresh control unit 111 shifts the refresh start position in the horizontal direction for each region in the first frame. If there is an unrefreshed area on the left side of the refresh start position in the same area in the first frame, the area that was the refresh area in the first frame in the second frame refers to the unrefreshed area in the first frame. Motion compensation is performed as an image, and the image may be disturbed. Therefore, in the second frame, the area that was the refresh area in the first frame is called an intermediate refresh area that has not been completely refreshed.

  When an area obtained by adding a refresh width to the refresh start position protrudes from the right end of the input image, only the extended width is treated as a refresh area from the left end of the input screen. At this time, since the area protruding in the first frame does not refer to the unrefreshed area of the first frame in the second frame, it is guaranteed that the area is refreshed. Therefore, the area that protrudes in the first frame becomes a complete refresh area.

  The refresh is completed when the entire screen becomes a completely refreshed area or a refresh area as in the Nth frame. When the entire screen becomes an intermediate refreshed area, a completely refreshed area, or a refresh area as in the N'th frame, the intermediate refreshed area may be disturbed again, but the user There is an advantage that a return image of a certain quality can be presented at an early stage.

  According to the present embodiment, the refresh area is widely distributed in the horizontal direction by shifting the refresh start position, so that the horizontal refresh range is expanded. As a result, a wide range of refreshing is possible, and when viewed in the horizontal direction, it is possible to prevent the refreshing from being significantly delayed depending on the region, and it is possible to quickly present an overview of the entire screen to the user.

By combining the present embodiment with the first embodiment or the second embodiment, more precise refresh control can be realized.
In the present embodiment, the number of divisions is four, but the number of divisions may be other than this.

Embodiments of the present invention will be described below with reference to the drawings. In addition, description is abbreviate | omitted about the part same as Example 1, and a different part is demonstrated in detail.
Processing performed by the refresh control unit 111 of this embodiment to shorten the refresh processing time will be described with reference to FIG.

  The refresh control unit 111 divides the input image in the vertical direction and sets a refresh area. For example, the refresh control unit 111 divides the input image into four in the horizontal direction, and sequentially sets the first area, the second area, the third area, and the fourth area from the left. Since it is divided in the vertical direction, the refresh start position is different in each region. In each area, as in the first embodiment, the refresh control unit 111 moves the refresh start position to the right by the refresh width in the second frame.

  At this time, after the second area, the area that was the refresh area in the first frame may refer to the unrefreshed area of the first frame in the second frame. May be disturbed. Therefore, this area becomes an intermediate refresh area. The area refreshed in the first area becomes a completely refreshed area.

  According to this embodiment, since the completely refreshed area or the intermediate refreshed area is widely distributed in the horizontal direction, the area that can be refreshed in one frame is widened. By further increasing the number of horizontal divisions, a wider range of refresh becomes possible. This can prevent the refresh of the right region from being delayed significantly when viewed in the horizontal direction.

By combining the present embodiment with the first embodiment or the second embodiment, more precise refresh control can be realized.
In the present embodiment, the number of divisions is four, but the number of divisions may be other than this.

Embodiments of the present invention will be described below with reference to the drawings. In addition, description is abbreviate | omitted about the part same as Example 1, and a different part is demonstrated in detail.
Processing performed by the refresh control unit 111 of this embodiment to shorten the refresh processing time will be described with reference to FIG.

  The refresh controller 111 sets the refresh area by dividing the input image into two in the vertical direction. For example, the input image is divided into two in the horizontal direction, and the first area and the second area are sequentially arranged from the left. As in the first embodiment, the first region starts the refresh start position (P (1, 1)) from the left end in the first frame. The second area is a place where the refresh start position (P (1, 2)) is moved to the left by the refresh width (W) from the right end of the image input in the first frame.

  In the second frame, the first area moves the refresh start position to the right by W as in the first embodiment. The second area moves the refresh start position to the left by W. The area that was the refresh area of the first area in the first frame becomes the completely refreshed area in the second frame. The region that was the refresh region of the second region in the first frame is defined as a completely refreshed region by limiting the motion prediction range so that the second frame does not refer to the unrefreshed region of the first frame. Become. When the entire screen becomes a completely refreshed area or a refresh area, the refresh is completed.

According to this embodiment, even when refresh is performed by dividing into a plurality of areas, all the refresh areas are complete refresh areas, so that the entire screen is completely refreshed by one refresh. Therefore, the refresh time can be shortened compared to the case where the refresh is started from the middle of the image as in the fourth embodiment. Further, since refresh is performed from both the left end and the right end by dividing into a plurality of regions, it is possible to prevent the refresh of the right region from being delayed when viewed in the horizontal direction. In combination with 2, more precise refresh control can be realized.
In this embodiment, the number of divisions is two, but the division may be performed by other division numbers.

  In addition, this invention is not limited to an above-described Example, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. In addition, a part of the configuration of a certain embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of a certain embodiment. Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

  Each of the above-described configurations, functions, processing units, processing means, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor. Information such as programs, tables, and files for realizing each function can be stored in a recording device such as a memory, a hard disk, an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

  Further, the control lines and information lines indicate what is considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. Actually, it may be considered that almost all the components are connected to each other.

DESCRIPTION OF SYMBOLS 101 Image input part 102 Subtractor 103 Switch 104 DCT / quantization part 105 Variable length encoding part 106 Inverse quantization / IDCT part 107 Buffer memory 108 Motion prediction part 109 Motion compensation part 110 Encoding control part 111 Refresh control part 112 OR circuit Part 113 memory

Claims (9)

In an encoding device that encodes a moving image by switching between intra-frame encoding and inter-frame encoding,
A refresh control unit that controls a refresh operation that compensates for an error that has occurred in the previous frame by intra-frame encoding a partial region in the frame as a refresh region,
The refresh control unit
Control to set the position and range of the refresh area for each frame,
An encoding device characterized by the above. The encoding device according to claim 1, wherein the refresh control unit includes:
Calculate the feature amount of the pixel information of the encoding target frame, the past frame from the frame, or the future frame from the frame,
Setting the position and range of the refresh region of the frame so that the calculated code amount of the frame is smaller than the target code amount acquired from the storage unit;
An encoding device characterized by the above. The encoding device according to claim 2, wherein
The refresh control unit
Split the frame into horizontal regions,
One or both of the feature amount of the pixel information of the frame and the feature amount of the region are calculated in each region, and the calculated feature amount is smaller than the target code amount acquired from the storage unit. Setting the location and range of the refresh area of the area,
An encoding device characterized by the above. The encoding device according to claim 2, wherein
The refresh control unit
Split the frame into horizontal regions,
Set the start position of the refresh area to be different in each area,
An encoding apparatus characterized by that. The encoding device according to claim 2, wherein
The refresh control unit
Split the frame into vertical regions,
Perform refresh processing in each area,
An encoding apparatus characterized by that. The encoding device according to claim 5, wherein
The refresh control unit
Dividing the frame vertically into two, with the left side of the frame as the first area and the right side as the second area,
Let the refresh start position of the first area be the leftmost position of the frame,
The refresh start position of the second area is set to the left of the refresh area range from the right end of the frame.
In subsequent frames,
The refresh start position of the first area is set to the right by the range of the refresh area from the refresh start position of the first area of the previous frame.
The refresh start position of the second area is set to the left position by the refresh area from the refresh start position of the second area of the previous frame.
An encoding apparatus characterized by that. In the encoding device according to any one of claims 2 to 6,
The refresh control unit
Setting the minimum or maximum value of the refresh area range,
An encoding device characterized by the above. In the encoding device according to any one of claims 2 to 6,
The refresh control unit
Using the variance of the pixel value of the frame as a feature quantity of the pixel information of the frame;
An encoding device characterized by the above. In a coding method for coding a moving image by switching between intra-frame coding and inter-frame coding,
Intra-frame coding with a partial area in the frame as a refresh area, and performing a refresh operation to compensate for errors that occurred in the previous frame,
Control to set the position and range of the refresh area for each frame,
An encoding method characterized by the above.

Images