JP2019097030A - Image coding apparatus, imaging apparatus, image coding method, and program - Google Patents

Abstract

To provide a technique for suppressing an increase in the overall generated code amount while performing quantization control based on a target image quality for one of two images to be encoded.SOLUTION: An image coding apparatus includes encoding means that performs encoding processing due to quantization on a first image and a second image, and one of the first image and the second image is an image generated on the basis of the other image, and control means that controls the quantization of the first image on the basis of target image quality and controls the quantization of the second image on the basis of a target code amount.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image coding apparatus, an imaging apparatus, an image coding method, and a program.

  High efficiency video coding (HEVC) coding, which is an international standard coding standard for moving pictures, is beginning to spread. In the HEVC encoding scheme, a still picture profile has been newly introduced (see Non-Patent Document 1). This is a profile for encoding a still picture by HEVC of a moving picture coding scheme. This is expected to spread still image files using HEVC in the future.

  In HEVC, CABAC (Context-based Adaptive Binary Arithmetic Coding) is used for entropy coding. Although CABAC has an advantage of high compression rate, it has a feature that parallel processing is difficult. Since the encoding process is sequentially performed bit by bit, it takes a processing time in proportion to the code amount. In addition, CABAC is not only HEVC but H. H.264 is also adopted.

  The use of MPF (Multi Picture Format) is assumed as one of the container formats of HEVC still image files. This is a format that assumes that a plurality of still images are stored in one file. For example, in addition to a still image (main image) which is a normal captured image, one file can simultaneously include still images (sub-images) subjected to resolution conversion to 2 k size or 4 k size.

  In JPEG (Joint Photographic Experts Group), which is a coding standard for conventional still images, the quantization value is constant in the screen, and a quantization table is generally prepared for each image quality mode such as normal or fine. . For example, the quantization table is not changed according to the encoding difficulty (complexity) of the image. This is because it is desirable to make the image quality level constant for each image quality mode.

  When the same processing is performed in HEVC, the quantization value in the screen is made constant for each image quality mode. Originally, in HEVC, it is possible to change the quantization value for each CU (Coding Unit), but it is possible to keep the image quality level constant by using a fixed quantization value predetermined for each image quality mode. It becomes.

H. 265 / HEVC textbook, Atsushi Okubo [Supervised], Suzuki Teruhiko, Takamura Seiyuki, Nakajo Ken [jointly edited], published on October 21, 2013

  However, when quantization control with priority on image quality (for example, control using a constant quantization value in the screen according to the image quality mode) is performed for both the main image and the sub image, the coding difficulty of the image (complex) Depending on the nature of the code, the code amount may increase significantly.

  The present invention has been made in view of such a situation, and when encoding two images, while performing quantization control based on the target image quality for one of the images, It aims at providing the technology which controls increase.

  In order to solve the above-mentioned problems, the present invention is an encoding means for performing encoding processing involving quantization on a first image and a second image, wherein the first image and the second image are encoded. Coding means, one of the images being an image generated based on the other of the first image and the second image, and the quantization of the first image based on a target image quality Control means for controlling the quantization of the second image based on a target code amount, and an image coding apparatus characterized by comprising:

  Other features of the present invention will become more apparent from the description in the accompanying drawings and the following detailed description of the invention.

  According to the present invention, when encoding two images, it is possible to suppress an increase in the overall generated code amount while performing quantization control based on the target image quality for one of the images.

FIG. 1 is a block diagram showing a configuration of an imaging device 100. FIG. 7 is a conceptual diagram comparing generated code amounts in the case of normal quantization control with generated code amounts in the first embodiment. 6 is a flowchart of encoding processing performed by the imaging device 100. FIG. 10 is a conceptual diagram of generated code amounts in the second embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The technical scope of the present invention is determined by the scope of the claims, and is not limited by the following individual embodiments. Moreover, not all combinations of features described in the embodiments are essential to the present invention. Also, features described in different embodiments may be combined as appropriate.

First Embodiment
FIG. 1 is a block diagram showing the configuration of an imaging apparatus 100 including the image coding apparatus according to the first embodiment. Although the example which encodes by a HEVC encoding system is demonstrated below, the encoding system of this embodiment is not limited to a HEVC encoding system.

  In the present embodiment, the imaging apparatus 100 performs encoding processing on two images (still images) of the main image and the sub image, associates the two encoded images with one another, and records them in one file. The process in the case of performing will be described. Hereinafter, a recording method for recording a plurality of images in one file is referred to as MPF (Multi Picture Format).

  First, encoding processing for one still picture will be described. In FIG. 1, the functional blocks of HEVC regarding moving images are omitted. A light flux from a subject is input to the imaging unit 102 through the lens 101. The imaging unit 102 converts the input light flux into digital pixel data, and outputs the digital pixel data to the development processing unit 103.

  The development processing unit 103 performs various image processing such as debayering processing, flaw correction, noise removal, and color conversion to the YCbCr format. After image processing, an image in a format that can be compressed and encoded is input to the encoding frame buffer 104 as an encoding target image to be encoded.

  The intra prediction unit 106 performs intra prediction (in-screen prediction) for each block of the encoding target image. In intra prediction, the intra prediction unit 106 uses the coding target image stored in the coding frame buffer 104 and the peripheral image in the past coded screen stored in the intra prediction unit 106. Based on the intra prediction, determine the prediction mode. The intra prediction unit 106 obtains a pixel difference between the encoding target image and the intra prediction image according to the determined prediction mode, and generates a difference image. The generated difference image is output to the orthogonal transform unit 107.

  The orthogonal transform unit 107 generates transform coefficients by performing discrete cosine transform on the difference image, and outputs the transform coefficients to the quantization unit 108.

  The quantization unit 108 quantizes the transform coefficient sent from the orthogonal transform unit 107 in accordance with the quantization step (quantization value) output from the quantization control unit 109. The quantized transform coefficients are output to the variable-length coding unit 110 to generate a coded stream. In addition, the quantized transform coefficient is output to the inverse quantization unit 112 in order to create a local decoded image which is required in the parameter calculation of the adaptive offset processing.

  The variable-length coding unit 110 performs variable-length coding by performing zigzag scan or alternate scan on the quantized transform coefficients. Also, the variable-length coding unit 110 performs variable-length coding on coding scheme information such as an intra prediction mode, a quantization value, block division information, and a parameter for adaptive offset processing. The variable-length coding unit 110 generates a coded stream by adding coded coding method information to the coded transform coefficient. The generated encoded stream is stored in the stream buffer 111. In addition, the variable-length coding unit 110 calculates the generated code amount for each block at the time of coding, and outputs the calculated code amount to the quantization control unit 109.

  The coding scheme setting unit 113 determines the quantization control scheme in the quantization control unit 109. The quantization control unit 109 determines the quantization value according to the generated code amount sent from the variable-length coding unit 110 and the quantization control method determined by the coding method setting unit 113. Details of the quantization control method in the quantization control unit 109 will be described later.

  The inverse quantization unit 112 performs inverse quantization on the quantized transform coefficient sent from the quantization unit 108, and generates a transform coefficient for local decoding. The transform coefficients are output to the inverse orthogonal transform unit 114.

  The inverse orthogonal transform unit 114 generates a difference image by performing inverse discrete cosine transform on the received transform coefficient. The generated difference image is output to the motion compensation unit 115.

  The motion compensation unit 115 adds the predicted image sent from the intra prediction unit 106 and the difference image sent from the inverse orthogonal transform unit 114 to generate image data for local decoding. The generated image data is output to the deblocking filter unit 116.

  The deblocking filter unit 116 applies a deblocking filter to the sent image data. The image after the deblocking filter is output to the adaptive offset processing unit 117.

  The adaptive offset processing unit 117 selects any of band offset processing, edge offset processing, or no processing, and determines a band position, an edge direction, an offset value, etc. for which the adaptive offset processing is to be performed. The adaptive offset processing unit 117 performs variable-length coding to include parameters for adaptive offset processing (information on which processing has been selected as adaptive offset processing, band position, edge direction, offset value, etc.) in a coded stream. Output to the part 110.

  By the above processing, a coded stream of one still image is created.

  Next, an operation in the case where one still image is photographed and two still images of the main image not subjected to reduction processing and the sub image subjected to the reduction processing are mutually associated and recorded will be described. Here, it is assumed that the encoding target image input to the encoding frame buffer 104 by the development processing unit 103 described above is used as a main image.

  The development processing unit 103 adds the main image to the encoded frame buffer 104 and also outputs the main image to the reduction processing unit 105. The reduction processing unit 105 performs reduction processing on the main image to generate a sub image. The generated sub-image is stored in the coding frame buffer 104.

  The encoding process is performed on the main image stored in the encoded frame buffer 104 as described above, and the encoded stream of the main image is stored in the stream buffer 111. After the encoding process of the main image is performed, the encoding process of the sub image is performed, and the encoded stream of the sub image is stored in the stream buffer 111.

  After the coded streams of both the main image and the sub image are generated, these two coded streams are input to the multiplexing unit 118. The multiplexing unit 118 multiplexes the two encoded streams according to the MPF to generate an MPF stream. The multiplexing unit 118 records the MPF stream as one file on the recording medium 119.

  Next, details of quantization control performed by the imaging device 100 will be described. In general, in a still image, encoding is performed with the quantization value fixed in the screen. Further, as the quantization value, a value determined in advance according to the image quality mode or the like is used. By performing encoding using the quantization value determined in this manner, the image quality level can be maintained, but the amount of generated code depends on the input image (image to be encoded).

  The encoding scheme setting unit 113 holds the target image quality and the target code amount. Also, the coding method setting unit 113 holds information indicating whether the current image to be coded is a main image or a sub image. The imaging apparatus 100 may be configured to allow the user to set the target image quality and the target code amount.

  If the current image to be encoded is a main image, the encoding method setting unit 113 notifies the quantization control unit 109 of the target image quality and instructs to perform quantization control with image quality priority. The quantization control unit 109 performs quantization control according to the instruction. For example, as quantization control with image quality priority, the quantization control unit 109 determines a constant quantization value in the screen (entire screen) based on the target image quality as in the conventional quantization control of a still image. By performing quantization control in this manner, the image quality level of the main image can be maintained.

  If the current image to be encoded is a sub-image, the encoding scheme setting unit 113 notifies the quantization control unit 109 of the target code amount and instructs to perform quantization control with code amount priority. The quantization control unit 109 performs quantization control according to the instruction. For example, the coding method setting unit 113 performs quantization control so that the generated code amount falls within the target code amount. In this case, the variable-length coding unit 110 calculates the generated code amount for each CU and inputs the calculated amount to the quantization control unit 109. The quantization control unit 109 determines the quantization value to be used for the next CU by observing the difference between the generated code amount and the target code amount for each CU. In this way, quantization control can be performed so that the generated code amount of the entire screen falls within the target code amount.

  Since the generated code amount of the sub image is suppressed to a predetermined (or set by the user) target code amount, an increase in the generated code amount is suppressed even if the sub image is complicated. Therefore, in general, the sub-image can be encoded with a smaller code amount as compared to the case where the quantization value is constant in the screen. That is, the sum of the code amount of the main image and the code amount of the sub image can be reduced compared to the case where both the main image and the sub image are quantized using a constant quantization value in the screen.

  FIG. 2 is a conceptual diagram comparing the generated code amount in the case of normal quantization control with the generated code amount in the first embodiment. In normal quantization control, encoding is performed using constant quantization values in the screen for both the main image and the sub-image. The sum of the generated code amount 201 of the main image and the generated code amount 202 of the sub image is the total generated code amount.

  In the quantization control in the present embodiment, encoding is performed using a fixed quantization value in the screen (entire screen) for the main image. Also, for the sub-image, encoding is performed such that the generated code amount falls within the target code amount Target_Pic (so that the generated code amount does not exceed the target code amount). The generated code amount 203 of the main image is the same as the generated code amount 201 of the main image in the normal quantization control. Since the quantization control is performed so that the generated code amount 204 of the sub image falls within Target_Pic, the generated code amount 204 of the sub image is smaller than the generated code amount 202 of the sub image in the normal quantization control. As a result, compared to normal quantization control, as the generated code amount 204 of the sub image is smaller, the generated code amount as a whole can be reduced.

  In HEVC, coding is performed using CABAC (Context-based Adaptive Binary Arithmetic Coding), which is arithmetic coding, as a method of variable-length coding. In CABAC, since the code occurrence probability is updated bit by bit, it is difficult to perform processing in parallel. Therefore, the processing performance of CABAC depends on the code amount input to CABAC. The smaller the code amount, the shorter the processing time and the higher the processing performance.

  According to the present embodiment, it is possible to reduce the total code amount of the main image and the sub image, as compared to encoding in which normal quantization control is performed. Therefore, it is possible to increase the processing performance in CABAC as the amount of code is reduced.

  FIG. 3 is a flowchart of the encoding process performed by the imaging device 100. Here, among the encoding processing, processing mainly related to quantization control will be described, and description of other processing (for example, intra prediction processing and the like) will be omitted. In addition, the functions of the quantization unit 108, the quantization control unit 109, and the variable-length coding unit 110 in this flowchart can be implemented by any hardware or software. For example, in the case of software implementation, these functions can be realized by the CPU of the imaging device 100 developing and executing a control program stored in the ROM in the RAM.

  In step S301, the quantization control unit 109 acquires the target image quality and the target code amount from the encoding method setting unit 113.

  In step S302, the quantization control unit 109 determines the quantization value for the main image based on the target image quality, and notifies the quantization unit 108 of the determined quantization value.

  In step S303, the quantization unit 108 quantizes the main image based on the quantization value for the main image notified from the quantization control unit 109.

  In S304, the variable-length coding unit 110 performs variable-length coding on the quantized main image.

  The subsequent processing of S305 to S307 is repeatedly performed for each CU until encoding of all CUs of the sub-image is completed.

  In step S305, the quantization control unit 109 generates the entire sub-image based on the target code amount, the sum of the generated sub-image code amounts notified so far from the variable-length coding unit 110, and the number of remaining CUs. The quantization value for the sub image is determined so that the code amount falls within the target code amount. The quantization control unit 109 notifies the quantization unit 108 of the determined quantization value.

  In step S306, the quantization unit 108 quantizes the CU of the sub image based on the quantization value for the sub image notified from the quantization control unit 109 in step S305.

  In S307, the variable-length coding unit 110 performs variable-length coding on the CU of the sub-image quantized in S306, and notifies the quantization control unit 109 of the generated code amount.

  When the processing of S305 to S307 is performed for all CUs of the sub-image, encoding of the main image and the sub-image is completed.

  As described above, according to the first embodiment, the imaging apparatus 100 controls the quantization of the main image based on the target image quality, and controls the quantization of the sub-image based on the target code amount. This makes it possible to suppress an increase in the overall generated code amount.

  In the above description, the sub image is generated by reducing the size of the main image. However, the sub-image of the present embodiment is not limited to the reduced image of the main image, and any type of image generated based on the main image can be used as the sub-image.

  In the above description, the quantization of the main image (one image) is controlled based on the target image quality, and the quantization of the sub image (the other image) is controlled based on the target code amount. The relationship between the main image and the sub image may be reversed. That is, the imaging apparatus 100 may be configured to control the quantization of the sub image based on the target image quality and control the quantization of the main image based on the target code amount.

  Further, in the above description, as an example of quantization control based on the target code amount, a configuration has been described in which the generated code amount of the sub image is controlled not to exceed the target code amount. However, the quantization control based on the target code amount is not limited to this, and any quantization control in which the target code amount is considered from any point of view is adopted as the quantization control based on the target code amount in this embodiment. Can. For example, the imaging apparatus 100 may perform quantization control so that the generated code amount of the sub image is close to the target code amount.

Second Embodiment
In the second embodiment, a configuration will be described in which the target code amount of the sub image is determined based on the generated code amount of the main image. In the present embodiment, the basic configuration of the imaging device 100 is the same as that of the first embodiment (see FIG. 1). The differences from the first embodiment will be mainly described below.

  In S301 of FIG. 3, instead of acquiring the target code amount of the sub image from the encoding method setting unit 113, the quantization control unit 109 acquires a total target code amount obtained by combining the main image and the sub image. The total target code amount is, for example, determined in advance according to the required performance. For example, in the case where the processing time of encoding is determined based on constraints such as continuous shooting performance, etc., a predetermined code amount that meets the processing time is used as the total target code amount.

In S304 of FIG. 3, after performing variable-length coding on the main image, the variable-length coding unit 110 notifies the quantization control unit 109 of the generated code amount of the entire main image. Thereafter, the quantization control unit 109 determines the target code amount of the sub-image by subtracting the generated code amount of the entire main image from the total target code amount according to the following equation (1).
Sub_Target_Pic = Total_Target_Pic-Main_Pic (1)
Here, Sub_Target_Pic indicates the target code amount of the sub image, Total_Target_Pic indicates the total target code amount, and Main_Pic indicates the generated code amount of the entire main image.

  The other processes in FIG. 3 are the same as in the first embodiment. According to the encoding process of this embodiment, the generated code amount of the sub image is controlled such that the generated code amount of the sum of the main image and the sub image falls within the total target code amount (Total_Target_Pic). This makes it possible to suppress an increase in the overall generated code amount. In addition, in the case of CABAC, it becomes possible to suppress the performance deterioration associated with the increase in the generated code amount.

  FIG. 4 is a conceptual diagram of generated code amounts in the second embodiment. At the end of encoding of the main image, the generated code amount 401 of the main image is notified to the quantization control unit 109. The quantization control unit 109 determines the target code amount (Sub_Target_Pic) of the sub-image by subtracting the generated code amount 401 of the main image from the total target code amount (Total_Target_Pic) according to the above-mentioned equation (1). Then, the quantization control unit 109 performs quantization control of the sub image such that the generated code amount 402 of the sub image falls within the target code amount (Sub_Target_Pic) of the sub image. As a result, the sum of the generated code amount 401 of the main image and the generated code amount 402 of the sub image falls within the total target code amount (Total_Target_Pic).

  As described above, according to the second embodiment, the imaging device 100 determines the target code amount of the sub image based on the generated code amount of the main image. This makes it possible to suppress an increase in the overall generated code amount.

Other Embodiments
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

  100: imaging device, 105: reduction processing unit, 107: orthogonal transform unit, 108: quantization unit, 109: quantization control unit, 110: variable length coding unit, 111: stream buffer, 113: coding scheme setting unit , 118 ... multiplexing unit, 119 ... recording medium

Claims (13)

Encoding means for performing encoding processing involving quantization on a first image and a second image, wherein one of the first image and the second image is the first image and the second image. Encoding means, which is an image generated based on the other of the second images;
Control means for controlling the quantization of the first image based on a target image quality and controlling the quantization of the second image based on a target code amount;
An image coding apparatus comprising: The image coding apparatus according to claim 1, further comprising a determination unit that determines the target code amount based on a code amount of the first image generated by the encoding process. The image coding apparatus according to claim 2, wherein the determination means uses a value obtained by subtracting the code amount of the first image from a predetermined code amount as the target code amount. The control means controls the quantization of the first image so that a constant quantization value is used in the entire screen of the first image. The image coding apparatus as described in a term. The control means controls the quantization of the second image such that the code amount of the second image generated by the encoding process does not exceed the target code amount. The image coding apparatus according to any one of items 1 to 4. The image according to any one of claims 1 to 5, further comprising recording means for correlating and recording the first image and the second image subjected to the encoding process. Encoding device. The image coding apparatus according to claim 6, wherein the recording unit records the first image and the second image subjected to the coding process as one file. The image coding apparatus according to any one of claims 1 to 7, wherein the second image is an image generated based on the first image. The image coding apparatus according to claim 8, wherein the second image is an image generated by reducing the size of the first image. The image coding apparatus according to any one of claims 1 to 9, wherein the coding process includes context-based adaptive binary arithmetic coding (CABAC). An image coding apparatus according to any one of claims 1 to 10.
Imaging means for generating the other of the first image and the second image;
Generation means for generating the one of the first image and the second image based on the other of the first image and the second image;
An imaging apparatus comprising: An image coding method performed by an image coding apparatus, comprising:
An encoding process for performing encoding with quantization on a first image and a second image, wherein one of the first image and the second image is the first image and the second image. An encoding step, which is an image generated based on the other of the second images;
A control step of controlling the quantization of the first image based on a target image quality and controlling the quantization of the second image based on a target code amount;
An image coding method comprising:   The program for functioning a computer as each means of the image coding apparatus of any one of Claims 1-10.

Images