JP2021069094A - Image encoding apparatus, control method thereof and program - Google Patents

Abstract

To enable a reduction of a circuit scale by achieving efficient use of an encoding buffer in accordance with a compression ratio.SOLUTION: An image encoding apparatus, which encodes RAW image data, comprises: a buffer memory in which encoded data is temporarily stored; a plane conversion part for converting the RAW image data into a luminance plane composed of a luminance component and planes of a plurality of kinds of color difference components other than luminance; a wavelet conversion part for performing wavelet conversion on each plane obtained at the plane conversion part; an encoding part for quantizing and encoding a coefficient at each subband generated at the wavelet conversion part and storing obtained encoded data in the buffer memory; a recording part for recording the encoded data stored in the buffer memory on a recording medium; and a control part for controlling a ratio among reserved buffer sizes in the buffer memory for respective planes in accordance with a compression ratio of the RAW image data.SELECTED DRAWING: Figure 5

Description

The present invention relates to an image data coding technique.

Currently, digital imaging devices for recording moving images such as digital video cameras have become widespread, and in recent years, a method for recording RAW images has been applied not only to still images but also to moving images. Although the amount of data required for recording a RAW image is enormous, correction and deterioration of the original image can be minimized, and the degree of freedom in image editing after shooting is high. Therefore, RAW image recording is preferably used by advanced users among those who use the image pickup apparatus.

When recording a moving image of a RAW image, compression coding that compresses the amount of data is required so that a moving image for a certain period of time can be recorded on a predetermined recording medium. The RAW image is an image of a Bayer array in which pixels of each of the R, G, and B color components are arranged in a mosaic pattern. Since the adjacent pixels in the Bayer array have different color components, the correlation between the adjacent pixels is low. Therefore, it is difficult to obtain high compression efficiency even if the encoding is performed as it is. Therefore, only pixels having the same color component are extracted from the RAW image to generate a plurality of planes. Then, a plane conversion technique that enhances the correlation between pixels in the plane and improves the compression efficiency by performing coding for each plane is generally used as one of the compression coding methods.

Further, as a conventional typical compression coding method, H.I. 264 (H.264 / MPEG-4 Part10: Advanced Video Coding) is known. In this compression coding method, the amount of data is compressed by utilizing the time redundancy and spatial redundancy of the moving image for each block consisting of a predetermined number of pixels in one frame.

The above H. In 264, compression coding is realized by combining motion detection and motion compensation for time redundancy, discrete cosine transform (DCT) as frequency conversion for spatial redundancy, and technologies such as quantization and entropy coding. ing. However, if the compression ratio is increased to a certain extent or more, the block distortion peculiar to the DCT transform becomes remarkable, and the image deterioration becomes subjectively noticeable.

Therefore, a subband coding technology that uses the Discrete Wavelet Transform (DWT), which applies low-frequency and high-frequency filtering in the horizontal and vertical directions and decomposes into frequency bands called subbands, has been adopted in the JPEG2000 system and the like. Has been done. Compared with the coding technique using DCT, this subband coding is characterized in that block distortion is less likely to occur and the compression characteristics at the time of high compression are good.

For example, the technique described in Patent Document 1 converts RAW image data into a luminance plane composed of a luminance component and three planes other than the luminance. After that, the signal of each plane is further separated into frequency components by using DWT conversion, a plurality of subbands as shown in FIG. 1 are generated, and then each subband is encoded to efficiently perform a RAW image. The data is being compressed.

FIG. 1 shows the subbands generated when the wavelet transform is executed twice. When the wavelet transform is performed once, four subbands {LL, HL, LH, HH} are generated. Here, L represents a low frequency band and H represents a high frequency band. The first of the two letters of the alphabet corresponds to the horizontal band, and the second corresponds to the vertical band. And the number at the beginning of the subband represents the decomposition level. For example, 1HL represents (conversion coefficient) of the decomposition level “1” of the high frequency component in the horizontal direction and the low frequency component in the vertical direction. There is no limit to the number of wavelet transforms. Further, the second and subsequent wavelet transforms are recursively performed with respect to the subband LL obtained by the immediately preceding wavelet transform. Therefore, there is only one subband LL when the wavelet transform is performed a set number of times. Since FIG. 1 is an example in which the wavelet transform is performed twice, seven subbands are generated.

By compressing the RAW image data and reducing the data size by using the above technique or the like, it becomes possible to record for a longer time at the time of moving image recording.

JP-A-2017-85319

However, in the case of compression using subband coding by generating a luminance plane and a plane other than the luminance plane from the RAW image data as described in Patent Document 1, attention should be paid to the allocation of the coding buffer to each plane. There is a need.

Here, the storage of the coded data in the buffer memory at the time of moving image recording will be described with reference to FIGS. 2 (a) to 2'c). In FIG. 2A, RAW image data of a bayer array for 4 frames is acquired at the time of moving image recording, and the coded data generated by plane conversion, wavelet transform, and entropy coding is secured for each plane in the buffer memory. An example of storing in the encoding buffer is shown.

FIG. 2 (b) shows a case where the R, G1, G2, and B planes are separated from the RAW image data, and FIG. 2 (c) shows the RAW image data as U, which is a three-color difference other than Y (luminance) and brightness. , V, GH The case of the Y, U, V, GH plane obtained by interposing the conversion process to GH is shown. Further, FIGS. 2 (b) (i) and 2 (c) (i) show the coding buffer for each plane at the time of low compression, respectively, as shown in FIGS. 2 (b) (ii) and 2 (c) (ii). Shows the coding buffer for each plane at the time of medium compression, and FIGS. 2 (b) (iii) and 2 (c) (iii) show the coding buffer for each plane at the time of high compression. It should be understood that the pattern of the data stored in the buffer here corresponds to the pattern of the plane shown in FIG.

In the case of the R, G1, G2, and B planes shown in FIG. 2B, the coding buffer can be efficiently used by using the coding buffer having the same ratio regardless of the compression ratio. On the other hand, in the case of the Y, U, V, and GH planes shown in FIG. 2C, which will be described later, the luminance Y plane is more dominant in image quality than the other color difference planes. Is coded by allocating a larger amount of code than a plane other than luminance. Therefore, if the same ratio of coding buffers is assigned to each plane regardless of the compression rate, redundant coding buffers are given to the planes other than the luminance plane, and the circuit scale increases due to the increase in buffer memory. It ends up.

In view of the above problems, an object of the present invention is to provide an image coding apparatus capable of efficiently using a coding buffer according to a compression ratio and reducing the circuit scale.

In order to solve this problem, for example, the image coding apparatus of the present invention has the following configuration. That is,
An image coding device that encodes RAW image data.
A buffer memory for temporarily storing encoded data, and
A luminance plane composed of luminance components and a plane conversion means for converting RAW image data into planes of multiple types of color difference components other than luminance, and
A wavelet transforming means that performs wavelet transforming on each plane obtained by the plane transforming means,
A coding means that quantizes and encodes the coefficient of each subband generated by the wavelet transform means and stores the obtained coded data in the buffer memory.
A recording means for recording the coded data stored in the buffer memory on a recording medium, and
It has a control means for controlling the ratio of the buffer size for each plane secured in the buffer memory according to the compression rate of the RAW image data.

According to the present invention, redundant coding buffers can be reduced and an increase in circuit scale can be suppressed.

The figure which shows the example of the subband generation by the wavelet transform. The figure which shows the storage state of a coded buffer and coded data. The block block diagram of the image coding apparatus in 1st Embodiment. The figure which shows the storage state of the coding buffer in 1st Embodiment. The flowchart which shows the processing procedure of the image coding apparatus in 1st Embodiment. The block block diagram of the image coding apparatus in the 2nd Embodiment. The figure which shows the storage state when the code amount in all planes does not exceed a threshold value. The figure which shows the storage state when the code amount in a plane exceeds a threshold value. The figure which shows the relationship of the corresponding block line in each plane. The flowchart which shows the processing procedure of the target code amount calculation part in 2nd Embodiment. The flowchart which shows the processing procedure of the target code amount calculation part in 2nd Embodiment. The flowchart which shows the operation of the image pickup apparatus in the 2nd Embodiment. The flowchart which shows the operation of the image pickup apparatus in 3rd Embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate explanations are omitted.

[First Embodiment]
FIG. 3 shows a configuration when the image coding device of the first embodiment is applied to the image pickup device 300.

The imaging device 300 includes a CPU 301, an imaging unit 302, a plane conversion unit 303, a wavelet transform unit 304, a quantization unit 305, an entropy coding unit 306, a memory I / F 307, a buffer memory 308, a recording processing unit 309, and a recording medium 310. , And has a user interface 350. The user interface 350 includes a display device for displaying captured images and various menus to the user, and operation units such as switches, buttons, and a touch panel that receive various instruction inputs (including setting of compression rate) from the user. Will be done.

The CPU 301 has a memory for storing a control program executed by the CPU and for using it as a work area, and controls the entire processing of the image pickup apparatus 300.

The image pickup unit 302 includes an image pickup element such as a CCD image sensor or a CMOS image sensor that converts optical information into an electric signal. It is assumed that the image pickup elements are arranged in a Bayer arrangement. The image pickup unit 302 sends the image data of the Bayer array obtained by converting the electric signal obtained by the image sensor into a digital signal to the plane conversion unit 303 as RAW image data.

When focusing on adjacent 2 × 2 pixels in the Bayer array image, the breakdown is one R (red) component pixel, one B (blue) component pixel, and a diagonal positional relationship. It is composed of two G (green) component pixels in. Since there are two pixels of the G component, one is expressed as a G1 pixel and the other as a G2 pixel for convenience. The Bayer array is an array in which these 2 × 2 pixel patterns are repeatedly arranged.

The plane transform unit 303 separates four planes each composed of a single component from the RAW image data sent from the image pickup unit 302, and sends them to the wavelet transform unit 304 in a preset order. For example, the four planes are an R plane composed of only R component pixels, a G1 plane composed of only G1 component pixels, a G2 plane composed of only G2 component pixels, and a B component only pixel. It is a B plane composed of. Further, the plane conversion unit 303 uses, for example, the following equations (1) to (4) to obtain one Y (luminance component) plane and three color difference component planes (U plane, V plane) from the RAW image data. , And GH plane) may be derived.
Y = (R + B + G1 + G2) / 4… (1)
U = B- (G1 + G2) / 2… (2)
V = R- (G1 + G2) / 2… (3)
GH = G1-G2… (4)
The description continues assuming that the plane conversion unit 303 in the embodiment converts RAW image data into Y, U, V, and GH planes. The plane conversion unit 303 may implement a plurality of calculation formulas and select the calculation formula to be used in the CPU 301. Further, in the embodiment, U, V, and GH are taken as examples as the type of color difference, but the present invention is not limited to this, and other color differences may be used.

The wavelet transform unit 304 wavelet transforms each plane (each of which can be regarded as a single component image data) input by the plane transform unit 303 to generate a plurality of subbands. It is assumed that the number of wavelet transforms (number of decompositions) performed by the wavelet transform unit 304 is set by the CPU 301. Further, the wavelet transform unit 304 outputs the conversion coefficient for one line of each subband to the quantization unit 305 each time a conversion coefficient for one line is generated in each subband.

The quantization unit 305 uses the quantization parameter Qp to quantize the conversion coefficient of each plane input from the wavelet transform unit 304 in units of one coefficient. The quantization parameter Qp is a parameter in which the larger the value is, the smaller the value after quantization becomes and the amount of code can be reduced, but the deterioration of image quality becomes remarkable. Further, the conversion coefficient of each plane may be quantized for each plane or in parallel for all planes.

The entropy coding unit 306 performs entropy coding using a Huffman code, an arithmetic code, or the like for each subband quantized by the quantization unit 305, and generates coded data for each subband. Further, the entropy coding unit 306 temporarily stores the coded data of each subband in the buffer memory 308 based on the address specified by the CPU 301. The size of the coding buffer for each plane or subband is set by the CPU 301.

The memory I / F 307 arbitrates the buffer memory access request from each processing unit including the entropy encoding unit 306, and controls reading and writing to the buffer memory 308.

The buffer memory 308 is a storage area composed of a volatile memory for storing various data including the coded data of each subband sent from the entropy coding unit 306.

The recording processing unit 309 records the encoded data stored in the buffer memory 308 as a file on the recording medium 310 in accordance with the recording instruction from the CPU 301. The recording medium 310 is a recording medium composed of a non-volatile memory. The recording medium 310 is typically a removable recording medium represented by an SD card.

Subsequently, the allocation of the coded buffer on the buffer memory 308 according to the compression rate by the CPU 301 will be described with reference to FIG.

FIG. 4A shows frames of four consecutive RAW images. FIG. 3B shows the accumulation state of coded data when the Y, U, V, and GH planes are separated from the RAW image and the buffer sizes are assigned to each of them at the same ratio. FIG. 3C shows the accumulation state of coded data when the Y, U, V, and GH planes are separated from the RAW image and the buffer size is assigned to each of them at a ratio according to the code amount. In order to make it easy to understand the correspondence between the coded data stored in the buffer and the four RAW images, each is represented by a corresponding pattern.

Further, in the embodiment, when the encoded data for four consecutive planes is stored in the buffer, or even if there is only one plane in which the ratio of the encoded data stored in the buffer reaches the set threshold value. For example, it will be described as assuming that the coded data of all planes is recorded in a file.

In the present embodiment, it is assumed that the code amount ratio of each plane is statistically obtained for each compression rate from a large amount of image data in advance, and the following code amount ratio can be obtained. The code amount ratio of each plane according to the compression rate is not limited to this, but the user operates the user interface 150 to select one of the three levels of compression rate (code amount ratio). It shall be. It should be noted that the higher the compression ratio, the smaller the code amount.

<Statistically determined code amount ratio of each plane for each compression rate>
・ For low compression
Y: U: V: GH = 1: 1: 1: 1… (5)
・ For medium compression
Y: U: V: GH = 1.2: 1: 1: 1… (6)
・ For high compression
Y: U: V: GH = 1.4: 1: 1: 1… (7)
In FIG. 4B, it is assumed that the size ratio of the coding buffer between the planes is the same regardless of the compression rate and the coding is performed, and the coding amount at that time is the coding amount ratio for each plane based on the statistical data. Shows the case. At this time, as shown in FIG. 4B, the higher the compression ratio, the higher the coding buffer usage rate of the Y (luminance) plane as compared with the color difference plane other than the luminance plane, and the coding buffer in the plane other than the luminance plane. It shows that it has been secured redundantly.

Therefore, in the present embodiment, as shown in FIG. 4C, the ratio of the coded buffer and the size of each plane secured in the buffer memory 308 is changed according to the compression rate. Here, it is assumed that the coding buffer ratio is assigned as follows.

<Assignment of coding buffer ratio of each plane according to compression rate>
・ For low compression
Y: U: V: GH = 1: 1: 1: 1… (8)
・ For medium compression
Y: U: V: GH = 1.2: 1: 1: 1… (9)
・ For high compression
Y: U: V: GH = 1.4: 1: 1: 1… (10)
By setting the coding buffer ratio based on statistics at each compression rate as described above, in the case of FIG. 4 (c), the planes other than the luminance are redundant as compared with the case of FIG. 4 (b). It is possible to reduce the size of the coded buffer.

So far, the allocation of the coding buffer for each plane has been described. Next, focusing on one of the four planes after plane separation, the allocation ratio to each subband obtained from the plane of interest will be described.

In the present embodiment, the coding buffer for each subband shall be allocated based on the area ratio of each subband.

Subband division by the discrete wavelet transform halves the number of horizontal and vertical coefficients each time the decomposition level increases by one level, resulting in a quarter of the area.

Therefore, each size of the subband {2LL, 2HL, 2LH, 2HH} has a one-fourth relationship with each size of the subband {1HL, 1LH, 1HH}. From the above, the coding buffer is assigned to each subband as follows.

<Code-coded buffer ratio for each subband>
2LL: 2HL: 2LH: 2HH: 1HL: 1LH: 1HH = 1: 1: 1: 1: 1: 4: 4: 4… (11)
The allocation of the buffer memory of each subband is constant regardless of the compression rate. Further, although one of the four planes has been described here, the same allocation is made for the other planes.

Subsequently, the details of the operation of the image coding apparatus according to the first embodiment will be described with reference to the flowchart shown in FIG. As the operation here, the encoded RAW image data is stored in the buffer memory 308 and the encoded RAW image data already stored in the encoded buffer is recorded in the recording medium 310. Are running at the same time.

First, in S501, the CPU 301 determines the mode of the compression rate related to shooting set by the user interface 350. Subsequently, in S502, the CPU 301 determines the allocation of the coding buffer for each plane in the buffer memory 308 based on the determined compression ratio setting.

Next, in S503, the CPU 301 allocates the buffer size of each subband based on the area ratio of each subband to the coded buffer size assigned to each plane, and outputs these to the output of the entropy coding unit 306. Set as the destination.

Next, in S504, the CPU 301 starts driving the imaging unit 302. Then, the CPU 301 also starts driving the plane conversion unit 303, the wavelet conversion unit 304, the quantization unit 305, and the entropy coding unit 306, generates coded data for each of a plurality of subbands, and writes to the buffer memory 308. Let me do it.

Next, in S505, the CPU 301 completes the coding of one frame.

Then, in S506, the CPU 301 determines whether or not a recording stop instruction has been received from the user, and if so, whether or not the recording of the coded data in the buffer memory of the final frame on the recording medium 310 has been completed. If the determination is made and the recording of the final frame is completed, this process is terminated. Further, when the recording is continued, the CPU 301 returns the process to S504 and encodes the frame of.

As described above, according to the first embodiment, it is possible to deal with the bias of the generated code amount between the planes due to the compression rate, and by efficiently using the buffer memory, coding can be performed without increasing the circuit scale. It becomes.

In the image coding apparatus of the first embodiment, the coding ratio of each plane is changed for each compression rate, but the ratio of the coding buffer may be changed according to the frame rate as well as the compression rate. Good. That is, the coding buffer ratio is determined by using the relationship that the compression rate increases as the frame rate increases. Further, in the above description, the number of stages of the compression rate that can be selected by the user is described as "3", but this number is not particularly limited.

Further, since the allocation to each subband is not performed based on the actually generated coded data size, it is desirable to have a margin as the size of the buffer memory. Regarding the margin at this time, an area corresponding to the coding buffer ratio is secured in each plane.

[Second Embodiment]
In the first embodiment, the coding buffer ratio between the planes is changed according to the compression rate in view of the fact that the code amount ratio between the planes differs depending on the compression rate. In the second embodiment, as compared with the first embodiment, the case where the code amount is still biased between the planes even if the coding buffer ratio of each plane is changed according to the compression rate is dealt with. , Code amount control will be described.

Also in the second embodiment, an example applied to an image pickup device as an image coding device will be described. FIG. 6 is a block configuration diagram of the image pickup apparatus 600 according to the second embodiment.

The image pickup device 600 includes a CPU 601 and an imaging unit 602, a plane conversion unit 603, a wavelet conversion unit 604, a quantization unit 605, an entropy coding unit 606, a buffer usage rate calculation unit 607, a memory I / F 608, and a buffer memory 609. It has a recording processing unit 610, a recording medium 611, a code amount control unit 612, a generated code amount holding unit 613, a difference calculation unit 614, a target code amount calculation unit 615, a quantization control unit 616, and a user interface 650.

Here, the points different from the first embodiment in the apparatus configuration of the second embodiment will be described. The buffer usage rate calculation unit 607 calculates the usage rate of the coded buffer of each plane in the buffer memory 609 after the coding of one frame is completed. At that time, the calculated coding buffer usage rate of each plane is as shown in FIG. 7 with respect to the preset threshold values (TH (Y), TH (U), TH (V), TH (GH)). There are cases where all planes do not exceed, and cases where the code amount exceeds even one plane as shown in FIGS. 8A to 8D. 8 (a) shows the luminance Y plane, FIG. 8 (b) shows the color difference U plane, FIG. 8 (c) shows the color difference V plane, and FIG. 8 (d) shows the code amount threshold value in the color difference GH plane. It shows the case of exceeding. As a method of setting the threshold value, for example, assuming that a margin of 20% of the coding buffer is taken with respect to the original buffer ratio, the amount of the coding buffer at this time is 120% including the margin. Therefore, the threshold value set in each plane is defined as the case where a code amount larger than 90% of them is used, and 120% × 0.9 = 108% is defined as the threshold value.

In the present embodiment, the control method of the code amount control unit 612 is changed depending on whether or not the usage rate of the coded buffer of each plane calculated by the buffer usage rate calculation unit 607 exceeds the threshold value. The explanation will be described later. The threshold value set in each plane is arbitrary. The allocation of the coding buffer of each plane at this time is as shown below.

<Assignment of coding buffer ratio for each plane>
Y: U: V: GH = 1.2: 1: 1: 1… (12)

First, a control method of the code amount control unit 612 when the code amounts of all planes do not exceed the threshold value will be described.

<Control unit>
As described above, the coding unit is a line for each subband, but the quantization control unit is a set of coding results of each subband at the same pixel position. That is, as shown in FIG. 9, one line of the decomposition level 2 subband {2LL, 2LL, 2HL, 2HH} and two lines of the decomposition level 1 subband {1LH, 1HL, 1HH} are used once for Qp. It is a control unit by. Hereinafter, this control unit will be referred to as a block line. Also, let i be the block line number. One block line corresponds to 16 lines of input RAW image data.

The generated code amount holding unit 613 holds the generated code amount of the coded data notified from the entropy coding unit 606. The target code amount calculation unit 615 calculates the target code amount of the frame when it is determined that it is the timing to control the Qp of the head block line of each frame. The details of the target code amount calculation method at that time will be described later. The difference calculation unit 614 calculates the difference between the generated code amount for each block line and the target code amount for each block line, and further calculates the integrated difference amount which is the integrated value of the difference. Further, it is assumed that the integrated difference amount is initialized to "0" at the timing of controlling the Qp of the first block line of the frame.

The quantization control unit 616 calculates (updates) Qp based on the integrated difference amount notified from the difference calculation unit 614 (details will be described later). The code amount is controlled by the above.

<Quantization value calculation>
As one of the quantization parameter calculation methods, there is a known technique shown in MPEG2 Test Model 5. According to this Test Model 5, the block line of interest is the i-th, and the integrated difference amount calculated by the difference calculation unit 614 up to immediately before the block line of interest is ΣE [i-1], the reference quantization parameter Qp_ref, and the control sensitivity ". When defined as "r", the quantization parameter Qp [i] of the block line of interest can be calculated as follows.
Qp [i] = Qp_ref ＋ r × ΣE [i-1]… (13)
Note that Qp_ref is the initial quantization parameter Qp_ini at the beginning of the frame, and is the Qp of the final block line of the immediately preceding frame in other regions. This is a process performed by ΣE [i-1] to ensure the continuity of Qp, and is initialized at the beginning of the frame. Further, the control sensitivity "r" is a parameter that sharply fluctuates Qp as it is larger, while improving the controllability of the code amount.

By using Test Model 5, it is possible to control the code amount by setting the quantization parameter to be large if the generated code amount is large with respect to the target code amount and to set the quantization parameter to be small if it is small.

The quantization control unit 616 uses the quantization parameter Qp [i] of the block line of interest determined by using Eq. (13) for the actual quantization in each subband, and the subband Qp (= Qp [pl] [ It is further converted to sb]) and notified to the quantization unit 605. Note that pl and sb indicate the corresponding plane and the corresponding subband, respectively. The quantization control unit 616 applies the quantization parameter Qp [i] of the block line of interest determined by the equation (13) to the matrix mtx having each preset plane or subband as shown in the equation (14). By doing so, the Qp for each subband is calculated.
Qp [pl] [sb] = Qp [i] × mtx [pl] [sb]… (14)
In general, by setting the Qp larger for the high-frequency subband and the Qp smaller for the low-frequency subband and controlling the code amount, it is difficult to visually recognize due to human visual characteristics, and the high-frequency component of the image data is generated. The amount of code is compressed to improve the coding efficiency. Therefore, the matrix is set so that the higher the subband, the larger the Qp, and the lower the subband, the smaller the Qp.

<Calculation of target code amount>
<Processing flow>
The target code amount calculation unit 615 is driven when determining the Qp at the beginning of the frame, and calculates the target code amount of one frame. As a method of calculating the target code amount of each frame, the sum of the value obtained by calculating the target code amount per frame from the set bit rate and the difference value obtained by subtracting the generated code amount from the target code amount in the previous frame is added to each frame. Let it be the target code amount of.

FIG. 10 is a flowchart showing a processing procedure of the target code amount calculation unit 615. Hereinafter, a method of calculating the target code amount for each block line by the target code amount calculation unit 615 will be described with reference to the figure.

In S1001, the target code amount calculation unit 615 calculates the remaining amount of code amount. The remaining code amount is a value obtained by subtracting the sum of the generated code amounts of the coded block lines from the target code amount of the frame of interest.

In S1002, if the remaining amount of code is 0 or less, the target code amount calculation unit 615 clips the remaining amount of code to 0 and performs the subsequent processing. This process is carried out in order to prevent the target code amount from becoming negative.

In S1003, the target code amount calculation unit 615 calculates the target code amount for each block line. The target code amount for each block line is calculated by the remaining amount of code / the number of uncoded block lines. From the above, the target code amount for each block line is determined.

Next, a control method of the code amount control unit 612 when there is even one plane whose code amount exceeds the preset threshold value will be described.

In this case, the control method of the code amount control unit 612 is not the code amount control for each frame but independently for each plane in order to prevent the code amount of the coding buffer for the plane exceeding the threshold value from being further increased. Code amount control.

<Control unit>
The control unit is the block line unit described above. The generated code amount holding unit 613 holds the generated code amount which is the coded data notified from the entropy coding unit 606 for each plane. The target code amount calculation unit 615 calculates the target code amount of each plane when it is determined that it is the timing to control the Qp of the head block line. The details of the target code amount calculation method in each plane at that time will be described later. The difference calculation unit 614 calculates the difference between the generated code amount for each block line of each plane and the target code amount for each block line of each plane, and further calculates the integrated difference amount which is the integrated value of the difference. Further, the integrated difference amount is initialized to 0 at the timing of controlling the Qp of the leading block line of each plane. The quantization control unit 616 calculates Qp based on the integrated difference amount notified from the difference calculation unit 614 (details will be described later). The code amount is controlled by the above.

<Quantization value calculation>
The quantization control unit 616 uses the integrated difference amount ΣE [i-1] calculated by the difference calculation unit 614 for each block line of each plane and the reference quantization parameters Q _Yini , Q _Uini , Q _Vini , and Q _{G Hini of each plane.} And, using the control sensitivities r _Y , r _U , r _V , and r _GH of each plane, the quantization parameters of the block line of interest for each plane Q _Y [i], Q _U [i], Q _V [i], Q _GH [i] is calculated as follows.
Q _Y [i] = Q _Yini + r _Y × ΣE [i-1]… (15)
Q _U [i] = Q _Uini + r _U × ΣE [i-1]… (16)
Q _V [i] = Q _Vini + r _V × ΣE [i-1]… (17)
Q _GH [i] = Q _GHini + r _GH × ΣE [i-1]… (18)
However, in consideration of the color balance, the quantization parameters of the U plane and V plane are _{compared in magnitude between Q U} [i] and Q _V [i], and the larger quantization parameter is used as a common quantization parameter. To do.

The quantization control unit 616 uses the quantization parameter Qp of the block line of interest of each plane determined according to Eqs. (15) to (18) as the subband parameter Qp {Qp [] that is actually used for quantization in each subband. It is further converted into Y] [sb], Qp [U] [sb], Qp [V] [sb], Qp [GH] [sb]) and notified to the quantization unit 605. Here, "sb" indicates the corresponding subband. The quantization control unit 616 uses the matrix mtx having the block line quantization parameters determined by the equations (15) to (18) for each subband preset in each plane, and the quantization control units 616 use the equations (19) to (22). ) To calculate the subband quantization parameter Qp.
Qp [Y] [sb] = Q _Y [i] × mtx [sb]… (19)
Qp [U] [sb] = Q _U [i] × mtx [sb]… (20)
Qp [V] [sb] = Q _V [i] × mtx [sb]… (21)
Qp [GH] [sb] = Q _GH [i] × mtx [sb]… (22)

<Calculation of target code amount>
<Processing flow>
The target code amount calculation unit 615 is driven when determining the Qp at the beginning of the frame, and calculates the target code amount of each plane. Here, as a method of calculating the target code amount for each plane, for example, for each plane regarding the case where the code amount of the Y plane shown in FIG. 8A exceeds a preset threshold value (TH (Y)). The calculation of the target code amount will be described.

Here, the generated code amount of the Y plane of the frame of interest is Y (S _n ), the generated code amount of the U plane is U (S _n ), the generated code amount of the V plane is V (S _n ), and the generated code of the GH plane. _{Let the} quantity be GH (S n) and the target code quantity of the frame of interest be T _n . At this time, the following relationship holds.
T _n = Y (T _n ) + V (T _n ) + U (T _n ) + GH (T _n )… (23)

Further, with respect to the frame of interest, the generated code amount of the Y plane in the previous frame is Y (S _n-1 ), the generated code amount of the U plane is U (S _n-1 ), and the generated code amount of the V plane is V. (S _n-1 ), the generated code amount of the same GH plane is GH (S _n-1 ), and the target code amount of the previous frame is T _n-1 .

First, the target code amount Y (T _n ) of the Y plane in the frame of interest does not increase the coding buffer of the Y plane any more, and the maximum code amount is allocated among them (Equation (12)). Set as follows in consideration of the coding buffer ratio for each plane indicated by.
Y (T _n ) = T _n × 1.2 / (1.2 + 1 + 1 + 1)… (24)

Similarly, it is possible to set the target code amount for planes other than the Y plane by using Eq. (12), but the code is coded in frame units until then, rather than allocating the code amount by the coded buffer ratio. The image quality is more stable if the code amount is followed when the amount is controlled. Therefore, in addition to the Y plane, the target code amount is set for the frame of interest in consideration of the code amount of the previous frame.

Here, the difference between the generated code amount Y (S _n-1 ) of the Y plane of the previous frame and the code amount when the Y plane is assigned to the Y plane according to the coding buffer ratio with respect to the frame of interest is the difference between the Y planes. If the surplus code amount ΔY _n is defined, ΔY _n can be defined as follows.
ΔY _n = Y (S _n-1 ) -T _n-1 × 1.2 / (1.2 + 1 + 1 + 1)… (25)

The target code amount is set so that the surplus code amount ΔY _n of the Y plane is assigned to the target code amounts of the U plane, the V plane, and the GH plane, which are planes other than the Y plane in the frame of interest. The order of priority for allocating the code amount for the surplus code amount is Y plane> U plane = V plane> GH plane. For example, what is normally assigned as Y: U: V: GH = 1: 1: 1: 1, is assigned as much as the surplus code amount as Y: U: V: GH = 3: 2: 2: 1. Change the ratio. However, the allocation ratio for each plane is not limited to this. Here, since planes other than the Y plane are targeted, in the present embodiment, the code amount allocation ratio is changed in the relationship of U: V: GH = 2: 2: 1 for _{the surplus code amount ΔY n of the Y plane.}

Based on the above, the target code amount U (T _{n ) of the U plane in the frame of interest is given} by the following equation (26) based on U (S _n-1 ), ΔY n, T _n , and T _n-1. .. At this time, the target code amount T _n of the frame of interest and the target code amount T _n-1 of the previous frame are not necessarily the same because the code amount is controlled. Therefore, the frame of interest is scaled and calculated as follows.

Further, the target code amount V (T _n ) of the V plane in the frame of interest is calculated by the following equation (27) based on V (S _n-1 ), ΔY _n , T _n , and T _n-1. .. At this time, scaling is performed in the same manner as the U plane.

Further, the target code amount GH (T _n ) of the GH plane in the frame of interest is calculated by the following equation (28) based on GH (S _n-1 ), ΔY _n , T _n , and T _n-1. To. At this time, scaling is performed in the same manner as the U plane and the V plane.

Here, the case where the code amount of the Y plane exceeds the preset threshold value has been described, but the target code amount is set in the same way when the code amount of the Y plane exceeds the threshold value set in advance or when the code amount of the Y plane exceeds the threshold value. decide.

FIG. 11 is a flowchart showing a processing procedure of the target code amount calculation unit 615. Hereinafter, a method of calculating the target code amount for each block line of the target code amount calculation unit 615 will be described with reference to the figure.

In S1101, the target code amount calculation unit 615 calculates the remaining amount of code amount. The code amount remaining amount is a value obtained by subtracting the sum of the generated code amounts of the coded block lines for each plane from the target code amount for each plane of the frame of interest.

In S1102, if the remaining amount of code is 0 or less, the target code amount calculation unit 615 clips the remaining amount of code to 0 and performs the subsequent processing. This process is carried out in order to prevent the target code amount from becoming negative.

In S1103, the target code amount calculation unit 615 calculates the target code amount for each plane for each block line. The target code amount for each block line of each plane is calculated by the remaining amount of code amount of each plane / the number of uncoded block lines. From the above, the target code amount for each block line is determined.

Subsequently, the details of the operation related to the image coding in the image pickup apparatus according to the second embodiment will be described with reference to the flowchart shown in FIG. As the operation here, the storage of the encoded RAW image data in the buffer memory 609 and the recording of the coded data already stored in the coding buffer in the recording medium 611 are executed at the same time. And.

First, in S1201, the CPU 601 determines the mode of the compression rate related to shooting set by the user interface 650. Subsequently, in S1202, the CPU 601 determines the allocation of the coding buffer for each plane based on the determined compression ratio setting. Next, in S1203, the CPU 601 allocates the coded buffer size assigned to each plane based on the area ratio of each subband, and sets it as the output destination of the entropy coding unit 606.

Next, in S1204, the CPU 601 starts driving the imaging unit 602. Then, the CPU 601 also starts driving the plane conversion unit 603, the wavelet conversion unit 604, the quantization unit 605, and the entropy coding unit 606 to generate coded data for each of a plurality of subbands and write them to the buffer memory 308. To do.

Next, in S1205, the CPU 601 completes the coding of one frame. Then, in S1206, the CPU 601 determines whether or not the user has instructed to stop recording, and if so, whether or not the recording of the coded data in the buffer memory of the final frame on the recording medium 611 is completed. judge. When the recording of the final frame is completed, the CPU 601 ends this process. If the recording is continued, the CPU 601 advances the process to S1207.

In S1207, the CPU 601 causes the buffer usage rate calculation unit 607 to calculate the coded buffer usage rate of each plane. Next, in S1208, the CPU 601 determines whether or not the code amount exceeds the threshold value in each plane. If the coded buffer usage exceeds a preset threshold, the CPU 601 proceeds to S1209. In this S1209, the CPU 601 determines the target code amount in plane units and controls the code amount in plane units in order to control the code amount so as not to increase the coding buffer of the plane whose code amount exceeds the threshold value. To do. At this time, the target code amount of the plane whose code amount is exceeded is reduced, and the code amount of that amount is allocated to the other planes. The priority for allocating the reduced code amount is Y plane> U plane = V plane> GH plane.

If the CPU 601 determines in S1208 that the code amount of all planes is less than the threshold value, the process proceeds to S1210. In this S1210, the CPU 601 determines a target code amount for each frame and controls the code amount for each frame.

As described above, by performing the processing of the second embodiment, the circuit scale can be suppressed without increasing the coding buffer even when the coding amount is biased between the planes and the coding amount is likely to exceed the coding buffer. Can be encoded.

[Third Embodiment]
Subsequently, the third embodiment will be described. In the second embodiment, it has been described that the code amount control is changed when the code amount exceeds the preset threshold value of the coded buffer usage rate of each plane. In the third embodiment, a case where the coding ratio for each plane exceeds a predetermined value or more with respect to the buffer ratio set in advance as shown in the equation (12) will be described. The apparatus configuration is the same as that of FIG. 6 of the second embodiment, and the description thereof will be omitted.
The processing content related to image coding according to the third embodiment will be described with reference to the flowchart shown in FIG. As the operation here, the storage of the encoded RAW image data in the buffer memory 609 and the recording of the encoded RAW data already stored in the coding buffer in the recording medium 611 are executed at the same time. It is assumed that there is.

First, in S1301, the CPU 601 determines the mode of the compression rate related to shooting set by the user interface 650. Subsequently, in S1302, the CPU 601 determines the allocation of the coding buffer for each plane based on the determined compression ratio setting. Next, in S1303, the CPU 601 allocates the coded buffer size assigned to each plane based on the area ratio of each subband, and sets it as the output destination of the entropy coding unit 606. Since the buffer size between subbands is the area ratio, it is also a fixed ratio.

Next, in S1304, the CPU 601 starts driving the imaging unit 602. Then, the CPU 601 also starts driving the plane conversion unit 603, the wavelet conversion unit 604, the quantization unit 605, and the entropy coding unit 606 to generate coded data for each of a plurality of subbands and write them to the buffer memory 308. To do.

Next, in S1305, the CPU 601 completes the coding of one frame. Then, in S1306, the CPU 601 determines whether or not the user has instructed to stop recording, and if so, whether or not the recording of the coded data in the buffer memory of the final frame on the recording medium 611 is completed. judge. When the recording of the final frame is completed, the CPU 601 ends this process. If the recording is continued, the CPU 601 advances the process to S1307.

In S1307, the CPU 601 causes the buffer usage rate calculation unit 607 to calculate the coded buffer usage rate of each plane.

Next, in S1308, the CPU 601 calculates the code amount ratio of each plane from the generated code amount of each plane input from the generated code amount holding unit 613 to the target code amount calculation unit 615. Further, the target code amount ratio of each plane in the frame of interest stored in advance in the target code amount calculation unit 615 is compared with the code amount ratio of each plane based on the generated code amount, and whether or not even one plane exceeds a predetermined value or more. Is determined. The predetermined value here is arbitrary. Further, instead of changing from the code amount control for each frame to the code amount control for each plane from the next frame when the predetermined value or more is exceeded, the code amount control may be changed when the code amount is exceeded n times in a row. In addition, n is an arbitrary setting.

If the coding ratio for each plane in the buffer usage rate is equal to or higher than a predetermined value, the CPU 601 proceeds to S1309. In this S1309, the CPU 601 determines the target code amount in plane units and controls the code amount in plane units so that a large code amount does not occur with respect to the preset buffer ratio, as in the second embodiment. I do.

On the other hand, when it is determined that the ratio of the buffer usage rates of all planes is less than a predetermined value, the CPU 601 proceeds to S1310. In this S1310, the CPU 601 determines a target code amount for each frame and controls the code amount for each frame, as in the second embodiment.

According to the image coding procedure of the third embodiment, it is possible to reduce the circuit scale without increasing the coding buffer even when the generated code amount ratio is biased with respect to the coding buffer ratio. Become.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

300 ... Imaging device, 301 ... CPU, 302 ... Imaging unit, 303 ... Plain converter, 304 ... Wavelet transform, 305 ... Quantization unit, 306 ... Entropy encoding unit, 307 ... Memory I / F, 308 ... Buffer memory , 309 ... Recording processing unit, 310 ... Recording medium

Claims (14)

An image coding device that encodes RAW image data.
A buffer memory for temporarily storing encoded data, and
A luminance plane composed of luminance components and a plane conversion means for converting RAW image data into planes of multiple types of color difference components other than luminance, and
A wavelet transforming means that performs wavelet transforming on each plane obtained by the plane transforming means,
A coding means that quantizes and encodes the coefficient of each subband generated by the wavelet transform means and stores the obtained coded data in the buffer memory.
A recording means for recording the coded data stored in the buffer memory on a recording medium, and
An image coding apparatus comprising: a control means for controlling the ratio of the buffer size for each plane secured in the buffer memory according to the compression rate of the RAW image data. The image coding according to claim 1, wherein the control means increases the ratio of the buffer size of the luminance plane to the plurality of types of color difference planes as the compression ratio of the RAW image data increases. apparatus. When the compression rate of the RAW image data is a second compression rate higher than that of the first compression rate, the control means has the plurality of types of color difference planes as compared with the case of the first compression rate. The image coding apparatus according to claim 1, wherein the ratio of the buffer size of the luminance plane to the brightness plane is increased. The image coding apparatus according to any one of claims 1 to 3, further comprising a setting means for setting a compression rate of the RAW image data according to a user's operation. The control means includes a usage rate calculation means for calculating the usage rate of each plane in the buffer memory.
The image coding apparatus according to claim 1, wherein the unit for controlling the code amount of the frame of interest is changed according to the buffer usage rate of each plane in the frame preceding the frame of interest obtained by calculation. The control means
When the buffer usage rate is less than a predetermined value in all planes in the previous frame, the target code amount of the frame of interest is determined and the code amount is controlled in frame units.
The fifth aspect of claim 5, wherein when the usage rate of the buffers of one or more planes in the previous frame exceeds a predetermined value, the target code amount in the plane unit is determined and the code amount is controlled in the plane unit. Image encoding device. The control means
When the coding ratio for each plane to the buffer usage of all planes in the previous frame is less than a predetermined value, the target code amount of the frame of interest is determined and the code amount is controlled for each frame.
Claim 5 is characterized in that when the coding ratio for each plane to the buffer ratio of one or more planes in the previous frame is equal to or more than a predetermined value, the target code amount for each plane is determined and the code amount is controlled for each plane. The image encoding device according to. The control means controls the code amount of the coding means in units of planes, subtracts the target code amount of the plane in the frame of interest, and applies the reduced code amount to the planes other than the plane in the frame of interest. The image coding apparatus according to any one of claims 1 to 7, wherein the image coding apparatus is assigned to each plane at a predetermined ratio. The image coding apparatus according to claim 8, wherein the predetermined ratio makes the luminance plane higher than the other color difference planes. The plane conversion means converts the RAW image data of the Bayer array into one luminance plane and three color difference planes.
The image coding apparatus according to claim 8, wherein the predetermined ratio is the same ratio for two of the three color difference planes. The image coding apparatus according to any one of claims 1 to 10, wherein the control means controls the ratio of the buffer size corresponding to each plane according to the compression rate according to the frame rate. The image coding apparatus according to any one of claims 1 to 11, wherein the control means fixes the buffer size of each subband generated from one plane. It is a control method of an image coding device that has a buffer memory for temporarily storing coded data and encodes RAW image data.
A luminance plane composed of luminance components and a plane conversion process for converting RAW image data into planes of multiple types of color difference components other than luminance components.
A wavelet transform step of performing wavelet transform on each plane obtained in the plane transform step, and a wavelet transform step.
A coding step in which the coefficients of each subband generated in the wavelet transform step are quantized and encoded, and the obtained coded data is stored in the buffer memory.
A recording step of recording the coded data stored in the buffer memory on a recording medium, and
A control method for an image coding apparatus, comprising a control step of controlling the ratio of the buffer size for each plane secured in the buffer memory according to the compression rate of the RAW image data. A program for causing the computer to execute each step of the method according to claim 13, which is read and executed by the computer.

Images