JP2020072390A - Image encoding device, control method thereof, and program - Google Patents

Abstract

To provide an image encoding device capable of generating a compressed RAW image in an image recording format capable of displaying reduced-size images and performing streaming reproduction at high speed.SOLUTION: An image encoding device 100 includes: a plane conversion unit 105 that converts a RAW image into plane images of color components; a wavelet transform unit 106 that performs hierarchical frequency decomposition on the plane images to generate a plurality of subbands; a quantization unit 107 that compresses and encodes the plurality of subbands to generate a plurality of compressed subbands; an entropy coding unit 108; and a recording processing unit 111 that collectively arranges the plurality of compressed subbands and compressed subband information for each group based on the number of layers in frequency decomposition, and generates a recording file including a compressed RAW area in which the plurality of compressed subbands arranged are stored, and an information management area in which information on the plurality of compressed subbands is stored.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image encoding device, a control method thereof, and a program, and more particularly, to an image encoding device that compresses a RAW image, a control method thereof, and a program.

  In recent years, due to the increase in the number of pixels of CMOS image sensors, the number of recorded pixels per frame of moving images by digital cameras and digital camcorders has reached higher areas such as 6K and 8K from the areas of conventional standards such as Full HD and QFHD. is doing.

  Along with this, the data size per image has increased, and the processing amount in the developing process for generating the display image from the RAW image output from the image sensor has increased, leading to the output of the display image. There is a problem that it takes time.

  Further, when transmitting a display image or a RAW image itself from an imaging device such as a digital camera to a device such as a display for display, the band of the transmission path is pressed more than ever. In some cases, there are problems in that the bandwidth of the transmission path is insufficient in displaying an image in real time, the image is delayed, and dropped frames occur.

  These problems affect streaming reproduction in which an image is sent to an external terminal via a network such as the Internet and the image is displayed on the external terminal side. It also affects the generation of a reduced image used when displaying the image of the subject as a preview on the back monitor of the digital camera for shooting assistance.

  In streaming playback or preview, when displaying an image on the screen of a smartphone, which is an external terminal, or on the rear monitor of a digital camera, it is possible to display a reduced image obtained by reducing the image because the resolution is smaller than the captured image. It is common.

  However, when the development processing or the reduction processing is performed from the RAW image, as described above, the processing amount of the development processing is very large, and a delay occurs until the reduced image is displayed.

  Therefore, reduction processing is performed after RGB multiplexing is performed on the output signal from each pixel of the image sensor in which the R (red), G (green), and B (blue) color filters are arranged in the Bayer array. Japanese Patent Application Laid-Open No. 2004-242242 describes a method of making a reduced image and displaying it.

  On the other hand, recently, a technique for compressing and recording a RAW image acquired from an image sensor has been known.

  The reason why compressed RAW images have come to be handled is that the size of the RAW image to be recorded has increased due to the above-described flow of increasing the number of pixels, and the cost of recording media has also increased. Is mentioned.

  In addition, in order to have resistance to image editing by post-production such as color correction in movie production, there is an increasing need to handle RAW images that have all the information of the image sensor that was thrown away by ordinary photographing equipment. However, on the other hand, the data size of the RAW image is further enlarged.

  Furthermore, during streaming playback, the display image is sent in the form of a compressed RAW image instead of the RAW image to the external terminal, and after being developed and reduced on the external terminal side, it is displayed, so that the image is sent to the external terminal. The required transfer bit rate is low. For this reason, it can be expected to prevent dropped frames from being displayed due to transfer delay.

  For example, in the technique described in Patent Document 2, the compressed RAW image is generated from the RAW image after the sub-sampling is performed for each color component from the RAW image from the imaging unit 103 to generate a plane image as shown in FIG. This is performed by compressing (encoding) each plane image.

Japanese Patent No. 6049481 JP, 2003-125209, A

  However, since the processing time of the reduction processing increases as the number of pixels of the image pickup element increases, even if the technique described in Patent Document 1 is used, a delay occurs until the reduced image is displayed on the image pickup apparatus. It is possible that it will end up.

  Further, when performing streaming playback of a compressed RAW image generated by the technique described in Patent Document 2, decompression processing of the compressed RAW image is also required in the process of displaying the reduced image, and the processing amount further increases, so that the streaming playback is delayed. Is likely to occur.

  In view of the above problems, the present invention provides an image encoding device capable of generating a compressed RAW image in an image recording format capable of displaying a reduced image and streaming reproduction at high speed, a control method thereof, and a program. The purpose is to do.

  An image coding apparatus according to claim 1 of the present invention is configured to convert a RAW image into four plane images of a luminance component and first to third color difference components, and for the four plane images. A plurality of sub-bands each of which is hierarchically decomposed into a plurality of sub-bands, and a second generation unit that compresses and encodes the plurality of sub-bands to generate a plurality of compressed sub-bands; Arranging means for collectively arranging the compressed subbands for each group based on the number of layers in the frequency decomposition, a compressed RAW area storing the arranged compressed subbands, and the plurality of compressed subbands. And a third generation unit that generates a recording file including an information management area in which information is stored.

  According to the present invention, it is possible to generate a compressed RAW image in an image recording format capable of displaying a reduced image and streaming reproduction at high speed.

3 is a block diagram showing a hardware configuration of an image encoding device according to Embodiment 1. FIG. FIG. 3 is a diagram showing a schematic configuration of RAW image encoding processing according to the first embodiment. FIG. 3 is a diagram showing subbands generated when the number of wavelets by the wavelet transform process in FIG. 2 is 2. It is a figure which shows the example of calculation of the quantization step size used by the quantization process in FIG. FIG. 6 is a diagram showing hierarchical display of a RAW image according to the first embodiment. FIG. 6 is a diagram for explaining a method of generating a reduced image by multiplexing plain images according to the first embodiment. FIG. 3 is a diagram showing an image recording format according to the first embodiment. 7B is a table showing parameters stored in the resolution header in FIG. 7A and used when hierarchically displaying a RAW image. It is a figure for explaining the number of times of reading a compressed subband from a recording file required for displaying a reduced image for every image recording format. FIG. 3 is a diagram showing an example of separating a RAW image into a plain image by the color separation processing in FIG. 2. 6 is a flowchart illustrating a procedure of image encoding processing according to the first embodiment. FIG. 8 is a diagram showing a hierarchical display of a RAW image according to the second embodiment. FIG. 9 is a diagram illustrating a method of expanding the resolution of color difference components according to the second embodiment. FIG. 9 is a diagram showing a modification of the color difference component resolution expanding method according to the second embodiment. FIG. 8 is a diagram showing an image recording format according to a second embodiment. FIG. 13B is a table showing parameters used when hierarchically displaying a RAW image, which is stored in the resolution header in FIG. 13A. FIG.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a block diagram of the hardware configuration of the image coding apparatus according to the first embodiment.

  In FIG. 1, the image encoding device 100 includes a CPU 101, an optical unit 102, an image capturing unit 103, a color separation unit 104, a plane conversion unit 105, a wavelet conversion unit 106, a quantization unit 107, and an entropy encoding unit 108. Further, the image coding apparatus 100 includes a memory I / F 109, a memory 110, a recording processing unit 111, and a recording medium 112.

  The CPU 101 has a memory that stores a control program executed by the CPU 101, and controls the entire processing of the image encoding device 100. The CPU 101 also has a program for accessing the memory I / F 109 and rearranging the coded streams (compressed RAW data) stored in the memory 110.

  The optical unit 102 is composed of a lens optical system capable of optical zoom including an optical lens, diaphragm, focus control, and lens driving unit, and sends the optical information acquired through the lens optical system to the image capturing unit 103.

  The image pickup unit 103 includes an image pickup device that converts the optical information sent from the optical unit 102 into an electric signal, receives a shooting instruction from the CPU 101, converts the electric signal obtained by the image pickup device into a digital signal, and forms a RAW image. To generate. The imaging unit 103 stores the generated RAW image in the memory 110 via the memory I / F 109 and sends a read permission signal to the color separation unit 104. The color separation unit 104 starts the operation by using the read permission signal received from the image pickup unit 103 as a signal. The image pickup device of the image pickup unit 103 may be any device that converts optical information into an electric signal, and examples thereof include a CCD image sensor and a CMOS sensor.

  The color separation unit 104 receives a read permission signal from the image pickup unit 103, reads a RAW image from the memory 110, and performs color plane separation for generating a plane image of R, B, G1, and G2 components from the RAW image. The color separation unit 104 sends each generated plane image to the plane conversion unit 105.

  The plane conversion unit 105 performs plane conversion on each plane image sent from the color separation unit 104 to generate a plane image of a luminance (Y) component and a plane image of a color difference (C1, C2, C3) component. To do. The plane conversion unit 105 sends each generated plane image to the wavelet conversion unit 106. Note that the plane conversion unit 105 may be mounted with a plurality of calculation formulas and the calculation formula actually used by the CPU 101 may be selected.

  The wavelet transform unit 106 performs hierarchical frequency decomposition on each plane image sent from the plane transform unit 105 and performs wavelet transform to generate a plurality of subbands. The wavelet transform unit 106 sends the generated subbands to the wavelet transform unit 106. The number of wavelet transforms performed by the wavelet transform unit 106 (hereinafter referred to as “wavelet count”) is set by the CPU 101.

  The quantization unit 107 quantizes each subband sent from the wavelet transform unit 106 based on a predetermined quantization step size. The quantizing unit 107 sends the quantized subbands to the entropy coding unit 108. Here, the quantization step size used in the quantization unit 107 may be set while weighting differently for each subband from the one quantization step size given by the CPU 101. Further, the quantization step size may be set individually for each subband by the CPU 101.

  The entropy coding unit 108 performs entropy coding using a Huffman code, arithmetic code, or the like on each quantized subband sent from the quantization unit 107 to generate a coded stream. Further, the entropy coding unit 108 stores the coded stream in the memory 110 via the memory I / F 109.

  The memory I / F 109 arbitrates a memory access request from each processing unit of the image encoding device 100 such as the CPU 101, the image capturing unit 103, the color separation unit 104, the entropy encoding unit 108, and the recording processing unit 111, and the memory 110. Read / write control for.

  The memory 110 is a storage area configured by, for example, a volatile memory for storing various data output from each processing unit of the image encoding device 100 such as the CPU 101, the image capturing unit 103, and the entropy encoding unit 108.

  The recording processing unit 111 records various data such as an encoded stream stored in the memory 110 on the recording medium 112 according to a recording command from the CPU 101. Further, according to the read instruction from the CPU 101, the various data are read.

  The recording medium 112 is a recording medium including a non-volatile memory, for example.

  Next, an encoding process 200 for encoding a RAW image into the image recording format according to the first embodiment to generate a compressed RAW image will be described with reference to FIG.

  As shown in FIG. 2, the encoding process 200 includes a color separation process 201, a plane conversion process 202, a wavelet conversion process 203, a quantization process 204, and an entropy encoding process 205.

  The color separation process 201 is a process executed by the color separation unit 104, and as shown in FIG. 9, generates four plane images of R, B, G1, and G2 components from the input RAW image.

  The plane conversion process 202 is a process executed by the plane conversion unit 105. Specifically, plane conversion is performed on each plane image generated by the color separation processing 201 to obtain a plane image of a luminance (Y) component and four plane images of color difference (C1, C2, C3) components. To generate. Here, the plane conversion is generated based on the following mathematical expression (1), for example. In addition, if the calculation formula uses one of the color difference components as the difference component of the G1 and G2 components like the C3 component of Formula 1, a calculation formula using a matrix according to the following Formula (2) is used. You may prepare more than one.

  The wavelet transform process 203 is a process executed by the wavelet transform unit 106. The wavelet transform process is performed on each plane image of the luminance and color difference components generated by the plane transform process 202 to generate subbands as shown in FIG. To do.

  FIG. 3 is a diagram showing subbands generated when the number of wavelets by the wavelet transform process 203 is two.

  When the number of wavelets for each plane image of the luminance and color difference components generated by the plane conversion processing 202 is one, four subbands 1LL, 1HL, 1LH, and 1HH are generated from each plane image. Here, L represents a low range and H represents a high range. The numbers before L and H represent the decomposition level. For example, 1HL represents a subband image having a decomposition level (the number of wavelets) = 1 in which the horizontal direction is a high frequency component and the vertical direction is a low frequency component. There is no limit to the number of wavelets. The second and subsequent wavelet transforms are recursively performed on the subband LL obtained by the immediately preceding wavelet transform. Therefore, there is only one sub-band LL when the wavelet transform is performed a predetermined number of times. Since FIG. 3 shows the case where the number of wavelets is 2, seven subbands 1HL, 1LH, 1HH, 2HL, 2LH, 2HH, and 2LL are generated. Returning to FIG. 2, the quantization process 204 is a process executed by the quantization unit 107, and quantizes each subband generated by the wavelet transform process 203 based on a predetermined quantization step size. Here, the quantization step size used in the quantization processing 204 is calculated from a preset base quantization step size and a weighting coefficient for each subband, as shown in FIG. Hereinafter, with reference to FIG. 4, a subband (hereinafter, referred to as “Y component”) generated from a Y plane image will be described as an example in which the number of wavelets is 1 and the base quantization step size is set to 4. Of sub-bands.)).

  In the quantization processing 204, the weighting factors 1, 1.5 and 1.5.2 set in the four sub-bands 1LL, 1HL, 1LH and 1HH of the Y component are respectively multiplied by 4 which is the base quantization step size. To be done. As a result, the quantization step sizes of the four subbands 1LL, 1HL, 1LH, and 1HH of the Y component are calculated as 4, 6, 6, and 8. In the present embodiment, the quantization step size used in the quantization processing 204 is set to the same value of 4 in each subband, but the value is set individually in each subband. Good.

  Returning to FIG. 2, the entropy coding process 205 is a process executed by the entropy coding unit 108, and uses entropy using Huffman code, arithmetic code, or the like for each subband quantized by the quantization process 204. Encode. As a result, each subband is compression-coded to generate a plurality of compressed subbands.

  Here, it is assumed that the compressed RAW image described above is composed of a plurality of compressed subbands generated by the entropy coding processing 205.

  Next, a method of performing reduced display of a RAW image using a plurality of compressed subbands generated by the encoding process 200 of FIG. 2 will be described with reference to FIG.

  FIG. 5 is a diagram showing an image display of a RAW image having a size of 8192 pixels in the horizontal direction and 4320 lines in the vertical direction in a hierarchical size (hereinafter, referred to as “hierarchical display”). In the example of FIG. 5, the number of wavelets is one.

  FIG. 5 shows images 500a to 500c having display sizes of three layers (layers 1 to 3) and the subbands of the luminance and color difference components used to generate the images.

  The image 500a in the layer 3 is an image having a display size of 8192 pixels in the horizontal direction and 4320 lines in the vertical direction, which is the size of the RAW image as it is. To the right of FIG. 5, the luminance and color difference components used to generate the image 500a are displayed. Indicates a subband.

  The image 500b in the layer 2 is an image having a display size of horizontal 4096 pixels and vertical 2160 lines, which is reduced by half in the horizontal direction and by half in the vertical direction with respect to the RAW image. To the right of FIG. 5, the sub-bands of the luma color difference components used to generate the image 500b are shown.

  The image 500c in the layer 1 is an image having a display size of horizontal 2048 pixels and vertical 1080 lines, which is reduced to a quarter in the horizontal direction and a quarter in the vertical direction with respect to the RAW image. To the right of FIG. 5, the sub-bands of the luma color difference components used to generate the image 500c are shown.

  An image having the same display size as the RAW image of horizontal 8192 pixels and vertical 4320 lines, such as the image 500a, is generated by decoding the compressed RAW image and developing the decoded RAW image. Here, the development processing applies debayer processing (demosaic processing) to the RAW image, converts it into a signal composed of luminance and color difference, removes noise included in each signal, corrects optical distortion, and corrects the image. Refers to processing such as conversion. It should be noted that in order to generate such an image 500a having the display size as it is, it is necessary to decode all the compressed subbands generated from the RAW image.

  A reduced image having a display size of 4096 horizontal pixels and 2160 vertical lines, such as the image 500b, is not subjected to RAW image development processing as described later, and the Y component of the RAW image and the plain image of two color difference components are multiplexed. Generate and generate. As a result, the image 500b is generated as an image having a three-component luminance color difference component for each pixel. Here, in the present embodiment, the CPU 101 selects the C1 and C2 components as the color difference components (display color difference components) for displaying the two reduced images among the three color difference components C1 to C3. (Selection means). It should be noted that generation of such an image 500b having a display size of 4096 horizontal pixels and 2160 vertical lines requires decoding of all compressed subbands of Y, C1, and C2 components generated from the RAW image. The display color difference component selected by the CPU 101 is not limited to this as long as it is two of the three color difference components C1 to C3.

  A reduced image having a display size of 2048 horizontal pixels and 1080 vertical lines, such as the image 500c, is not subjected to the RAW image development process as in the image 500b, and is a sub-component of Y, C1, C2 components 1LL generated from the RAW image. Generate by multiplexing the bands. As a result, the image 500c is generated as an image having a size of 1LL having three components of Y, C1, and C2 for each pixel. In order to generate the image 500c having the display size of horizontal 2048 pixels and vertical 1080 lines, it is necessary to decode the 1LL compressed subband of the Y, C1, and C2 components generated from the RAW image.

  Next, a method of generating a reduced image by multiplexing plane images of Y, C1, and C2 components will be described with reference to FIG. That is, FIG. 6 exemplifies a case where the generated reduced image is an image 500b having a display size of 4096 horizontal pixels and 2160 vertical lines. In the description here, the plane images of the C1 and C2 components are plane-converted based on the above formula (1) or formula (2). That is, the C1 component is a component corresponding to the Cb component that represents the hue and saturation of blue colors, and the C2 component is the component that corresponds to the Cr component that represents the hue and saturation of red colors.

  As shown in the above equations (1) and (2), the calculation formula used in the plane conversion in this embodiment is the average of the G1 and G2 components, the R component, and the B component of the color components forming the RAW image. It is used to generate a luminance (Y) component and two color difference (C1, C2) components. Further, the difference between the G1 and G2 components is generated as the third color difference (C3) component.

  Further, since the number of pixels of each plane image of the luminance component and the color difference component is the same, it can be considered that the luminance component and the color difference component of each plane image are arranged at the same pixel position in the size of the plane image.

  Therefore, the color difference components (C1 and C2 components) other than the luminance component at the same pixel position of the plain image and the C3 component which is the difference between the G1 and G2 components are used. As a result, the three RGB components can be aligned with each pixel position of the image 500b, and can be used for display as they are.

  In the example of FIG. 6, each of the Y, C1, C2, and C3 plane images is generated by performing color separation and plane conversion on the RAW image. Further, although not shown in FIG. 6, when the wavelet transform is performed once, sub-bands 1LL, 1HL, 1LH, and 1HH are respectively generated from the Y, C1, C2, and C3 plane images.

  When generating a reduced image, first, the generated subbands are combined and returned to a plane image of Y, C1, and C2 components. After that, the plane images of the Y, C1, and C2 components are multiplexed, and an image having the same resolution as the plane image having three components of the Y component, C1 (≈Cb) component, and C2 (≈Cr) component in one pixel To generate. Then, by converting the three components of Y, C1, and C2 of the generated image into the three components of R, G, and B, an image 500b having an R component, a G component, and a B component in one pixel is generated. .. By displaying this image 500b, it is possible to reduce the display size of the RAW image in the display size of Layer 2.

  In this embodiment, the reduced image having three components of the Y component and the two color difference components C1 and C2 is generated, but the present invention is not limited to this. For example, a reduced image having all the components, that is, the Y component and the three color difference components C1 to C3 may be generated. For example, a reduced image may be generated by generating a reduced size RAW image from 1LL subbands of Y, C1, C2, and C3 components and developing the reduced size RAW image. Further, as described above, when the image 500a is generated, the process of decoding the compressed RAW image of the Y, C1, C2, and C3 components and returning it to the RAW image is performed. Similar processing may be performed on the compressed RAW image to generate a reduced image.

  Next, the image recording format according to the first embodiment will be described with reference to FIGS. 7A and 7B.

  The image recording format 700 shown in FIG. 7A includes an information management unit 701 and a compression RAW unit 702.

  The information management unit 701 includes a main header 703, a resolution header 704, and a subband header 705.

  The compression RAW unit 702 compresses the compressed data of each sub-band generated by compressing the RAW image, that is, the compressed sub-band, for each group of layers based on the hierarchical display of the RAW image similarly to the sub-band header 705 described later. Store all at once.

  The main header 703 stores information such as the image size of the captured RAW image, the number of pixel bits, the number of wavelets, and a mathematical formula used for plane conversion.

  The resolution header 704 stores parameters used when hierarchically displaying a RAW image, that is, decoding information for decoding a compressed RAW image. Specifically, the display size of the decoded image (reduced image), the number of wavelets, the number of subbands used to generate the reduced image, the read data size, and a flag indicating whether the decoded image is a RAW image (FIG. 7B). "Development" in the table corresponds to) is stored. Here, the read data size refers to the data size of the compressed subband required when generating the reduced image. The resolution header 704 has a flag indicating whether or not it is a RAW image, in order to determine whether or not to perform development processing after decoding by referring to this flag when displaying a reduced image. That is, the resolution header 704 stores information regarding each display layer shown in FIG. 5 as a parameter.

  Here, as an example, of the parameters stored in the resolution header 704, the information of the layer 2 used for displaying the image 500b will be described with reference to the table of FIG. 7B.

  In the information of the layer 2, the display size is 4096 pixels × 2160 lines, the number of wavelets is 1, and the number of used subbands is 12. Further, the read data size is the total size of the compressed sub-bands grouped in the groups of layers 1 and 2 of the compression RAW unit 702, and the development process is not necessary at the time of display. Note that the image 500b of the layer 2 is generated using four subbands of 1LL, 1HL, 1LH, and 1HH of the Y, C1, and C2 components, respectively, so the number of used subbands is 12. Further, among the 12 subbands, the 1LL subbands of the Y, C1, and C2 components are combined by the compression RAW unit 702 as compressed subbands of the layer 1 group. On the other hand, the remaining subbands are combined by the compression RAW unit 702 as compressed subbands of the layer 2 group. Therefore, the read data size is the sum of the data sizes of the compressed subbands that are grouped in the layers 1 and 2 by the compression RAW unit 702.

  That is, when the number of wavelets is N, the read data size included in the information of the layer n (1 ≦ n ≦ N + 1) is the sum of the data sizes of the compressed subbands grouped in the layers 1 to n.

  Returning to FIG. 7A, the subband header 705 stores parameters (compressed subband information) for each compressed subband held in the compression RAW unit 702. This parameter is the data size of each compressed subband, the quantization parameter, which subband in the wavelet decomposition (for example, the 2HL subband shown in FIG. 3), and the Y, C1, C2, and C components. Including information of which of

  The arrangement of the compression subbands in the compression RAW unit 702 is as shown in FIG. 7A when the number of wavelets is N. That is, the NLL compression subbands are arranged in the order of Y, C1, and C2 components in the layer 1 group, and the NHL, NLH, and NHH compression subbands are arranged in the order of Y, C1, and C2 components in the layer 2 group. Bands are arranged. In the layer 3 group, compressed subband groups of (N-1) HL, (N-1) LH, and (N-1) HH are arranged in the order of Y, C1, and C2 components. In the layer 4 group, the compressed subband groups of (N-2) HL, (N-2) LH, and (N-2) HH are arranged in the order of Y, C1, and C2 components. In the same way, a plurality of compressed subbands are arranged in the groups of layers 5 to N-1. In the group of hierarchy N, 2HL, 2LH, and 2HH compressed subband groups are arranged in the order of Y, C1, and C2 components. Further, the compressed subband groups of 1HL, 1LH, and 1HH are arranged in the order of Y, C1, and C2 components in the group of hierarchical layer N + 1, and all the compressed subbands of C3 that are not used for reduced display in the final group of hierarchical layer N + 2. Are lined up. The parameters relating to each compressed subband held in the subband header 705 are also grouped into hierarchical groups in the same manner as the arrangement of the compressed subbands in the compression RAW unit 702.

  As described above, in the compression RAW unit 702, first, the groups of compressed subbands of the components used in the reduced display, that is, the Y component and the display color difference components (C1 and C2 components) are arranged in the order of the reduced display layers 1 to N + 1. It is arranged together for each group of layers. That is, the groups of compressed subbands of the Y, C1, and C2 components are arranged in order from the group of the lower hierarchy to the group of the higher hierarchy. At this time, the compressed subband of the Y component is arranged first in each group. After that, the compressed subbands of the color difference component (C3 component) that are not used in the reduced display are collectively arranged in a group (RAW reconstruction data group) different from the group of the reduced display hierarchy.

  Taking the reduced display of FIG. 5 as an example, in the compression RAW unit 702, 1LL compressed subbands are arranged in the order of Y, C1, and C2 components in the layer 1 group, and Y, C1, and C2 components are arranged in the layer 2 group. Compressed subband groups of 1HL, 1LH, and 1HH are arranged in order. Further, all subbands of C3 are arranged in the last layer 3 group.

  The information stored as the read data size in the resolution header 704 does not have to be the sum of the data sizes of the compressed subbands read from the compression RAW unit 702 when the reduced image is generated. For example, it may be the sum of the data sizes of the compressed subbands that are grouped into the layers of the compression RAW unit 702 corresponding to the display layer of the reduced image. Also, in this case, this information may be used when reading out the compressed subband necessary for display.

  If the resolution header 704 shown in FIG. 7A is taken as an example, the read data size of layer 1 is the NLL compressed subbands of the Y, C1, and C2 components that are grouped into the group of layer 1 of the compression RAW unit 702. It is the sum of the data sizes. The read data size of the layer 2 is the sum of the data sizes of the NHL / NLH / NHH compressed subband groups of Y, C1, and C2 components that are grouped into the layer 2 group of the compression RAW unit 702. The read data size of the layer 3 is (N-1) HL / (N-1) LH / (N-1) HH of each of the Y, C1, and C2 components that are grouped into the layer 3 group of the compression RAW unit 702. It is the sum of the data sizes of the compressed subband group. The read data size of the layer 4 is (N−2) HL / (N−2) LH / (N−2) HH for each of the Y, C1, and C2 components that are grouped into the layer 4 group of the compression RAW unit 702. It is the sum of the data sizes of the compressed subband group. In the same manner, the read data size of layers 5 to N-1 also becomes the sum of the data sizes of the compressed subband groups of the groups of layers 5 to N-1. The read data size of the layer N is the sum of the data sizes of the 2HL / 2LH / 2HH compressed subband groups of the Y, C1, and C2 components that are grouped into the layer N group of the compression RAW unit 702. Similarly, the read data size of the layer N + 1 is the sum of the data sizes of the 1HL / 1LH / 1HH compressed subband groups of the Y, C1, and C2 components that are combined into the layer N + 1 group of the compression RAW unit 702.

  As described above, according to the first embodiment, the RAW image is encoded into the compressed RAW image according to the image recording format 700 shown in FIG. 7A to generate the recording file, and the generated recording file is stored in the recording medium 112. As a result, it is possible to efficiently extract the compressed sub-band required when the recording file is read from the recording medium 112 and the reduced image is reproduced by the image encoding device 100. Further, in the case of transferring to an external display device for streaming reproduction, it is possible to efficiently take out a necessary compressed subband and transfer it to a display device such as a smartphone. Here, the recording file refers to data that stores a plurality of compressed subbands forming a compressed RAW image, decoding information for decoding the compressed RAW image, and the like according to the image recording format 700.

  With reference to FIG. 8, the number of times of reading the compressed subband from the recording file necessary for displaying the reduced image for each of the image recording format 700 of this embodiment and two known image recording formats will be described.

  FIG. 8 shows how to arrange compressed subbands in a recording file for each of the image recording format of this embodiment and two known image recording formats.

  The recording file 800a has compression subbands arranged in an image recording format having an inclusion relation of resolution⊃plane⊃subband. The inclusion relation of resolution⊃plane⊃subband is defined by a known image recording format such as JPEG2000.

  The recording file 800b has compressed subbands arranged in an image recording format that has an inclusion relation of plane⊃resolution⊃subband. The inclusion relation of plane⊃resolution⊃subband is defined by a known image recording format such as JPEG2000 as in the recording file 800a.

  The recording file 800c has compressed subbands arranged in the image recording format 700 of FIG. 7A.

  Taking the case of displaying the image 500b having the display size of horizontal 4096 pixels and vertical 2160 lines shown in FIG. 5 as an example, it is necessary to read the compressed subbands in the portion shown by the dotted line in FIG. 8 from the recording files 800a to 800c.

  Specifically, in the recording file 800a, it is necessary to read twice for the 1LL subband of the Y, C1, and C2 components and the 1HL, 1LH, and 1HH subband groups of the Y, C1, and C2 components. Further, in the recording file 800b, the Y-component 1LL, 1HL, 1LH, and 1HH subband groups, the C1 component 1LL, 1HL, 1LH, and 1HH subband groups, and the C2 component 1LL, 1HL, 1LH, and 1HH subbands. The group requires three reads. As described above, in the recording files 800a and 800b, it is necessary to read the compressed subband a plurality of times. On the other hand, in the recording file 800c, the 1LL, 1HL, 1LH, and 1HH subband groups of the Y, C1, and C2 components are grouped together. Therefore, when the CPU 101 (reading unit) reads the compressed sub-band necessary for displaying the image 500b from the recording file 800c recorded on the recording medium 112, the reading process is performed only once.

  Furthermore, when performing stream reproduction in which an external display device reproduces a compressed RAW image as a moving image, the above-described read processing is performed for the number of compressed RAW images included in the moving image.

  That is, as the number of times of reading processing from one recording file increases, the load on the hardware of the image encoding device 100, particularly the CPU 101, stresses the processing for image display, and displays reduced images and stream reproduction. It will cause a delay in the case. For this reason, it is preferable that the number of reading processes from one recording file is small.

  Therefore, in the present embodiment, the compressed subbands are arranged according to the arrangement of the compression RAW unit 702 of the image recording format 700 like the recording file 800c. As a result, when displaying the reduced image, the number of times of read processing for reading the compressed subband necessary for display from the recording file stored in the recording medium 112 can be reduced. In addition, when the compressed subband required for the streaming reproduction is sent to the external display device, the required compressed subband can be efficiently read from the recording file stored in the recording medium 112.

  As described above, by applying the image recording format 700 according to the first embodiment to the recording file, it is possible to reduce the display delay caused by the reading process from the recording file.

  Next, the image coding processing according to the first embodiment will be described with reference to the flowchart in FIG.

  First, the CPU 101 gives an image capturing start instruction to the image capturing unit 103 (step S1001).

  After that, the imaging unit 103 acquires the RAW image and temporarily stores it in the memory 110 (step S1002).

  Subsequently, the color separation unit 104 (conversion means) reads the RAW image stored in the memory 110, and color plane separation for generating a plane image of four components of R, B, G1, and G2 from the read RAW image. Is performed (step S1003).

  After that, the plane conversion unit 105 (conversion means) performs plane conversion on the plane images of R, B, G1, and G2 components to generate plane images of four components of Y, C1, C2, and C3 ( Step S1004).

  Subsequently, the wavelet transform unit 106 (first generation unit) performs wavelet transform for generating a plurality of subbands from the Y, C1, C2, and C3 component plane images (step S1005). At this time, the memory 110 temporarily stores which subband in the wavelet decomposition the generated subbands are, and which of Y, C1, C2, and C components.

  Then, the quantization unit 107 quantizes each subband (step S1006). At this time, the quantization parameter of each subband is temporarily stored in the memory 110.

  Subsequently, the entropy coding unit 108 (second generation unit) performs entropy coding in which each quantized subband is compressed to generate a coded stream, and then the coded stream is temporarily stored in the memory 110. It is stored (step S1007). At this time, the data size of each compressed subband is also temporarily stored in the memory 110.

  After that, the CPU 101 (sorting means) sorts the encoded data (compressed subbands) of each subband included in the encoded stream temporarily stored in the memory 110 in the order shown in FIG. 7A (step S1008).

  Subsequently, the CPU 101 generates header information according to the information management unit 701 of the image recording format 700 of FIG. 7A (step S1009). Specifically, the CPU 101 (rearranging means) rearranges the information regarding each compressed subband temporarily stored in the memory 110 in the order shown in FIG. 7A, and generates header information.

  Finally, the CPU 101 (fourth generation unit) generates a recording file and records it in the recording medium 112 (step S1010). That is, the encoded data rearranged in step S1008 is written on the recording medium 112 in the compressed RAW area of the recording file corresponding to the compressed RAW unit 702. Further, the information of the header generated in step S1009 is written in the information management area of the recording file corresponding to the information management unit 701. Then, this process is completed.

  As described above, by applying the image recording format 700 according to the first embodiment to the recording file, when the reduced image is displayed, the compressed subband required for the display is read once from the recording file stored in the recording medium 112. Can be read with. Further, by applying the image recording format 700 according to the first embodiment to a recording file, it is possible to send a required compressed subband to the display device at the minimum required number of transfers during streaming reproduction.

  As described above, by applying the image recording format 700 according to the first embodiment to a recording file, it is possible to reduce the load due to the reading process from the recording file and the transfer process by streaming, and thus the reduced display and the streaming reproduction can be performed at high speed. You can

  Next, the image recording format according to the second embodiment will be described.

  The image recording format according to the second embodiment enables the resolution of the color difference component to be equal to or lower than the resolution of the luminance component when the reduced image is generated. As a result, it can be expected that the read amount of the compressed subband will be further reduced as compared with the case of using the image recording format 700 according to the first embodiment.

  The image recording format 700 according to the first embodiment multiplexes the three-component plane image of Y, C1 (≈Cb), and C2 (≈Cr) when the reduced image (image 500b) of the layer 2 is generated. After that, an image in the format of YCbCr444 (YUV444) having three components of Y component, C1 component, and C2 component in one pixel is generated.

  On the other hand, in the image recording format according to the second embodiment, the resolution of the C1 and C2 component plane images used to generate the layer 2 reduced image is set in the horizontal direction or the horizontal / vertical direction with respect to the Y plane image resolution. Thinning is performed so as to be half (hereinafter referred to as "color difference thinning"). This enables image generation in the YUV422 and YUV420 formats. In the present embodiment, the color difference thinning is performed so that the resolution of the C1 and C2 component plane images is half the resolution of the Y plane image in the horizontal direction. However, color difference thinning in either the horizontal or vertical direction is performed. Should be done.

  A case where the resolutions of the C1 and C2 plane images are reduced with respect to the Y plane image when generating the reduced image of the layer 2 will be described with reference to FIG.

  FIG. 11 is a diagram showing hierarchical display of a RAW image having a size of 8192 pixels in the horizontal direction and 4320 lines in the vertical direction. In the example of FIG. 11, the number of wavelets is one.

  FIG. 11 shows images 1100a to 1100c having display sizes of three layers (layers 1 to 3) and subbands of the luminance and color difference components used to generate the images. It should be noted that, as shown in FIG. 11, the sub-band of the luminance / chrominance component used to generate the image 1100b of the layer 2 defines the number of compression sub-bands of the color-difference component used when the image 1100b is generated. ~ 3.

  The image 1100a in the layer 3 is an image having a display size of 8192 pixels in the horizontal direction and 4320 lines in the vertical direction, which is the size of the RAW image as it is. To the right of FIG. 11, the luminance and color difference components used for generating the image 1100a are displayed. Indicates a subband.

  The image 1100b in the layer 2 is an image having a display size of horizontal 4096 pixels and vertical 2160 lines, which is reduced by half in the horizontal direction and by half in the vertical direction with respect to the RAW image. In FIG. 11, on the right side, the subbands of the luminance and color difference components used to generate the image 1100b are shown. Specifically, the sub-band group 1101 shows three component sub-bands corresponding to the sub-layer 1, the sub-band group 1102 shows three component sub-bands corresponding to the sub-layer 2, and the sub-band group Reference numeral 1103 indicates sub-bands of three components corresponding to sub-layer 3.

  The image 1100c in the layer 1 is an image having a display size of horizontal 2048 pixels and vertical 1080 lines, which is a quarter of the RAW image in the horizontal direction and a quarter in the vertical direction. In FIG. 11, on the right side thereof, the sub-bands of the luminance and color difference components used for generating the image 1100c are shown.

  Generation of an image having a display size of horizontal 8192 pixels and vertical 4320 lines, such as the image 1100a, is performed by the same method as in the first embodiment.

  A reduced image having a display size of 4096 horizontal pixels and 2160 vertical lines, such as the image 1100b, is generated by multiplexing the plain images of the Y, C1, and C2 components without developing the RAW image. As a result, the image 1100b is generated as an image having three color components of Y, C1, and C2 for each pixel. Here, in this embodiment, the image 1100b is generated by using the different subband groups 1101 to 1103 in the sublayers 1 to 3.

  Specifically, for sub-tier 1, as shown in sub-band group 1101, using YLL 1LL, 1HL, 1LH, 1HH subbands, and C1, C2 component 1LL subbands, YUV420 format The reduced image of 1 is generated as the image 1100b.

  Also, for sub-layer 2, as shown in sub-band group 1102, using YLL 1LL, 1HL, 1LH, 1HH subbands, and C1, C2 component 1LL, 1LH subbands, the YUV422 format is used. A reduced image is generated as the image 1100b.

  For sub-layer 3, as shown in sub-band group 1103, a YUV444 format reduced image is generated as image 1100b using the sub-bands of 1LL, 1HL, 1LH, and 1HH of Y, C1, and C2 components.

  Display of an image having a display size of 2048 horizontal pixels and 1080 vertical lines, such as the image 1100c, is performed by the same method as in the first embodiment.

  Here, generation of reduced images in each of the sub-layers 1 to 3 of the layer 2 shown in FIG. 11 will be described with reference to FIGS. 12A and 12B.

  In the subband group 1101 of the sublayer 1 of the layer 2 shown in FIG. 11, the number of pixels of the C1 and C2 component plane images is halved in the horizontal / vertical direction with respect to the number of pixels of the Y component plane image. Thinned out. Therefore, as shown in FIG. 12A (a), when the three RGB components of the reduced image in the YUV420 format are generated from the subband group 1101 of the sublayer 1, the plane images of the C1 and C2 components are horizontally and vertically aligned. It is necessary to perform double upsampling (resolution expansion).

  On the other hand, in the subband group 1102 of the sublayer 2 of the layer 2 shown in FIG. 11, the number of pixels of the C1 and C2 component plane images is halved in the horizontal direction with respect to the number of pixels of the Y component plane image. Thinned out. Therefore, as shown in FIG. 12A (b), when the three RGB components in the YUV422 format reduced image are generated from the subband group 1102 of the sublayer 2, the plane images of the C1 and C2 components are doubled in the horizontal direction. Need to up-sample.

  The subband group 1102 may be thinned out in the vertical direction instead of the horizontal direction. In that case, however, upsampling needs to be performed in the vertical direction on the subbands of the C1 and C2 components.

  Next, FIGS. 12A and 12B show an example in which adjacent pixels are copied as they are at the time of upsampling. In addition, in FIGS. 12A (a) and 12 (b), each pixel of the Y component is described as (horizontal coordinate, vertical coordinate), and each pixel of the C1 and C2 component is described as [horizontal coordinate, vertical coordinate]. ..

  In FIG. 12A (a), the pixel [1,0], for example, is copied in the right, lower, and lower right directions by upsampling. On the other hand, in FIG. 12A (b), for example, the pixel of [1, 0] is copied to the right by upsampling.

  The upsampling method is not limited to the methods shown in FIGS. 12A (a) and 12 (b) as long as the YUV422 format reduced image and the YUV420 format reduced image can be generated using the subband groups 1101 and 1102 of FIG. For example, as shown in FIGS. 12B (a) and 12B (b), the resolution of the plane image of C1 and C2 components may be upsampled at the stage of wavelet synthesis.

  That is, for the subband group 1101, as shown in FIG. 12B (a), the coefficients of the subbands that are not read, specifically, the 1HL, 1LH, and 1HH subbands of the C1 and C2 component plane images, respectively. It is assumed that the coefficients are all zero. Wavelet synthesis is performed in this state.

  Similarly, for the subband group 1102, as shown in FIG. 12B (b), all the coefficients of the subbands that are not read, specifically, the coefficients of the 1HL and 1HH subbands of the C1 and C2 component plane images are all Considered to be composed of zero. Wavelet synthesis is performed in this state.

  Here, as an example of the wavelet transform using a 5 / 3-tap filter, the calculation shown in FIGS. 12B (a) and 12B assuming that the coefficients of the sub-bands that are not read are all configured with 0s. explain.

  The wavelet transform for the horizontal direction or the vertical direction is the following filter calculation.

ｓはウェーブレット変換の対象となる信号、ここではプレーン画像の画素値を表す。

s represents a signal to be subjected to the wavelet transform, here, a pixel value of a plane image.

  The wavelet synthesis in the horizontal direction or the vertical direction is the following calculation.

となる。 Here, it is assumed that the wavelet transform coefficient values of the 1HL and 1HH subbands of C1 / C2 are all 0, that is, when H in the calculation of wavelet synthesis in the horizontal direction is 0,

Becomes

  From the calculation formula, in the horizontal direction of the sub-bands of the C1 and C2 component plane images, the pixel value at the odd position is the average of the pixel values at the even positions on both sides. That is, it is assumed that all the sub-band coefficients that are not read are composed of 0s, and the wavelet synthesis is performed, so that the upsampling process is performed to make the resolution of the C1 and C2 component plane images correspond to the Y component plane image. Processing equivalent to that of is possible.

  Next, the image recording format 1300 according to the second embodiment will be described with reference to FIGS. 13A and 13B. In addition, in the configuration of the image recording format according to the second embodiment, the same components as those of the image recording format according to the first embodiment are denoted by the same reference numerals and description thereof will be omitted.

  The image recording format 1300 shown in FIG. 13A includes an information management unit 1301 and a compression RAW unit 1302.

  The information management unit 1301 includes a main header 1303, a resolution header 1304, and a subband header 1305.

  The compression RAW unit 1302 collects the compressed data of each subband generated by compressing the RAW image, that is, the compressed subband for each group of layers based on the hierarchical display of the RAW image similarly to the subband header 1305 described later. To store.

  Like the main header 703 of the image recording format 700 of the first embodiment, the main header 1303 stores information such as the image size of the captured RAW image, the number of pixel bits, the number of wavelets, and a mathematical formula used for plane conversion. ..

  The resolution header 1304 stores parameters used when hierarchically displaying a RAW image. Specifically, the display size of the decoded image, a flag indicating whether or not color differences are thinned out with respect to luminance (corresponding to “color difference thinning out” in the table of FIG. 13B), the number of wavelets, the number of subbands used, and reading The data size and a flag indicating whether or not the decoded image is a RAW image are stored.

  If the flag indicating whether the color difference is thinned out with respect to the luminance is 1 as in the layer 2, that is, the pixels with the color difference are thinned out with respect to the luminance, the number of subbands to be used and the read data size are used. The flag indicating whether the decoded image is a RAW image is invalid. In this case, the sub resolution header 1306 described later will be referred to.

  When the flag indicating whether the resolutions between luminance / color difference are the same is 1, it is considered that the decoded image is not a RAW image, and the development process at the time of generating the display image is not performed, that is, invalidated.

  That is, as shown in FIG. 13B, the resolution header 1304 stores information regarding each display layer shown in FIG. 13A as a parameter, like the resolution header 704 of FIG. 7B.

  The sub-resolution header 1306 stores parameters used in sub-layers included in a layer in which pixels of color difference with respect to luminance are thinned out of each display layer. Specifically, for each sub-layer, the data size of the color difference components of C1 and C2, the presence / absence of wavelet synthesis in the horizontal direction, the presence / absence of wavelet synthesis in the vertical direction, the number of sub-bands used, and the read data size are included.

  Here, as an example, of the parameters stored in the resolution header 1304, information of layers 1 to 3 used for displaying the images 1100a to 1100c will be described with reference to the table of FIG. 13B.

  The information of layer 1 has a display size of 2048 pixels × 1080 lines, pixels of color difference with respect to luminance are not thinned, the number of wavelets is 0, and the number of used subbands is 3. Further, the read data size is the total size of the compressed subbands grouped in the layer 1 of the compression RAW unit 1302, and the development process is not necessary at the time of display. Since the image 1100c of the layer 1 is generated by using 1LL subbands of Y, C1, and C2 components, the number of used subbands is three. Further, since the above three subbands are all combined as a compressed subband of layer 1 by the compression RAW unit 1302, the read data size is the total size of layer 1.

  In the information of the layer 2, the display size is 4096 pixels × 2160 lines, pixels with color difference with respect to luminance are thinned, and the number of wavelets is one. As for the number of used subbands, the sum of the data sizes of the used subbands, etc., the information shown in the table of sub resolution headers subordinate to layer 2 as shown in FIG. 13B will be referred to.

  Of the sub-layers subordinate to the layer 2, the information of the sub-layer 2 is that horizontal wavelet is not present, vertical wavelet is present, the number of sub-bands used is 8, read data size is layer 1, sub-layer 1, sub-layer 2 It is the total size of the compressed subbands collected by each. The image 1100b generated as the reduced image of the sub-layer 2 is generated by using the sub-bands of 1LL, 1HL, 1LH, 1HH of Y component and the sub-bands of 1LL, 1LH of C1, C2 components. The number of used subbands is eight. Also, among the above eight subbands, 1LL subbands of Y, C1, and C2 components are combined as a compressed subband of the layer 1 group of the compression RAW unit 1302. In addition, the 1HL sub-bands of the Y component and the 1LH sub-bands of the C1 and C2 components are combined as compressed subbands of the sub-layers 1 and 2 of the layer 2 of the compression RAW unit 1302. Therefore. The read data size is the sum of the data sizes of the compressed subbands that are grouped by the compression RAW unit 702 into groups of layer 1, sublayer 1, and sublayer 2.

  That is, when the number of wavelets is N, the read data size included in the information of sub-layer 1 of layer M (2 ≦ M ≦ N + 1) is summarized in sub-layer 1 of layers 1 to M-1 and layer M. It is the sum of the data sizes of the compressed subbands. The read data size included in the information of the sub-layer 2 of the layer M is the read data size included in the information of the sub-layer 1 of the layer M and the data size of the compressed sub-bands combined in the sub-layer 2 of the layer M. It becomes the sum. Similarly, the read data size included in the information of the sub-layer 3 of the layer M is the read data size included in the information of the sub-layer 2 of the layer M and the data size of the compressed sub-bands combined in the sub-layer 3 of the layer M. Is the sum of Here, in other words, the tier M refers to the number of tiers among tiers 1 to N + 1 used when displaying the reduced image, excluding the tier 1, which is the minimum tier.

  Returning to FIG. 13A, the subband header 1305 stores parameters for each compressed subband held by the compression RAW unit 1302. This parameter is the data size of each compressed subband, the quantization parameter, which subband in the wavelet decomposition (for example, the 2HL subband shown in FIG. 3), and the Y, C1, C2, and C3 components. Including information of which of

  The arrangement of the compression subbands in the compression RAW unit 1302 is as shown in FIG. 13A when the number of wavelets is N. That is, NLL compressed subbands are arranged in the group of Layer 1 in the order of Y, C1, and C2 components. In the layer 2 group, the Y component NHL, NLH, and NHH compressed subbands are arranged in the sublayer 1 group, and the C1 and C2 component NLH compressed subbands are arranged in the sublayer 2 group. .. Further, the compressed subbands of NHL and NHH of C1 and C2 components are arranged in the group of the sub layer 3. In the layer 3 group, the (N-1) HL, (N-1) LH, and (N-1) HH compressed subbands of the Y component are included in the sublayer 2 in the sublayer 1 group. In the group, (N-1) LH compressed subbands of C1 and C2 components are arranged. In addition, the compressed subbands of (N-1) HL and (N-1) HH of C1 and C2 components are arranged in the group of the sub-layer 3. In the same way, a plurality of compressed subbands are arranged in the groups of layers 4 to N-1. In the group of the layer N, the compressed subbands of 2HL, 2LH, and 2HH of the Y component are arranged in the group of the sublayer 1, and the compressed subbands of 2LH of the C1 and C2 components are arranged in the sublayer 2. .. In addition, 2HL and 2HH compressed subbands of C1 and C2 components are arranged in the sub-layer 3 group. In the group of the layer N + 1, the compressed subbands of 1HL, 1LH, and 1HH of the Y component are arranged in the group of the sublayer 1, and the compressed subbands of 1LH of the C1 and C2 components are arranged in the group of the sublayer 2. .. In addition, 1HL compressed subbands of C1 and C2 components are arranged in the sub-layer 3 group. All subbands of C3 are arranged in the last group N + 2. The parameters relating to each of the compressed subbands held in the subband header 1305 are also grouped into layers and groups of sublayers in the same manner as the arrangement of the compressed subbands in the compression RAW unit 1302.

  As described above, in the compression RAW unit 1302, the compressed subbands of the components (Y, C1, and C2 components) used in the reduced display are collectively arranged for each group of layers in the order of the reduced display layers 1 to N + 1. Further, in consideration of the case where color difference thinning is performed at the time of generating a reduced image, the group of the hierarchy M (2 ≦ M ≦ N + 1) is composed of the groups of sub-tiers 1 to 3. Specifically, as a group of sub-layer 1 (first sub-layer) used for displaying a reduced image of YUV420 format that requires upsampling in the horizontal / vertical directions, MHL, MLH, and MHH of Y components. The compressed subbands of are grouped together. In addition, the compressed subbands of the MLH of the C1 and C2 components are grouped together as a group of sublayer 2 (first color difference layer) used for displaying a reduced image in the YUV422 format that requires horizontal upsampling. .. Furthermore, the compressed subbands of MHL and MHH are grouped together as a group of sublayer 3 (second color difference layer) used for displaying a reduced image in the YUV444 format that does not require upsampling. After that, all the compressed subbands of the component (C3 component) not used in the reduced display are collectively arranged. As described with reference to FIG. 12A (b), the subband group 1102 may be thinned out in the vertical direction instead of the horizontal direction. In that case, the sub-layer 2 group includes compressed sub-bands for vertical up-sampling.

  By using such an image recording format 1300, in the image recording format 700 according to the first embodiment, only the reduced image in the YUV444 format can be generated, while the reduced image in the YUV422 format or the YUV420 format can also be generated. It will be possible. That is, by allowing the color difference thinning-out, the data size of the compressed subband required to generate the reduced image can be further reduced as compared with the first embodiment.

  As described above, according to the second embodiment, the RAW image is encoded into the compressed RAW image of the image recording format 1300 shown in FIG. 13A. This reduces the read amount of compressed subbands related to color difference components, and enables the read processing of the required compressed subbands from the recording file and the transfer processing of the compressed subbands required for streaming playback on an external display device. The load can be further reduced as compared with the first embodiment. That is, the display delay due to the reading process from the recording file can be further reduced as compared with the first embodiment.

  It should be noted that in the present embodiment, it is possible to generate an image in either format of YUV422 or YUV420, but it is also possible to generate only one of the images. In this case, only the compressed sub-band of the sub-layer corresponding to the format that enables image generation and the information related thereto are stored in the image recording format 1300.

[Other Examples]
Needless to say, the object of the present invention can be achieved by supplying a storage medium storing a program code of software for realizing the functions of the above-described embodiments to the apparatus. At this time, the computer (or CPU or MPU) including the control unit of the supplied device reads and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the function of the above-described embodiment, and the program code itself and the storage medium storing the program code constitute the present invention.

  As the storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a non-volatile memory card, a ROM or the like can be used.

  An OS (basic system or operating system) running on the device performs some or all of the processing based on the instructions of the program code, and the processing realizes the functions of the above-described embodiments. It goes without saying that cases are included.

  Further, the program code read from the storage medium may be written in a memory provided in a function expansion board inserted in the device or a function expansion unit connected to the computer to realize the functions of the above-described embodiments. Needless to say, it is included. At this time, the CPU or the like included in the function expansion board or function expansion unit performs a part or all of the actual processing based on the instruction of the program code.

101 CPU
102 optical section 103 image pickup section 104 color separation section 105 plane conversion section 106 wavelet conversion section 107 quantization section 108 entropy encoding section 109 memory I / F
110 memory 111 recording processing unit 112 recording medium

Claims (10)

Conversion means for converting the RAW image into four plane images of a luminance component and first to third color difference components;
First generation means for performing hierarchical frequency decomposition on the four plane images to generate a plurality of subbands, respectively;
Second generation means for compression-encoding the plurality of subbands to generate a plurality of compressed subbands;
Arrangement means that collectively arranges the plurality of compressed subbands for each group based on the number of layers in the frequency decomposition,
A third generation means for generating a recording file including a compressed RAW area in which the plurality of arranged compressed subbands are stored, and an information management area in which information about the plurality of compressed subbands is stored. An image coding device characterized by the above.   The arranging means assigns a group based on the number of layers in the frequency decomposition of the compression subbands of the luminance component, the first color difference component, and the second color difference component of the four plane images to a lower layer. In order from the group of higher hierarchy to the group of higher layers, and the third color difference component of the four plane images is compressed of the luminance component, the first color difference component, and the second color difference component. The image coding apparatus according to claim 1, wherein the image coding apparatus is arranged after the subband.   3. The image coding apparatus according to claim 1, wherein the arranging unit arranges the compression subband of the luminance component first in each group. The arranging means is
A selection unit that selects two color difference components as display color difference components from the first to third color difference components;
Of the plurality of compression subbands, the compression subbands of the luminance component and the selected display color difference component are collectively arranged for each group based on the number of layers,
Of the plurality of compressed subbands, the compressed subbands of the color difference components that are not selected as the display color difference components among the first to third color difference components are different from the RAW reproduction different from the group based on the number of layers. The image coding apparatus according to claim 1, wherein the image coding apparatus is arranged in a configuration data group. Further comprising a reading unit for reading, from the recording file, a compressed subband necessary for displaying one reduced image of a plurality of display layers corresponding to the respective number of layers in the frequency decomposition.
The plurality of display layers, the higher the number of layers in the corresponding frequency decomposition, the larger the display size of the reduced image displayed,
The information management area stores, as decoding information for decoding the compressed RAW image, information on a data size of a compressed subband necessary for displaying each reduced image of the plurality of display layers,
The data size of the compressed subbands necessary for displaying the reduced image of the display layer corresponding to the layer n in the frequency decomposition is the data size of the compressed subbands collectively arranged in groups based on each of the layers 1 to n. The image coding device according to claim 4, wherein the image coding device is a total sum.   The group based on the layer M excluding the minimum number of layers of the number of layers is a group based on the first sub-layer in which the compression subbands of the luminance component are collectively arranged and a compression subband of the color difference component for display are combined. 6. The image coding apparatus according to claim 5, wherein the image coding apparatus is configured by a group based on the second sub-layer arranged in the same manner. The second sub-tier is
A first color difference layer in which compressed subbands used for upsampling pixels in either the horizontal direction or the vertical direction of the display color difference component are collected;
7. The image coding apparatus according to claim 6, wherein the image coding apparatus is configured by a second color difference layer in which compressed subbands that do not require upsampling of pixels of the display color difference component are collected. The decryption information is
The data size of the compressed sub-band necessary for displaying the reduced image of each of the display layers corresponding to the first sub-layer and the first and second color difference layers,
The data size of the compressed sub-band required for displaying the reduced image of the display layer corresponding to the first sub-layer is equal to that of the group based on the first sub-layer of the layer M and the layers 1 to M-1. It is the sum of the data sizes of the compressed subbands that are collectively arranged in groups based on each
The data size of the compressed subband necessary for displaying the reduced image of the display layer corresponding to the first color difference layer is the compressed subband required for displaying the reduced image of the display layer corresponding to the first sublayer. And a data size of compressed subbands collectively arranged in a group based on the first color difference layer of the layer M,
The data size of the compressed sub-band required for displaying the reduced image in the display layer corresponding to the second color difference layer is the compressed sub-band required for displaying the reduced image in the display layer corresponding to the first color difference layer. 9. The image coding apparatus according to claim 7, wherein the data size is the sum of the data size of compressed subbands that are collectively arranged in a group based on the second color difference layer of the layer M. A conversion step of converting the RAW image into four plane images of a luminance component and first to third color difference components;
A first generation step of performing hierarchical frequency decomposition on the four plane images to generate a plurality of subbands, respectively;
A second generating step of compression-encoding the plurality of subbands to generate a plurality of compressed subbands;
An arrangement step of collectively arranging the plurality of compressed subbands for each group based on the number of layers in the frequency decomposition,
A third generation step of generating a recording file including a compressed RAW area in which the plurality of arranged compressed subbands are stored, and an information management area in which information regarding the plurality of compressed subbands is stored. Control method characterized by. A program for executing the control method according to claim 9.

Images