JP2008193530A - Image recorder, image recording method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To materialize such a coding technology that image quality of a main subject area is not deteriorated. <P>SOLUTION: This image recorder acquires a distance image representing distance to the subject and a main image obtained by imaging the subject and records each of the images by coding them, the image recorder is provided with: a distance calculation part 122 which divides the distance image into a plurality of blocks to calculate distance information for each block; a code amount determination part 530 which divides the main image into the plurality of blocks so as to correspond to the distance image and determines a code amount for each block of the main image based on the distance image for each block of the distance image; and a recording part 110 which codes and records the main image according to the determined coding amount. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a technique for simultaneously capturing a distance image representing a distance to a subject and a main image including at least luminance information of the subject, and encoding and recording each image.

  In recent years, in order to capture and record an image as digital data, a technique for encoding image information with a high compression rate and high image quality has been used. For moving images, an MPEG system that compresses and encodes image information using redundancy unique to image information is widely used. MPEG compression is a coding method in which DCT transform, quantization, variable length coding, and forward motion compensation interframe prediction are combined with bidirectional motion compensation interframe prediction. An image is encoded with three picture types of I, P, and B. An I picture is an image that is encoded using only its own information, a P picture is encoded by performing prediction from the I or P picture, and a B picture is an image that is encoded by bidirectional prediction.

  The image recording apparatus includes a DCT conversion unit, a quantization unit, and a quantization control unit as a basic configuration. The DCT transform unit performs two-dimensional orthogonal transform on a plurality of blocks (DCT blocks) obtained by dividing image information into a predetermined number of pixels. The quantization unit quantizes the DCT coefficient after conversion. The quantization control unit controls the quantization scale code to an appropriate value in consideration of the capacity of the output buffer.

  In addition, a technique has been proposed in which the amount of information after compression encoding is kept within a predetermined range and image quality deterioration is minimized (see, for example, Patent Document 1).

  Here, a process of determining the code amount according to the pattern of the image will be described.

  FIG. 11 is a block diagram showing a configuration of a conventional image recording apparatus that compresses and encodes an image by the MPEG method.

  In FIG. 11, 1100 is an image acquisition unit that acquires an image to be compressed. Reference numeral 1101 denotes an image frame buffer for storing and rearranging image data. Reference numeral 1102 denotes a subtraction unit that takes a difference between the input image and the local decoded image. 1103 is a DCT transform unit, 1104 is a quantization unit, 1105 is an inverse quantization unit, 1106 is an inverse DCT transform unit, 1107 is a motion compensation unit, 1108 is a motion estimation unit, 1109 is an encoding unit, and 1110 is an output buffer. . Reference numeral 1120 denotes a complexity calculation unit, and 1130 denotes a quantization control unit.

  Next, code amount determination processing by the image recording apparatus in FIG. 11 will be described.

  The image input to the image acquisition unit 1100 is accumulated in the image frame buffer 1101. The image frame buffer 1101 rearranges the image frames in the encoding order. Since the B picture refers to the temporally subsequent picture, the encoding order is after the picture to be referred is encoded. The images rearranged in the encoding order are divided into macroblock units, which are small areas having a predetermined size, and are output to the subtraction unit 1102.

  In the case of an I picture, the subtraction unit 1102 simply outputs data to the DCT conversion unit 1103 without performing subtraction processing. If the image is a B or P picture, a difference image between a prediction image based on inter-frame prediction described later and the current image is output to the DCT conversion unit 1103.

  The DCT conversion unit 1103 performs DCT conversion in units of DCT blocks and converts them into frequency components. The quantization unit 1104 quantizes the converted frequency component data according to the quantization parameter. The image data quantized by the quantization unit 1104 is output to the encoding unit 1109 and the inverse quantization unit 1105 for local decoding. The encoding unit 1109 performs variable length encoding that assigns a shorter code amount to data with a higher appearance frequency, temporarily stores the data in the output buffer 1110, and then outputs the encoded data.

  On the other hand, the inverse quantization unit 1105 inversely quantizes the image data quantized by the quantization unit 1104 and decodes it to frequency components. The prediction error image (difference image) is decoded by inverse orthogonal transform by the inverse DCT transform unit 1106. The motion estimation unit 1108 searches the local decoded image for a reference image having the smallest difference from the image in the inter-frame prediction mode, and calculates a motion vector. The motion compensation unit 1107 performs a calculation indicated by the motion vector and the reference direction information and outputs a motion compensated image.

  The quantization control unit 1130 determines a quantization parameter (quantization scale) for each macroblock based on the complexity of the pattern for each macroblock and the VBV buffer. Increasing the quantization scale enables rough quantization (less code amount assignment), and conversely, fine quantization (more code amount assignment).

  FIG. 12 shows a processing flow for determining the quantization parameter for each macroblock by the quantization control unit of FIG.

  In FIG. 12, in S1201, the allocated bit amount for each picture in the GOP (Group Of Pictures) is determined. Specifically, the code amount is allocated to the picture to be encoded based on the number of pictures not yet encoded in the GOP and the complexity of the previous picture. This distribution is repeated in the order of the encoded pictures in the GOP, and the picture target bit amount is calculated for each picture.

  In S1202, a quantization scale reference value is set for each macroblock. That is, in order to match the allocated bit amount for each picture obtained in S1201 with the actual generated bit amount, the reference value of the quantization scale is obtained by feedback control in units of macroblocks based on the capacity of the output buffer.

  In S1203, the code amount allocated to each block is corrected based on the complexity. Specifically, the quantization scale value is corrected based on the block complexity in units of blocks in order to reflect visual characteristics. While maintaining the frame target bit amount, the quantization scale is corrected to be smaller than the reference value in the macro block with low complexity, and the quantization scale is corrected to be larger than the reference value in the macro block with high complexity. Thus, adaptive quantization is performed in consideration of visual characteristics, particularly complexity. As a result, it is possible to assign a large amount of code to a flat portion that is visually conspicuously deteriorated, and to assign a small amount of code to a portion of a complex picture that is hardly conspicuous.

On the other hand, for example, Patent Document 2 discloses an imaging technique for generating a distance image including distance information indicating a distance to a subject.
JP 2004-194076 A JP-A-8-242469

  In the above-described conventional technology, since the code amount allocation is determined only by the image pattern, when the main subject in the image includes a complex pattern, the code amount allocation can be reduced, There is a problem that the main subject area is deteriorated.

  The present invention has been made in view of the above problems, and an object thereof is to realize an encoding technique that does not deteriorate the image quality of the main subject area.

  In order to solve the above problems and achieve the object, an image recording apparatus of the present invention acquires a distance image representing a distance to a subject and a main image obtained by imaging the subject, and encodes each image. In the image recording apparatus, the distance image is divided into a plurality of blocks, distance calculation means for calculating distance information for each block, and the main image is divided into a plurality of blocks so as to correspond to the distance image. And a code amount determining means for determining a code amount for each block of the main image based on distance information for each block of the distance image, and encoding the main image according to the determined code amount. Recording means for recording.

  The image recording method of the present invention is an image recording method for acquiring a distance image representing a distance to a subject and a main image obtained by imaging the subject, and encoding and recording each image. A distance calculating step of dividing the distance image into a plurality of blocks and calculating distance information for each block; and dividing the main image into a plurality of blocks so as to correspond to the distance image; A code amount determining step for determining a code amount for each block of the main image based on distance information for each, and a recording step for encoding and recording the main image according to the determined code amount. .

  According to the present invention, it is possible to perform encoding so as not to deteriorate the image quality of the main subject by assigning the code amount using not only the picture of the image but also the distance information with respect to the subject.

  The best mode for carrying out the present invention will be described below in detail with reference to the accompanying drawings.

  The embodiment described below is an example as means for realizing the present invention, and should be appropriately modified or changed according to the configuration and various conditions of the apparatus to which the present invention is applied. It is not limited to the embodiment.

[First Embodiment]
First, an example in which code amount allocation is determined based on subject distance information according to the first embodiment of the present invention will be described.

  FIG. 1 is a block diagram showing the configuration of the image recording apparatus according to the first embodiment of the present invention.

  In FIG. 1, reference numeral 100 denotes a main image acquisition unit that acquires a main image to be compressed. Reference numeral 101 denotes a main image frame buffer for accumulating and rearranging main image data. Reference numeral 102 denotes a subtraction unit that takes a difference between the input image and the local decoded image. 103 is a DCT transform unit, 104 is a quantization unit, 105 is an inverse quantization unit, 106 is an inverse DCT transform unit, 107 is a motion compensation unit, 108 is a motion estimation unit, 109 is an encoding unit, and 110 is an output buffer. . Reference numeral 120 denotes a distance image acquisition unit, 121 denotes a distance image frame buffer, 122 denotes a block distance calculation unit, and 130 denotes a quantization control unit. The main image and the distance image are simultaneously captured by the imaging unit 140. The main image is a moving image or a still image.

  The imaging unit 140 simultaneously captures and captures a lens optical system including a focus lens and a zoom lens into which a light beam from a subject is incident, a main image including a luminance signal and a color difference signal, and a distance image representing a distance to the subject. And an image pickup device such as a CMOS that can be obtained. For example, the image sensor irradiates the subject with light from the LED, and obtains the distance by measuring the time until the CMOS sensor receives the reflected light for each pixel.

  Note that the image recording apparatus according to the present embodiment includes an imaging unit 140 to constitute an imaging apparatus such as a digital camera. When the imaging unit 140 is not installed, an electronic device such as a PC (personal computer) that can be connected to a camera equipped with the imaging unit 140 via a USB interface or the like is configured.

  Next, the code amount determination process by the image recording apparatus of FIG. 1 will be described.

  The image input to the main image acquisition unit 100 is accumulated in the main image frame buffer 101. The main image frame buffer 101 rearranges the image frames in the encoding order. Images rearranged in the encoding order are divided into macroblock units and output to the subtraction unit 102. In the case of an I picture, the subtraction unit 102 simply outputs data to the DCT conversion unit 103 without performing subtraction processing. When the input image is a B or P picture, a difference image between a prediction image based on inter-frame prediction described later and the current image is output to the DCT conversion unit 103. The DCT conversion unit 103 performs DCT conversion in units of DCT blocks and converts them into frequency components. The quantization unit 104 quantizes the input frequency component data according to a quantization parameter described later. The image data quantized by the quantization unit 104 is output to the encoding unit 109 and the inverse quantization unit 105. The encoding unit 109 performs variable length encoding and records the bit stream on a recording medium such as an optical disk or a hard disk via the output buffer 110.

  On the other hand, the inverse quantization unit 105 performs inverse quantization and decodes the frequency component. A prediction error image is decoded by inverse orthogonal transform by the inverse DCT transform unit 106. The motion estimation unit 108 searches for a reference image having the smallest difference from the input image from the local decoded image in the inter-frame prediction mode, calculates a motion vector, and outputs it. The motion compensation unit 107 performs a calculation indicated by the motion vector and the reference direction information, and outputs a motion compensated image. The distance image acquisition unit 120 acquires a distance image corresponding to the main image.

  Here, the distance image will be described.

  FIG. 2A illustrates a main image, and FIG. 2B illustrates a distance image.

  The main image is an image including luminance and color signals to be encoded. The distance image is an image indicating the distance to the subject corresponding to the horizontal and vertical two-dimensional position of the main image (FIG. 2A) at the same time.

  In FIG. 2B, the distance image is expressed with 256 gradations. That is, it is composed of 256-step distance information of the subject. In this case, the distance image is expressed closer to white as the subject is closer to the camera, and closer to black as the subject is closer to the camera. In the distance image of FIG. 2B, the person area indicated by 200 in the main image (FIG. 2A) is at a short distance, and the background area indicated by 202 is at a long distance. .

  Returning to FIG. 1, the distance image acquired by the distance image acquisition unit 120 is accumulated in the distance image frame buffer 121. The distance image frame buffer 121 rearranges the distance image frames in the same order as the main image is encoded. The distance images rearranged in the encoding order are divided into regions equivalent to the blocks of the main image to be quantized and output to the block distance calculation unit 122. The block distance calculation unit 122 calculates an average distance (hereinafter, block distance) for each block of the distance image.

  FIG. 3 is a diagram illustrating a state in which the main image and the distance image in FIG. 2 are divided into blocks.

  When the block 301 of the main image (a) is quantized, the block 302 of the distance image (b) corresponding to the block 301 of the main image is input to the block distance calculation unit 122, and the representative distance of the block 302 is calculated by arithmetic processing. Calculated. Here, the average value of the block distance is calculated as the block representative distance. The block distance calculation unit 122 outputs the block representative distance calculated as described above to the quantization control unit 130 as block distance information.

  The quantization control unit 130 determines a quantization parameter (hereinafter referred to as a quantization scale) for each macroblock based on the distance information for each block and the buffer capacity, and outputs the quantization parameter to the quantization unit 104.

  FIG. 4 shows a processing flow for determining the quantization scale for each block by the quantization control unit of the present embodiment.

  In FIG. 4, first, in S401, the amount of allocated bits for each picture in the GOP is determined. Specifically, the code amount is allocated to the picture to be coded based on the number of pictures not coded in the GOP and the complexity of the previous picture. This distribution is repeated in the order of the encoded pictures in the GOP, and the picture target bit amount is calculated for each picture.

  In S402, a quantization scale reference value is set for each macroblock. That is, in order to match the allocated bit amount for each picture obtained in S401 with the actual generated bit amount, the reference value of the quantization scale is obtained by feedback control in units of macroblocks based on the capacity of the output buffer 110.

  In step S <b> 403, the quantization scale is corrected based on the distance information in units of blocks and is output to the quantization unit 104. Here, it is determined that there is a high possibility that a block having a short block distance is a main subject as indicated by 200 in FIG. 2, and the quantization scale is corrected to be smaller than the reference value. On the other hand, a block having a long block distance is determined to be a subject area having a low importance such as a background as indicated by 201 in FIG. 2, and the quantization scale is corrected to be larger than the reference value.

  According to the embodiment, the quantization scale is determined based on the distance information. Accordingly, a large amount of code is allocated to the subject 200 that is close to the distance shown in FIG. 2A and is likely to be the main subject, and the subject 201 that is likely to be far in the background as shown in FIG. 2A. Can be assigned a small amount of code. Therefore, encoding can be performed so that the image quality of the main subject does not deteriorate.

  In the above example, the code amount to be allocated to the block is determined based only on the distance information and the buffer capacity. However, the feature information on other images may be extracted and combined to determine the code amount allocation. For example, the complexity of the main image described in the conventional example can be calculated, and the code amount to be assigned can be determined by comprehensively considering the complexity information, the distance information, and the buffer information.

  Also, the quantization rate based on the distance information can be increased toward the center of the image.For example, a large amount of code is assigned to a subject that is near the distance in the center of the image, and the edge of the screen is closer to the distance. It is also possible to control so as not to increase the number of subjects in the room.

  In the above embodiment, the code amount is assigned to the block closer to the subject. However, a function (subject distance setting means) that allows the user to arbitrarily set the subject distance range where the subject exists may be provided. In this case, a large amount of code can be assigned to a block corresponding to the subject distance range set by the user. In addition, a function may be provided so that the user can arbitrarily designate a subject to which a large code amount is assigned. The code amount may be allocated only when the compression rate is higher than a specified value at which distortion due to quantization is easily visible.

  The block distance calculation unit 122 calculates the average value in the block as the block representative distance, but uses the maximum value, the minimum value, or the median value of the distance corresponding to the pixels in the block as the block representative distance. You can also

[Second Embodiment]
Next, a process for determining a quantization scale for each block based on image capturing conditions and distance information will be described as a second embodiment.

  FIG. 5 is a block diagram illustrating a configuration of the image recording apparatus according to the second embodiment. The same blocks as those in FIG. 1 are denoted by the same reference numerals and description thereof is omitted.

  In FIG. 5, reference numeral 530 denotes a quantization control unit that determines a quantization parameter, and reference numeral 540 denotes an imaging condition acquisition unit that acquires information about imaging conditions at the time of imaging a main image or a distance image.

  Next, the code amount determination process by the image recording apparatus of the present embodiment will be described.

  Since the basic operation until the main image acquisition unit 100 acquires and encodes the main image is the same as that of the first embodiment, description thereof is omitted here. The operation of acquiring the distance image in the distance image acquisition unit 120 and outputting the distance information for each block to the quantization control unit 530 is the same as that of the first embodiment.

  In the present embodiment, the operation of the quantization control unit 530 is changed with respect to the first embodiment, and an imaging condition acquisition unit 540 is added. The shooting condition acquisition unit 540 acquires shooting conditions when shooting the main image and the distance image. Here, the shooting condition is information indicating a camera setting state at the time of shooting such as a shooting mode such as “portrait mode” and “landscape mode”, a zoom state (telephoto side, wide angle side), and the like. In this embodiment, a case where the shooting condition is the shooting mode will be described as an example. The shooting mode information acquired by the shooting condition acquisition unit 540 is output to the quantization control unit 530.

  The quantization control unit 530 determines a quantization parameter (quantization scale) for each macroblock based on the distance information for each block, the output buffer capacity, and the shooting mode.

  FIG. 6 shows a processing flow for determining the quantization parameter for each block by the quantization control unit of the second embodiment.

  In FIG. 6, S401 and S402 are the same as those in the first embodiment.

  In step S603, it is determined whether the shooting mode is a mode for shooting mainly in the foreground. If it is determined that the foreground mode is selected, the process proceeds to S604. On the other hand, if the mode is not a mode for photographing a foreground, the process proceeds to S605. Here, examples of the shooting mode for shooting mainly in the foreground include “portrait mode” and “party mode”.

  FIG. 7A illustrates subjects photographed in “portrait mode” and FIG. 7B illustrates “landscape mode”. In the case of FIG. 7A, the main subject is a central person indicated by 700. On the other hand, in the case of FIG. 7B, the main subject is a distant view such as a tree.

  Returning to FIG. 6, in step S <b> 604, the quantization scale is corrected so as to increase the amount of code to be assigned to the subject region closer to the short distance side, and output to the quantization unit 104. In FIG. 7, the “portrait mode” in (a) is the image processed in this step. In FIG. 7A, the quantization scale is corrected so that a large amount of code is allocated to a person region at a short distance.

  On the other hand, in step S <b> 605, the quantization scale is corrected so as to increase the amount of code to be assigned to the subject area on the far side, and output to the quantization unit 104. In FIG. 7, the “landscape mode” in (b) is the image processed in this step. In FIG. 7B, the quantization scale is corrected so that a large amount of code is allocated to a subject such as a tree at a long distance. In the “landscape mode”, the control for increasing the code amount allocation may not be performed.

  By performing the quantization process based on the above-described flow, it is possible to assign a large amount of code to the image at the subject distance corresponding to the shooting mode.

  According to the above embodiment, the quantization scale is determined based on the shooting conditions and the distance information. As a result, when the shooting mode is a mode for mainly shooting a foreground, code amount allocation is increased for a subject with a short distance. In addition, when the mode is not a mode in which a close-up scene is mainly photographed, it is possible to allocate a large amount of code of a subject with a high possibility of a background having a long distance. Therefore, a large amount of code can be assigned to a more important subject, and encoding can be performed so that the image quality of the main subject does not deteriorate.

  The imaging conditions are acquired from the imaging unit 140 when the image recording apparatus according to the present embodiment is a digital camera equipped with the imaging unit 140, and the imaging unit is installed when the image recording apparatus is a PC or the like not equipped with the imaging unit 140. It can be acquired from the camera via a USB interface or the like.

  Further, although only the photographing mode has been described as an example of the photographing condition, it is also possible to control the quantization using photographing conditions other than the photographing mode. For example, a configuration using zoom information at the time of shooting is possible. In this case, when the zoom state is on the tele side, the composition as shown in FIG. 7A is likely to increase. Control to allocate quantity. On the other hand, when the zoom state is on the wide side, it is considered that the composition as shown in FIG. 7B increases, so that a large amount of code is assigned even to a subject area with a relatively long subject distance. To control.

  It is also possible to perform control using focus information (focus information) as a shooting condition. In this case, the subject distance corresponding to the focus position is calculated. Then, control is performed so that a large amount of code is assigned to a subject close to the subject distance.

[Third Embodiment]
Next, an example in which an edge is detected from a distance image and quantization is controlled based on the edge information will be described as a third embodiment according to the present invention.

  FIG. 8 is a block diagram illustrating a configuration of an image recording apparatus according to the third embodiment. The same blocks as those in FIG. 1 are denoted by the same reference numerals and description thereof is omitted.

  In FIG. 8, reference numeral 830 denotes a quantization control unit that determines quantization parameters, and reference numeral 840 denotes an edge information calculation unit that calculates edge information from a distance image.

  Next, the code amount determination process by the image recording apparatus of the present embodiment will be described.

  Since the basic operation until the main image acquisition unit 100 acquires and encodes the main image is the same as that of the first embodiment, description thereof is omitted here. In addition, the distance image acquisition unit 120 acquires distance images, accumulates them in the distance image frame buffer 121, and rearranges them in the order of encoding, as in the first embodiment.

  In the present embodiment, the operation in the quantization control unit 830 is changed with respect to the first embodiment, and an edge information calculation unit 840 is added.

  The edge information calculation unit 840 acquires an image corresponding to a block area to be encoded next from the main image frame buffer 101 and the distance image frame buffer 121.

  FIG. 9 is a diagram showing blocks to be encoded with the main image and the distance image.

  The edge information calculation unit 840 calculates the edge strength of the block to be processed next based on the main image and the distance image. Further, the complexity of the image is also calculated based on the main image in the same manner as the complexity calculating unit in the conventional example. The edge strength here is an index that increases as it is determined to be an edge. In this embodiment, the higher edge strength of the main image or the distance image is adopted as the edge strength of the block. For example, in the main image block 901 shown in FIG. 9A, since the background includes a complex component, it is determined as a complex part, and the edge strength is low. On the other hand, in the distance image shown in FIG. 9B, since the background and the person are clearly separated in the block 911 corresponding to the block 901, it is determined as an edge and the edge strength is increased. As a result, the edge strength by the block 911 of the distance image is adopted in the block areas indicated by reference numerals 901 and 911 in FIG. Conversely, the block shown in the main image block 902 has a high edge strength in the main image, but the corresponding block 912 in the distance image is determined to be a flat portion, and the edge strength is low. In the case of blocks 902 and 912, the edge strength by the block 902 of the main image is adopted. As described above, the main image is excellent in detecting the edge of the pattern, and the distance image is excellent in detecting the edge of the subject itself. For this reason, it is possible to accurately detect an edge by using both pieces of information. As for edge detection, various methods have been proposed so far, and details thereof are omitted.

  The quantization control unit 830 determines a quantization parameter (quantization scale) for each block based on the edge strength and output buffer for each block.

  FIG. 10 shows a processing flow for determining the quantization parameter for each block by the quantization control unit of the third embodiment.

  In FIG. 10, S401 and S402 are the same as those in the first embodiment.

  In step S <b> 1003, the quantization scale value is corrected based on the macroblock complexity in units of macroblocks to reflect visual characteristics, and is output to the quantization unit 104. For macroblocks with high edge strength or low complexity while maintaining the frame target bit amount, the quantization scale is corrected to be smaller than the reference value. For macroblocks with low edge strength and high complexity, the quantization scale is corrected from the reference value. Correct greatly. As a result, it is possible to assign a large amount of code to the edge portion and flat portion where deterioration is easily noticeable, and assign a small amount of code to a portion of a complex picture that is difficult to deteriorate.

  According to the embodiment, the edge of the subject is detected based on the distance image, and the quantization scale is corrected based on the edge detection result. Thereby, even if it is difficult to determine the edge from the pattern of the main image, the edge can be determined from the distance image, and the edge detection accuracy is improved. Also, by assigning a large amount of code to the edge portion and the flat portion and assigning a small amount of code to the complex portion of the pattern, the accuracy of the quantization processing can be improved.

  In the above embodiment, an example in which an edge is detected from both the main image and the distance image has been described. However, the code amount may be determined based on the edge detection result only in the distance image, and in this case, the edge detection accuracy is reduced. However, the calculation process can be simplified.

[Other Embodiments]
The object of the present invention can also be achieved by the following method. That is, a storage medium (or recording medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded is supplied to the system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but the present invention includes the following cases. That is, based on the instruction of the program code, an operating system (OS) or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

  Furthermore, the following cases are also included in the present invention. That is, the program code read from the storage medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, a CPU or the like provided in the function expansion card or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the above storage medium, the storage medium stores program codes corresponding to the procedure described above.

1 is a block diagram illustrating a configuration of an image recording apparatus according to a first embodiment of the present invention. It is a figure which illustrates the main image (a) and distance image (b) by 1st Embodiment based on this invention. It is a figure explaining each quantization process of the main image (a) and distance image (b) by 1st Embodiment which concerns on this invention. It is a flowchart which shows the code amount determination process of 1st Embodiment which concerns on this invention. It is a block diagram which shows the structure of the image recording device of 2nd Embodiment which concerns on this invention. It is a flowchart which shows the code amount determination process of 2nd Embodiment which concerns on this invention. It is a figure which illustrates the to-be-photographed object for every imaging | photography mode by 2nd Embodiment which concerns on this invention. It is a block diagram which shows the structure of the image recording device of 3rd Embodiment based on this invention. It is a figure explaining each quantization process of the main image (a) and distance image (b) by 3rd Embodiment which concerns on this invention. It is a flowchart which shows the code amount determination process of 3rd Embodiment which concerns on this invention. It is a block diagram which shows the structure of the image recording apparatus of a prior art example. It is a flowchart which shows the code amount determination process by a prior art example.

Explanation of symbols

100 main image acquisition unit 101 main image frame buffer 102 subtraction unit 103 DCT conversion unit 104 quantization unit 105 inverse quantization unit 106 inverse DCT conversion unit 107 motion compensation unit 108 motion estimation unit 109 encoding unit 110 output buffer 120 distance image acquisition Unit 121 distance image frame buffer 122 block distance calculation unit 130 quantization control unit

Claims (23)

In an image recording apparatus for acquiring a distance image representing a distance to a subject and a main image obtained by imaging the subject, and encoding and recording each image.
A distance calculating means for dividing the distance image into a plurality of blocks and calculating distance information for each block;
Code amount determining means for dividing the main image into a plurality of blocks so as to correspond to the distance image, and determining a code amount for each block of the main image based on distance information for each block of the distance image; ,
An image recording apparatus comprising: a recording unit that encodes and records the main image according to the determined code amount.   The image recording apparatus according to claim 1, further comprising an imaging unit that simultaneously acquires the main image and the distance image by imaging.   The image recording apparatus according to claim 2, wherein the code amount determination unit determines a code amount for each block of the main image using a quantization parameter for each block of the distance image. Further comprising feature information calculating means for calculating feature information for each block of the main image;
4. The image recording according to claim 1, wherein the code amount determination unit determines the code amount based on feature information of the main image and distance information for each block. 5. apparatus.   5. The image recording apparatus according to claim 1, wherein the code amount determination unit increases the code amount as the distance information for each block of the distance image is smaller. Shooting condition acquisition means for acquiring shooting conditions of the main image;
6. The image recording apparatus according to claim 1, further comprising: a subject distance setting unit that sets a subject distance to which the code amount is allocated based on the photographing condition.   The image recording apparatus according to claim 6, wherein the photographing condition is a photographing mode of the subject.   The image recording apparatus according to claim 7, wherein the subject distance setting unit sets the subject distance closer to a closer distance than a specified value when the shooting mode is a mode for mainly shooting a close-up scene.   9. The image recording according to claim 7, wherein the subject distance setting means sets the subject distance to a far side from a specified value when the photographing mode is a mode for photographing a distant view. apparatus.   The image recording apparatus according to claim 6, wherein the photographing condition is zoom information.   The image recording apparatus according to claim 10, wherein the subject distance setting unit sets the subject distance closer to a closer side as the zoom state is closer to a telephoto side.   12. The image recording apparatus according to claim 10, wherein the subject distance setting unit sets the subject distance to a far side as the zoom state is on a wide angle side.   The image recording apparatus according to claim 6, wherein the photographing condition is focus information.   The image recording apparatus according to claim 13, wherein the subject distance setting unit sets a range in which the focus is in focus as the subject distance.   15. The image recording apparatus according to claim 6, further comprising designation means for a user to designate the subject distance.   The image recording apparatus according to claim 6, further comprising means for designating a subject for setting the subject distance.   The image recording apparatus according to any one of claims 1 to 16, wherein the code amount determination unit increases the code amount toward a central portion of the main image.   18. The image recording apparatus according to claim 1, wherein the code amount determination unit assigns the code amount when a compression rate of the main image is higher than a predetermined value. Edge information calculating means for calculating edge information of the subject from the distance image;
The image recording apparatus according to claim 1, wherein the code amount determination unit determines the code amount based on edge information of the distance image. Edge information calculating means for calculating edge information of the subject from the main image;
The image recording apparatus according to claim 1, wherein the code amount determination unit determines the code amount based on edge information of the main image.   21. The image recording apparatus according to claim 19, wherein the code amount determination unit increases a code amount of a block corresponding to an edge portion of the subject. An image recording method for acquiring a distance image representing a distance to a subject and a main image obtained by imaging the subject, and encoding and recording each image,
A distance calculating step of dividing the distance image into a plurality of blocks and calculating distance information for each block;
A code amount determining step for dividing the main image into a plurality of blocks so as to correspond to the distance image, and determining a code amount for each block of the main image based on distance information for each block of the distance image; ,
And a recording step of encoding and recording the main image according to the determined code amount. A computer of an image recording apparatus for acquiring a distance image representing a distance to a subject and a main image obtained by imaging the subject and encoding and recording each image,
A distance calculating means for dividing the distance image into a plurality of blocks and calculating distance information for each block;
Code amount determining means for dividing the main image into a plurality of blocks so as to correspond to the distance image, and determining a code amount for each block of the main image based on distance information for each block of the distance image; ,
A program that functions as a recording unit that encodes and records the main image in accordance with the determined code amount.