JP4458408B2 - Image processing apparatus, image processing method, program, and information recording medium - Google Patents

Description

The present invention relates to image processing, and more specifically, an apparatus for reproducing a plurality of images having different in-focus distances from a composite image synthesized from a plurality of images photographed at different in-focus distances at the same angle of view. And a method.

  One of the problems in image storage and transmission is the large amount of image data. In particular, in recent image input devices such as digital cameras, image resolution has increased due to a dramatic increase in the number of pixels of an image sensor, and the amount of image data has become enormous.

  For such an image having a large amount of data, it is indispensable to perform an efficient compression process at the time of storage or transmission. And in order not to impair the merit of a high-definition image, it is necessary to reduce degradation of the image quality accompanying compression processing as much as possible.

  One image compression (encoding) method that satisfies these requirements is JPEG2000, which employs wavelet transform for frequency conversion. The outline of JPEG2000 will be described next.

  FIG. 10 is a block diagram showing the basic flow of JPEG2000 compression (encoding) / decompression (decoding) processing.

  The compression (encoding) process will be described. For example, a color image composed of three components of RGB and CMY is divided into one or more non-overlapping tiles for each component, and processing is performed for each tile of each component. First, for each tile, the color space conversion / inverse color conversion unit 100 performs component conversion (color space conversion) to luminance / color difference components, and then the two-dimensional wavelet conversion / inverse conversion unit 101 performs conversion. A two-dimensional wavelet transform (discrete wavelet transform) is performed for each tile of each component.

  The result of the wavelet transform is illustrated in FIG. In this example, the number of decomposition levels is 3, and the wavelet transform coefficients are shown in a deinterleaved state. 1LH, 1HL, and 1HH are LH, HL, and HH subbands at decomposition level 1, 2LH, 2HL, and 2HH are LH, HL, and HH subbands at decomposition level 2, and 3LH, 3HL, 3HH, and 3LL are at decomposition level 3. LH, HL, HH, and LL subbands, and the coefficient of the subband having a lower decomposition level has a higher frequency component.

  The wavelet transform coefficient is quantized by the quantization / inverse quantization unit 102 as necessary for each subband, and then entropy-coded in units of bit planes by the entropy coding / decoding unit 103. (To be precise, the bit plane is subdivided into three sub-bit planes and encoded). Then, the code forming / tag processing unit 104 truncates unnecessary codes, collects necessary codes, generates packets, arranges the packets in a predetermined order, and adds necessary tags or tag information. A code stream (encoded data) of the format is formed.

  Note that the color space conversion / inverse color conversion unit 100 is not essential. JPEG2000 defines a reversible wavelet transform called a 5 × 3 transform and an irreversible wavelet transform called a 9 × 7 transform. When the 5 × 3 wavelet transform is used, quantization (scalar quantization) by the quantization / inverse quantization unit 102 is not performed, but truncation in the code formation / tag processing unit 104 is equivalent to coefficient quantization. This is called post-quantization.

  The decompression (decoding) process is just the reverse of the compression process. The code stream is decomposed into a code stream of each tile of each component by the code formation / tag processing unit 104, entropy-decoded in units of bit planes by the entropy encoding / decoding unit 103, and the decoded wavelet transform coefficients are quantized, Inverse quantization is performed by the inverse quantization unit 102. Then, the two-dimensional wavelet transform / inverse transform unit 101 performs two-dimensional inverse wavelet transform on the wavelet transform coefficient for each component, and then the color space transform / inverse color transform unit 100 performs inverse component transform (inverse). (Color space conversion) is performed to restore the original RGB and CMY pixel values.

  The wavelet transform is also applied for purposes other than image compression. For example, a three-dimensional wavelet transform is applied to a tomographic image taken by a magnetic resonance tomography apparatus, and an intensity conversion function is applied only to the obtained high-frequency component, thereby reducing a noise component and a three-dimensional image. A technique for improving the overall image quality is known (see Patent Document 1).

JP 2000-30044 A

In a digital camera or the like, an object at and near the in-focus distance is clearly photographed, but blurring occurs in an image of an object that is closer or further away. In some cases, the taste of blur is emphasized, but conversely, an image focused on a subject within a wider distance range may be required. Conventionally, various techniques for synthesizing a plurality of images taken at the same angle of view and at different in-focus distances have been proposed. There is almost no technology to reproduce the images.

An object of the present invention is to provide an apparatus and method for reproducing a plurality of images with different focal length than the composite image synthesized from the captured multiple images in different focal length at the same angle.

The invention of claim 1 is an image processing apparatus that generates a plurality of reproduced images having different in-focus distances from a composite image synthesized from a plurality of images photographed at different in-focus distances at the same angle of view. With reference to the means for obtaining the frequency conversion coefficient of the composite image and the distance information given for each small area of the composite image, (a) the distance information that matches the in-focus distance of the reproduced image is given, The frequency conversion coefficient of the small area of the composite image is used as it is as the frequency conversion coefficient of the corresponding small area of the reproduced image, and (b) the small area of the composite image to which distance information that does not match the in-focus distance of the reproduced image is given. The frequency conversion coefficient obtained by quantizing the frequency conversion coefficient with the quantization parameter corresponding to the distance difference between the in-focus distance indicated by the distance information and the in-focus distance of the reproduced image is the frequency of the corresponding small area of the reproduced image. For conversion factor By Rukoto, and having means for generating a frequency transform coefficients of the plurality of reproduced image.

According to a second aspect of the present invention, the image processing apparatus according to the first aspect further includes means for performing inverse frequency conversion of a frequency conversion coefficient of a reproduced image to generate image data of the reproduced image.

The invention of claim 3 is the image processing apparatus of claim 1, wherein the frequency transform coefficient is a wavelet transform coefficient.

The invention of claim 4 is an image processing method for generating a plurality of reproduced images having different in-focus distances from a composite image synthesized from a plurality of images photographed at different in-focus distances at the same angle of view. Then, with reference to the process of obtaining the frequency conversion coefficient of the composite image and the distance information given for each small area of the composite image, (a) the distance information that matches the in-focus distance of the reproduced image is given, The frequency conversion coefficient of the small area of the composite image is used as it is as the frequency conversion coefficient of the corresponding small area of the reproduced image, and (b) the small area of the composite image to which distance information that does not match the in-focus distance of the reproduced image is given. The frequency conversion coefficient obtained by quantizing the frequency conversion coefficient with the quantization parameter corresponding to the distance difference between the in-focus distance indicated by the distance information and the in-focus distance of the reproduced image is the frequency of the corresponding small area of the reproduced image. For conversion factor By Rukoto, characterized in that it comprises a process of generating a frequency transform coefficients of the plurality of reproduced image.

According to a fifth aspect of the present invention, in the image processing method according to the fourth aspect of the present invention, the method further includes a process of performing inverse frequency conversion of a frequency conversion coefficient of a reproduced image to generate image data of the reproduced image.

The invention of claim 6 is the image processing method of claim 5, wherein the frequency transform coefficient is a wavelet transform coefficient.

A seventh aspect of the invention is a program for causing a computer to function as each means of the image processing apparatus according to any one of the first to third aspects of the invention.

The invention of claim 8 is a program for causing a computer to execute each process of the image processing method according to the invention of claim 7 or 8.

The invention of claim 9 is a computer-readable information recording medium on which the program according to the invention of claim 7 or 8 is recorded.

According to the present invention, a plurality of images with different in- focus distances (for example, images closer to a pre-combination image) than a composite image synthesized from a plurality of images taken at the same angle of view and at different in-focus distances. A plurality of images) can be reproduced . Furthermore, the image data of the reproduction image can be generated by performing the inverse frequency conversion of the frequency conversion coefficient of the reproduction image. Further, by using the wavelet transform coefficients as frequency transform coefficients, excellent affinity with the compression / decompression algorithm using wavelet transform such as JPEG2000, it is readily implemented in various systems employing such compression / decompression algorithm Become.

  FIG. 1 is a block diagram for explaining a typical configuration of an image processing apparatus according to the present invention. In FIG. 1, reference numeral 200 denotes a processing unit that performs image composition processing, 201 denotes a storage unit that is used as a working storage area for the processing unit 200, and 202 denotes a compression / decompression that compresses or decompresses an image using a JPEG2000 algorithm. An expansion unit 203 is a large-capacity auxiliary storage unit for storing image data and the like, and 204 is a data input / output unit for inputting or outputting data with an external device. Such an image processing apparatus can be realized by dedicated hardware, or can be realized by using a hardware resource such as a general-purpose computer such as a personal computer or a microcomputer.

  This image processing apparatus combines a plurality of images taken at the same angle of view and at different in-focus distances into one image (combination processing), and a plurality of images having different in-focus distances from the composite image, for example, synthesis Processing (separation reproduction processing) for generating a plurality of images close to the previous image can be performed.

  First, the synthesis process will be specifically described. A plurality of images for composition processing are taken in through the data input / output unit 204 as image data (pixel value data) or as a JPEG2000 code stream, and are temporarily stored in the auxiliary storage unit 203. At this time, the distance information of each image is also input and stored in a form embedded in the image data or in a form independent of the image data.

  When an image is input as image data, the processing unit 200 can immediately perform synthesis processing on the image data. When an image is input as a code stream, the compression / decompression unit 202 decompresses the code stream into image data, and performs synthesis processing on the image data. The composite image is stored in the auxiliary storage unit 203 in the form of image data or in the form of a code stream compressed by the compression / decompression unit 202. At this time, the distance information for each small area of the composite image is also stored. The data is stored in a form embedded in the composite image data or in an independent form. Of course, the composite image data and the distance information can be output to the outside through the data input / output unit 204.

In the synthesizing process, the amount of high-frequency component for each region is obtained for each of a plurality of images. In the embodiment described here, the high-frequency component is obtained based on the wavelet transform coefficient obtained using the compression / decompression unit 202. Calculate the amount of ingredients. However, it is also possible to calculate the amount of high frequency components using frequency conversion other than wavelet conversion .

[Example 1 of composition processing]
FIG. 2 is a flowchart showing an example 1 of the synthesis process, and FIG. 3 is a schematic diagram for explaining the synthesis process.

  In the synthesis processing example 1, it is assumed that a plurality of images taken at different focal distances with the same angle of view are accumulated in the auxiliary storage unit 203 as image data. For simplification of explanation, three images P1, P2, and P3 photographed at in-focus distances 1, 2, and 3 (distance 1 <distance 2 <distance 3) as shown in FIG. Although it is assumed that the images are combined, it is clear that four or more images can be combined by the same processing. As the number of images to be synthesized is increased and the difference in focus distance between them is reduced, a synthesized image closer to a pan-focus image can be obtained.

  The contents of the synthesis process will be described with reference to the flowchart of FIG. First, in step S <b> 1, the image data of the images P <b> 1, P <b> 2, P <b> 3 and the distance information (focusing distances 1, 2, 3) are read from the auxiliary storage unit 203 into the memory 201 under the control of the processing unit 200.

  In step S2, under the control of the processing unit 200, the compression / decompression unit 202 sequentially reads the image data of the images P1, P2, and P3 from the storage unit 201 and executes processing up to the two-dimensional wavelet transform of the compression processing. That is, the compression / decompression unit 202 has a basic configuration as shown in FIG. 10, and a color space conversion process by the color space conversion / inverse conversion unit 100 and a two-dimensional discrete wavelet conversion process by the two-dimensional wavelet conversion / inverse conversion unit 101. Run until. However, in this example, the two-dimensional wavelet transform is performed only on the luminance component after the color space conversion. Note that when the image data of the images P1, P2, and P3 is composed of, for example, RGB components, two-dimensional wavelet transform may be performed only on the G component without performing color space conversion. Further, when the images P1, P2, and P3 are monochrome images, only the single component is subjected to wavelet transformation. The wavelet transform coefficients of the images P1, P2, and P3 generated in this way (for example, subband coefficients of decomposition levels 1, 2, and 3 as shown in FIG. 11) are stored in the storage unit 201.

  In step S3, the processing unit 200 divides each image P1, P2, P3 into small regions such as 2 × 2 pixels and 4 × 4 pixels, for example. FIG. 3A schematically shows how the image is divided into regions.

  In step S4, the processing unit 200 selects one small area. In step S6, the processing unit 200 calculates the high-frequency component amount of each image in the small region from the wavelet transform coefficient of each image corresponding to the selected small region. In this example, the calculation method can be selected from the following calculation methods (1) to (5).

Calculation method (1): The sum of the absolute values of the HL subband coefficients corresponding to the small area is obtained for each decomposition level, and summed by adding a weight according to the decomposition level, thereby calculating the amount of high frequency components. Ask. For example, when using coefficients of decomposition levels 1 to 3, the amount of high frequency component is
Yh = a · Σ | 1HL | + b · Σ | 2HL | + c · Σ | 3HL | (1) (where a, b, and c are constants greater than or equal to 0)
Is calculated by Since the HL subband coefficient is a high frequency component in the horizontal direction, the amount of high frequency component in the horizontal direction is obtained by this calculation method.

Calculation method (2): The sum of the absolute values of the LH subband coefficients corresponding to the small area is obtained for each decomposition level, and summed by adding a weight according to the decomposition level, thereby calculating the amount of high frequency components. Ask. For example, when using coefficients of decomposition levels 1 to 3, the amount of high frequency component is
Yv = d · Σ | 1LH | + e · Σ | 2LH | + f · Σ | 3LH | (2) (where d, e, and f are constants greater than or equal to 0)
Is calculated by Since the LH subband coefficient is a high-frequency component in the vertical direction, this calculation method obtains the amount of high-frequency component in the vertical direction.

Calculation method (3): The sum of absolute values of the sub-band coefficients of HL and LH corresponding to the small region is obtained for each decomposition level, and the sum is obtained by adding weights according to the decomposition level and the sub-band. Thus, the amount of high frequency component is obtained. For example, when using coefficients of decomposition levels 1 to 3, the amount of high frequency component is
Yhv ＝ a ・ Σ | 1HL | + b ・ Σ | 2HL | + c ・ Σ | 3HL | + d ・ Σ | 1LH | + e ・ Σ | 2LH | + f ・ Σ | 3LH | (3) (a, b, c, c, d, e, f are constants of 0 or more)
Is calculated by That is, the amount of high-frequency components in the horizontal direction and the horizontal direction is obtained.

Calculation method (4): The sum of absolute values of the subband coefficients of HL, LH, and HH corresponding to the small region is obtained for each decomposition level, and the sum is obtained by adding weights according to the decomposition level and the subband. By doing so, the amount of high frequency components is obtained. For example, when the coefficients of decomposition levels 1 and 2 are used, the amount of high-frequency component is Y = a · Σ | 1HL | + b · Σ | 2HL | + c · Σ | 1LH | + d · Σ | 2LH | + e · Σ | 1HH | + f · Σ | 2HH | (4) (where a, b, c, c, d, e, and f are constants of 0 or more)
Is calculated by That is, the amount of high frequency components in the horizontal direction, the horizontal direction, and the oblique direction is obtained.

  Calculation method (5): A high-frequency component amount is calculated by the same method as the calculation method (1), (2), (3) or (4) using the wavelet transform coefficient of decomposition level 2 or higher. In this way, the method that does not use the wavelet transform coefficient at the decomposition level 1 for the calculation of the high-frequency component amount is advantageous when performing synthesis processing of a noisy image such as an image taken in a low illumination environment. This is because most of the noise components are reflected in the wavelet transform coefficients at the decomposition level 1, and the noise components are not so reflected in the wavelet transform coefficients at the upper level.

  When the high-frequency component amount of each image in the selected small region is calculated as described above, in step S7, the processing unit 200 compares the calculated high-frequency component amount of each image and calculates the value. Select the image with the maximum. In step S8, the processing unit 200 copies the image data of the small area of the image selected in the previous step to the corresponding small area of the composite image storage area secured in advance on the storage unit 201. The distance information (focusing distance) of the selected image is written in the distance information storage area secured in advance on the storage unit 201 in association with the small area.

  The processing unit 200 repeats steps S4 to S8 for each small region, so that for each small region, the image data having the maximum high-frequency component amount in the images P1, P2, and P3 is the corresponding region of the composite image storage area. Are sequentially copied. When the amount of high-frequency components in the small areas shaded in the images P1, P2, and P3 shown in FIG. 3A is the largest among the three images, the image is stored in the composite image storage area as shown in FIG. The image data of the small area shaded by P1, P2, and P3 is copied. In the composite image generated in this way, the small area where the image data of the image P1 is copied is given distance information corresponding to the in-focus distance 1, and the small area where the image data of the image P2 is copied is The distance information corresponding to the in-focus distance 2 is given, and the small area where the image data of the image P3 is copied is given the distance information corresponding to the in-focus distance 3.

  When the processing for all the small areas is completed (step S5, Yes), in step S9, the compression / decompression unit 202 applies the image data of the synthesized image generated in the synthesized image storage area according to the instruction from the processing unit 200. Compression processing is performed, and the code stream is written in the storage unit 201. In step S <b> 10, the image data of the composite image stored in the storage unit 201 and the distance information thereof are transferred to the auxiliary storage unit 203 under the control of the processing unit 200.

  Note that the distance information of the composite image may be embedded in the code stream of the composite image. In the case of a JPEG2000 code stream, for example, a method of describing distance information corresponding to each small area as a comment in the main header can be adopted.

  It is also possible to save the composite image as image data in the auxiliary storage unit 203. In this case, the process of step S9 is omitted, and the image data and distance information of the composite image are stored in the auxiliary storage unit 203 in step S10. Will be transferred.

As can be understood from the above description, in the synthesis processing example 1, the compression / decompression unit 202 functions as means for obtaining the wavelet transform coefficient (frequency transform coefficient in a broad sense) of each image before synthesis. Further, the processing unit 200 uses means for obtaining a high-frequency component amount for each small region of each image from a wavelet transform coefficient (frequency conversion coefficient in a broad sense), and uses image data of an image having the largest high-frequency component amount for each small region. Acts as means for generating image data of a composite image. It is obvious that such means can be realized by one or more programs on a computer. It is also possible to provide such a program and various information recording (storage) media readable by a computer, in which the program is recorded .

[Example 2 of composition processing]
FIG. 4 is a flowchart showing a second example of the synthesis process. In this synthesis processing example 2, it is assumed that a plurality of images shot at different focal distances at the same angle of view are accumulated in the auxiliary storage unit 203 as a JPEG2000 code stream. For simplification of description, three images P1, P2, and P3 photographed at in-focus distances 1, 2, and 3 (distance 1 <distance 2 <distance 3) as shown in FIG. Is to be synthesized.

  The contents of the synthesis process will be described with reference to the flowchart of FIG. First, in step S <b> 11, the code stream and distance information (focus distances 1, 2, 3) of the images P <b> 1, P <b> 2, P <b> 3 are read from the auxiliary storage unit 203 into the memory 201 under the control of the processing unit 200. If distance information is embedded in the code stream, only the code stream is read.

  In step S12, in accordance with an instruction from the processing unit 200, the compression / decompression unit 202 sequentially reads the code streams of the images P1, P2, and P3 from the storage unit 201 and performs decompression processing. The generated image data of the images P1, P2, and P3 is written in the storage unit 201. In addition, the wavelet transform coefficients (coefficients after inverse quantization) of the luminance components of the images P1, P2, and P3 obtained during the expansion process are also written in the storage unit 201. When the distance information is embedded in the code stream, the distance information is extracted and written in the storage unit 201 in the decoding process.

  Thereafter, the processing unit 200 executes steps S13 to S18, generates image data of the composite image in the storage unit 201, and assigns distance information to each small region of the composite image. In addition, since the processing content in step S13-S18 is the same as corresponding step S3-S8 in FIG. 2, description is not repeated.

  In step S19, the image data of the composite image is compressed by the compression / decompression unit 202, and the code stream is transferred to the auxiliary storage unit 203 together with the distance information in step S20.

  As described above, the distance information for each small area of the composite image may be embedded in the code stream of the composite image. It is also possible to save the composite image as image data in the auxiliary storage unit 203. In this case, the process of step S19 is omitted, and the image data and distance information of the composite image are stored in the auxiliary storage unit 203 in step S20. Will be transferred.

As can be understood from the above description, in the synthesis processing example 2, the compression / decompression unit 202 functions as a means for obtaining a wavelet transform coefficient (frequency transform coefficient in a broad sense) of each image before synthesis. The processing unit 200 functions as a means for obtaining the amount of high-frequency components for each small area of each image from the wavelet transform coefficient (frequency conversion coefficient in a broad sense), and the image having the largest amount of high-frequency component for each small area. It acts as means for generating image data of a composite image using image data. It is obvious that such means can be realized by one or more programs on a computer. It is also possible to provide such a program and various information recording (storage) media readable by a computer, in which the program is recorded .

[Separated regeneration processing]
A plurality of images having different in-focus distances can be generated from the synthesized image generated as described above using the distance information. Here, it is assumed that images Q1, Q2, and Q3 that are close to the images P1, P2, and P3 before synthesis are reproduced. Next, a specific example of separation / reproduction processing will be described.

[Example 1 of separation / regeneration processing]
FIG. 5 is a flowchart showing an example 1 of the separation / reproduction process, and FIG. 6 is a schematic diagram for explaining the separation / reproduction process. Here, it is assumed that a composite image synthesized from images P1, P2, and P3 photographed at different focus distances with the same angle of view is accumulated in the auxiliary storage unit 203 as a JPEG2000 code stream.

  The contents of the separation reproduction process will be described with reference to the flowchart of FIG. First, in step S <b> 21, the code stream and distance information of the composite image are read from the auxiliary storage unit 203 into the memory 201 under the control of the processing unit 200. If distance information is embedded in the code stream, only the code stream is read.

  In step S22, in accordance with an instruction from the processing unit 200, the compression / decompression unit 202 sequentially reads the code stream of the composite image from the storage unit 201, and performs decompression processing up to before the inverse wavelet transform. The wavelet transform coefficients of all components obtained by this decompression process are written in the storage unit 201. When the distance information is embedded in the code stream, the distance information is extracted and written in the storage unit 201 during the decoding process.

  In step S23, the processing unit 200 selects one small area of the combined image (the size of the small area at the time of combining is known). Next, in step S25, the processing unit 200 refers to the distance information regarding the selected small region and determines which of distance 1, distance 2, and distance 3. If distance 1, the process proceeds to step S26. If the distance is 3, the process proceeds to step S28.

  In step S <b> 26, the processing unit 200 stores the wavelet transform coefficient of the small region of the composite image corresponding to the small region in the storage area for the image Q <b> 1 corresponding to the pre-combination image P <b> 1 (distance 1). Write to. In step S <b> 27, the processing unit 200 stores the wavelet transform coefficient of the small region of the composite image in the storage unit 201 in advance, the corresponding small region of the storage area for the image Q <b> 2 corresponding to the pre-combination image P <b> 2 (distance 2). Write to. In step S28, the processing unit 200 stores the wavelet transform coefficient of the small area of the synthesized image in the storage area 201, the corresponding small area of the storage area for the image Q3 corresponding to the pre-combination image P3 (distance 3). Write to.

  In the case of the composite image schematically shown in FIG. 6A, by repeating the processes of steps S26, S27, and S28 for all the small regions, the images Q1, Q2 schematically shown in FIG. , Q3, the wavelet transform coefficients of the corresponding small area of the composite image are copied as they are into the small area shaded by Q3. It is necessary to interpolate the wavelet transform coefficient for the small areas not shaded in the images Q1, Q2, and Q3, and this interpolation processing is performed in steps S29 to S34.

  Steps S29 and S30 are interpolation processing steps when the selected small area is the distance 1. In step S29, the processing unit 200 quantizes the wavelet transform coefficient of the selected small region of the composite image using a quantization parameter corresponding to the distance difference between the distance 1 and the distance 2, and the quantized wavelet transform coefficient Is written in the corresponding small area of the storage area for the image Q1. At this time, the quantization parameter is determined so that the greater the distance difference, the stronger the quantization. The quantization parameter is determined for each subband and for each component. In the next step S29, the processing unit 200 quantizes the wavelet transform coefficient of the selected small region of the composite image using a quantization parameter corresponding to the distance difference between the distance 1 and the distance 3, and the wavelet after the quantization The conversion coefficient is written in the corresponding small area of the image Q3 storage area. By such processing, the wavelet transform coefficient is interpolated in one small region of the images Q2 and Q3.

  Steps S31 and S32 are interpolation processing steps when the selected small area is the distance 2. In step S31, the processing unit 200 quantizes the wavelet transform coefficient of the selected small region of the composite image using a quantization parameter corresponding to the distance difference between the distance 1 and the distance 2, and the quantized wavelet transform coefficient Is written in the corresponding small area of the storage area for the image Q1. In the next step S32, the processing unit 200 quantizes the wavelet transform coefficient of the selected small region of the composite image using a quantization parameter corresponding to the distance difference between the distance 2 and the distance 3, and the quantized wavelet The conversion coefficient is written in the corresponding small area of the image Q3 storage area. By such processing, the wavelet transform coefficient is interpolated into one small region of the images Q1 and Q3.

  Steps S33 and S34 are steps of interpolation processing when the selected small area is the distance 3. In step S33, the processing unit 200 quantizes the wavelet transform coefficient of the selected small region of the composite image using a quantization parameter corresponding to the distance difference between the distance 1 and the distance 3, and the quantized wavelet transform coefficient Is written in the corresponding small area of the storage area for the image Q1. In the next step S34, the processing unit 200 quantizes the wavelet transform coefficient of the selected small region of the composite image using the quantization parameter corresponding to the distance difference between the distance 2 and the distance 3, and the wavelet after the quantization The conversion coefficient is written in the corresponding small area of the image Q2 storage area. By such processing, the wavelet transform coefficient is interpolated into one small region of the images Q1 and Q2.

  The process returns from step S30, S32, S34 to step S23, and the same processing is performed for another small area. When the processing of steps S23 to S34 is repeated and the writing of the wavelet transform coefficients to all the small areas of the storage areas for the images Q1, Q2, and Q3 is completed (step S24, Yes), the process proceeds to step S35. , The compression / decompression unit 202 reads the wavelet transform coefficients of each image from the storage areas for the images Q1, Q2, and Q3, and performs the inverse wavelet transform and the inverse color space transform in order, whereby the images Q1, Q2 , Q3 image data is generated on the storage unit 201. The image data of the images Q1, Q2, and Q3 are not completely the same as the image data of the images P1, P2, and P3 before synthesis, but are quite approximate. The image data of the images Q1, Q2, and Q3 generated in this way is transferred to the auxiliary storage unit 203 under the control of the processing unit 200 in step S36.

  As can be understood from the above description, in this separation reproduction processing example 1, the compression / decompression unit 202 is a means for generating a wavelet transform coefficient (frequency transform coefficient in a broad sense) of a composite image from a code stream of the composite image. It also acts as a means for generating image data from the wavelet transform coefficients of each reproduced image. Further, the processing unit 200 uses the distance information of the composite image from the wavelet transform coefficient (frequency conversion coefficient in a broad sense) of the composite image, and calculates the wavelet transform coefficients (frequency conversion coefficient in a broad sense) of a plurality of reproduced images. Acts as a means to generate. It is obvious that such means can be realized by one or more programs on a computer. Such a program and various information recording (storage) media readable by a computer on which the program is recorded are also included in the present invention.

  Note that the image data of the reproduced images Q1, Q2, and Q3 can be transferred to the image display device, for example, through the data input / output unit 204 and displayed. In step S35, the compression / decompression unit 202 performs compression processing after quantization on the wavelet transform coefficients of the images Q1, Q2, and Q3 to generate a codestream of each image, and stores the codestream in the auxiliary storage unit 203. It is possible to store the data or to output the data through the data input / output unit 204. Further, the wavelet transform coefficient of the reproduced image can be stored in the auxiliary storage unit 203 as it is, and the wavelet transform coefficient is subjected to inverse wavelet transform (and inverse color space transform if necessary) by the compression / decompression unit 202. By performing the above, it is possible to reproduce the image of the reproduced image.

[Example 2 of separation reproduction processing]
FIG. 7 is a flowchart showing an example 2 of the separation reproduction process. Here, it is assumed that a composite image synthesized from images P1, P2, and P3 photographed at the same angle of view and at different focus distances is stored in the auxiliary storage unit 203 as image data.

  The contents of the separation reproduction process will be described with reference to the flowchart of FIG. First, in step S <b> 41, image data and distance information of a composite image are read from the auxiliary storage unit 203 into the memory 201 under the control of the processing unit 200.

  In step S42, in accordance with an instruction from the processing unit 200, the compression / decompression unit 202 reads the image data of the composite image from the storage unit 201, and performs the compression process up to the quantization process. The wavelet transform coefficients of all components obtained by this compression processing are written in the storage unit 201.

  In step S43, the processing unit 200 generates the wavelet transform coefficients of the images Q1, Q2, and Q3 corresponding to the images P1, P2, and P3 before synthesis by performing the same processing as in steps S23 to S34 in FIG. In step S44, in accordance with an instruction from the processing unit 200, the compression / decompression unit 202 sequentially performs inverse wavelet transform and inverse color space transform on the wavelet transform coefficients of the images Q1, Q2, and Q3, thereby Image data is generated on the storage unit 201. This image data is transferred to the auxiliary storage unit 203 under the control of the processing unit 200 in step S45.

  As can be understood from the above description, in this separated reproduction processing example 2, the compression / decompression unit 202 functions as a means for obtaining the wavelet transform coefficient (frequency transform coefficient in a broad sense) of the composite image from the image data of the composite image. In addition, it acts as means for generating image data by performing inverse wavelet transform (inverse frequency transform in a broad sense) on the wavelet transform coefficients of each reproduced image. Further, the processing unit 200 uses the composite image distance information from the composite image wavelet transform coefficients (frequency transform coefficients in a broad sense), and uses a plurality of image wavelet transform coefficients corresponding to the plurality of images before composition ( In a broad sense, it acts as a means for generating a frequency conversion coefficient). It is obvious that such means can be realized by one or more programs on a computer. Such a program and various information recording (storage) media readable by a computer on which the program is recorded are also included in the present invention.

  Note that the image data of the images Q1, Q2, and Q3 generated from the composite image can be transferred to the image display device, for example, for display via the data input / output unit 204. In step S44, the compression / decompression unit 202 performs a compression process after quantization on the wavelet transform coefficients of the images Q1, Q2, and Q3 to generate a codestream of each image, and stores the codestream in the auxiliary storage unit 203. It is possible to store the data or to output the data through the data input / output unit 204.

The description related to the image processing apparatus according to the present invention is also the description of the image processing method according to the present invention. That is, the flowchart of FIG. 5 or FIG. 7 also shows an example of the processing procedure of the image processing of the present invention. Such a processing procedure can be executed by a program using a computer. Such a program and various information recording (storage) media on which the program is recorded are also included in the present invention.

[Imaging device]
In order to apply the above-described image composition processing, it is necessary to shoot a plurality of images at the same angle of view and at different in-focus distances . Next, an example of an image pickup apparatus having a function of automatically performing such shooting will be described with reference to FIGS.

  FIG. 1 is a block diagram of the imaging apparatus. In FIG. 1, a general imaging optical system 300 includes an optical lens, a diaphragm mechanism, a shutter mechanism, and the like. Reference numeral 301 denotes a CCD-type or MOS-type imager, which color-separates an optical image formed by the imaging optical system 300 and converts it into an electric signal corresponding to the amount of light. A CDS / A / D converter 302 samples the output signal of the imager 301 and converts it into a digital signal, and includes a correlated double sampling (CDS) circuit and an analog / digital conversion circuit (A / D conversion circuit). A signal processing unit 303 includes a high-speed digital signal processor controlled by a program (microcode), for example. The signal processing unit 303 performs signal processing such as gamma correction processing, white balance adjustment processing, and enhancement processing for edge enhancement on the image data input from the CDS / A / D conversion unit 302, as well as the imager 301, CDS. Controls the A / D conversion unit 302 and the display unit 304, and detects information for auto focus control, automatic exposure control, white balance adjustment, and the like. The display unit 304 is a liquid crystal display device, for example, and is used for displaying images such as monitoring images (through images) and captured images, and displaying other information.

  The imaging optical system 300, the imager 301, the CDS / A / D conversion unit 302, and the signal processing unit 303 described above constitute an imaging unit that captures a subject and acquires image data of the image as a whole.

  Reference numeral 307 denotes a JPEG2000 compliant compression / decompression unit, which is used for compression processing of image data of a photographed image and decompression processing of a compressed code stream. Reference numeral 308 denotes an exchangeable recording medium (for example, various memory cards) on which the code stream compressed by the compression / decompression unit 307 is recorded. Reference numeral 309 denotes an external interface unit, through which information can be exchanged with an external personal computer or the like through a wired or wireless network or other transmission path.

  Reference numeral 306 denotes a control unit, which includes, for example, a microcomputer. The control unit 306 responds to user operation information input from the operation unit 310, information provided from the signal processing unit 303, and the like, and includes a shutter mechanism, an aperture mechanism, a focus mechanism, and a signal processing unit of the imaging optical system 300. 303, the compression / decompression unit 307, the recording medium 308, and the like are controlled. A memory 305 is used as a temporary storage area for image data and the like, and is also used as a working storage area for the signal processing unit 303, the control unit 306, and the compression / decompression unit 307. The operation unit 310 includes a release button for operating the imaging apparatus, other operation buttons (switches), and the like.

  Since the basic operation of this image pickup apparatus is the same as that of a conventional digital camera or the like, the description thereof will be omitted, and the operation related to the specific function of this image pickup apparatus will be described with reference to the flowchart shown in FIG.

  When the control unit 306 detects that the release button of the operation unit 310 is pressed (step S51, Yes), a plurality of different focusing distances D1, including the focusing distance, are determined from the focusing distance of the imaging optical system 300 at that time. D2,..., DN are set (step S52). For example, each in-focus distance has a preset distance difference around the in-focus distance when the release button is pressed (the in-focus distance set by the autofocus function or the manually-set in-focus distance). Set.

  Next, the control unit 306 sets the counter i to 0 (step S53), and operates the focus mechanism so as to match the focusing distance of the imaging optical system 300 with the focusing distance Di, thereby controlling one image. (Step S54). Then, the image data of the photographed image is compressed by the compression / decompression unit 307, and the code stream is written to the recording medium 308 together with the information on the focusing distance Di (step S55).

  Next, the control unit 306 increments the counter i (step S56), and performs a comparison determination between the incremented counter i value and the specified value N (step S57). If the value of the counter i is equal to or less than N (No at Step S57), the control unit 306 performs the control after Step S54, causes the second image to be taken at another in-focus distance Di, and the code stream of the image Are recorded on the recording medium 308.

  Under the same control, when shooting at all the in-focus distances set in step S52 is completed, the value of the counter i exceeds the specified value N (step S57, Yes), and thus a series of shooting operations is ended. It should be noted that the angle of view is the same for shooting at any focusing distance.

  Note that image data of a photographed image can be recorded on the recording medium 308 without being compressed, or can be transferred to an external personal computer or the like via the external interface unit 309.

Further, the function of the image processing apparatus of the present invention described with reference to FIGS. 1 to 4 can be incorporated in this imaging apparatus. In this case, for example, the function of the processing unit 200 in FIG. 1 can be realized by a program using a microcomputer configuring the control unit 306 and / or a digital signal processor configuring the signal processing unit 303. According to such a configuration, it is possible to execute a composition process for a plurality of captured images in the imaging apparatus .

It is a block diagram for demonstrating the image processing apparatus which concerns on this invention. It is a flowchart which shows an example of a synthetic | combination process. It is a schematic diagram for demonstrating a synthetic | combination process. It is a flowchart which shows another example of a synthetic | combination process. It is a flowchart which shows an example of a separation reproduction process. It is a schematic diagram for description of the separation reproduction processing. It is a flowchart which shows another example of a separation reproduction process. It is a block diagram which shows an example of the imaging device which has a function which changes and changes an in-focus distance with the same angle of view. It is a flowchart for demonstrating operation | movement of an imaging device. It is a block diagram for demonstrating the compression / decompression process of JPEG2000. It is a figure which shows the subband division | segmentation by the wavelet transformation of the number of decomposition levels of 3.

DESCRIPTION OF SYMBOLS 200 Processing part 201 Storage part 202 Compression / decompression part 203 Auxiliary storage part 204 Data input / output part P1, P2, P3 Image before composition Q1, Q2, Q3 Reproduction image 300 Imaging optical system 301 Imager 302 CDS / A / D conversion part 303 Signal Processing Unit 306 Control Unit 307 Compression / Expansion Unit 308 Recording Medium 310 Operation Unit

Claims (9)

An image processing device that generates a plurality of reproduced images with different in-focus distances from a composite image synthesized from a plurality of images shot at different in-focus distances at the same angle of view,
Means for obtaining a frequency conversion coefficient of the composite image;
With reference to the distance information assigned to each small area of the composite image, (a) the frequency conversion coefficient of the small area of the composite image to which the distance information matching the in-focus distance of the playback image is assigned is used as it is. (B) The frequency conversion coefficient of the small area of the composite image, to which the disagreement distance information is added to the in-focus distance of the reproduced image, is used as the frequency conversion coefficient of the corresponding small area of The frequency conversion coefficient quantized with the quantization parameter corresponding to the distance difference between the playback image and the focus distance of the playback image is used as the frequency conversion coefficient of the corresponding small area of the playback image, thereby converting the frequency of multiple playback images Means for generating coefficients;
An image processing apparatus comprising: 2. The image processing apparatus according to claim 1, further comprising means for performing inverse frequency conversion of a frequency conversion coefficient of a reproduced image to generate image data of the reproduced image. The image processing apparatus according to claim 1, wherein the frequency transform coefficient is a wavelet transform coefficient. An image processing method for generating a plurality of reproduced images having different in-focus distances from a composite image synthesized from a plurality of images taken at different in-focus distances at the same angle of view,
Processing to obtain a frequency conversion coefficient of the composite image;
With reference to the distance information assigned to each small area of the composite image, (a) the frequency conversion coefficient of the small area of the composite image to which the distance information matching the in-focus distance of the playback image is assigned is used as it is. (B) The frequency conversion coefficient of the small area of the composite image, to which the disagreement distance information is added to the in-focus distance of the reproduced image, is used as the frequency conversion coefficient of the corresponding small area of The frequency conversion coefficient quantized with the quantization parameter corresponding to the distance difference between the playback image and the focus distance of the playback image is used as the frequency conversion coefficient of the corresponding small area of the playback image, thereby converting the frequency of multiple playback images Processing to generate coefficients;
An image processing method comprising: 5. The image processing method according to claim 4, further comprising a process of performing inverse frequency conversion of a frequency conversion coefficient of a reproduced image to generate image data of the reproduced image. The image processing method according to claim 4, wherein the frequency transform coefficient is a wavelet transform coefficient. A program that causes a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 3. A program for causing a computer to execute each process of the image processing method according to any one of claims 4 to 6. A computer-readable information recording medium on which the program according to claim 7 or 8 is recorded.

Images