JP6278728B2 - Image processing apparatus, image generation apparatus, control method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image generation apparatus, a control method, and a program, and more particularly to an image processing technique for recording acquired light space information.

  Some imaging devices such as digital cameras record an intensity distribution of a reflected light beam from a subject incident on an imaging surface together with information on the incident direction. Information on the intensity distribution of the light beam including the incident direction recorded by such an imaging device is called light field information. It is also known that the light space information has the intensity distribution of the reflected light beam for each incident direction, and from this information, the intensity distribution of the reflected light beam on the surface of an arbitrary focal length of the imaging device can be reproduced. (Non-Patent Document 1). Therefore, from the light space information, an image focused on an arbitrary subject distance (reconstructed image) can be generated (reconstructed) after the light space information is recorded.

  As a configuration capable of recording light space information, a microlens array is arranged in front of a solid-state imaging device, and a plurality of pixels are associated with one microlens to separate light fluxes that have passed through the pupil region. An imaging device capable of recording in a pixel has been proposed.

Ren. Ng, 7 others, “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

  An imaging apparatus having a microlens array records light space information obtained by separating a light beam imaged on each microlens into the tens of pixels by a configuration in which tens of pixels are associated with each microlens. The number of pixels of the reconstructed image that can be generated from the light space information recorded in this way is at most the number of microlenses. In other words, in order to obtain an image with the same number of pixels, it is necessary to record light space information having an information amount several tens of times larger than data recorded by a general imaging device having no microlens array. become. For this reason, the processing of the light space information generated by the imaging device having the microlens array may cause problems such as an increase in circuit scale related to image processing and recording, and an operation delay during reading.

  The present invention has been made in view of the above-described problems of the prior art, and is an image processing apparatus, an image generation apparatus, and a control method capable of generating reconfigurable ray space information with a reduced amount of information. And to provide a program.

  In order to solve this problem, for example, an image processing apparatus of the present invention has the following configuration. That is, each pixel is incident on the same region included in the light space information and the acquisition means for acquiring light space information that records the light beam obtained by dividing the light beam from the subject into pupils, which is incident on the region corresponding to the pixel. A pixel means for calculating a variance of pixel values for a pixel group in which a luminous flux obtained by splitting the luminous flux into pupils is recorded, and based on the calculated variance, a pixel group corresponding to the variance is equal to or smaller than the number of pixels of the pixel group And generating means for generating ray space information in which the amount of information of ray space information is compressed.

  According to the present invention, it is possible to generate reconfigurable ray space information with a reduced amount of information.

1 is a block diagram illustrating a functional configuration example of a digital camera as an example of an image processing apparatus according to the present embodiment; The figure which shows typically the example of the data format which reduced the information amount of the light space information which concerns on this embodiment The figure which shows typically the example of the microlens array in this embodiment, and the pixel arrange | positioned on an imaging surface FIG. 5 is a diagram schematically illustrating a pixel selection method (selection type) and a representative value selection method according to the first embodiment. The figure which shows typically the interpolation process of the pixel which concerns on Embodiment 1. FIG. The figure which shows an example of the relationship between the dispersion value of the pixel value corresponding to a microlens, and a subject distance, and selection type 6 is a flowchart illustrating a selection type determination method according to the first embodiment. The figure which shows typically the selection classification and representative value selection method which concern on Embodiment 2. The figure which shows typically the interpolation process of the pixel value which concerns on Embodiment 2. FIG.

(Embodiment 1)
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following, the present invention is applied to a digital camera capable of recording light space information (LF data) composed of pixel values having information on the intensity of the reflected light flux from the subject and the incident direction as an example of an image processing apparatus. An example will be described. However, the present invention is not limited to a digital camera that can record LF data, but can be applied to any device that can read recorded LF data and generate LF data with a reduced amount of information. These devices may include, for example, mobile phones, game machines, tablet terminals, personal computers, and the like.

(1 Digital camera configuration)
FIG. 1 is a block diagram illustrating a functional configuration example of the digital camera according to the present embodiment. One or more of the functional blocks shown in FIG. 1 may be realized by hardware such as an ASIC or a programmable logic array (PLA), or may be realized by a programmable processor such as a CPU or MPU executing software. May be. Further, it may be realized by a combination of software and hardware. Therefore, in the following description, even when different functional blocks are described as the operation subject, the same hardware can be realized as the subject. In addition, the arrow shown with the broken line in FIG. 1 represents a control signal.

  The reflected light beam from the subject is collected by the imaging optical system 129, passes through the microlens array 127, and is projected onto the imaging device 101 for photoelectric conversion. FIG. 3 schematically shows an example of the microlens array 127 and the pixels arranged on the imaging surface in the present embodiment. In the digital camera of the present embodiment, for example, microlenses associated with 4 × 4 pixels at a ratio of 1 are arranged in a grid pattern on the front surface of the image sensor. In the example of FIG. 3, 1024 micro lenses are arranged in the horizontal direction and 768 micro lenses in the vertical direction. A region 303 is a region 303 indicated by a frame line in more detail. In the region 303, nine microlenses are arranged in a lattice pattern, and the reflected light beam that has passed through one microlens 302 is a pixel region (microlens block) of the image sensor associated with the microlens. ) 301. The projected light flux is received by each of the 16 pixels AP arranged in the microlens block 301 and recorded as LF data. In the case of having the microlens array having such a configuration, the total number of pixels in the LF data recorded by one photographing is 125829112 obtained by multiplying the number of microlenses by 16 pixels.

  The image sensor 101 has, for example, a structure in which image sensors are stacked in layers, each pixel having a three-layer structure, and outputs LF data in an RGB format in which RGB color components are separated. The output analog LF data is subjected to amplification processing and A / D conversion processing by an AFE (Analog Front End) 103. The digitized LF data is stored in the memory 105.

  In the digital camera of this embodiment, in order to reduce the information amount of LF data, a process of thinning out pixels is performed based on the dispersion of the pixel group of one microlens block. Therefore, the CPU 125 causes the dispersion calculation unit 123 to perform calculation processing on the pixels of each microlens block of the LF data. Specifically, the LF data output from the AFE 103 is output and stored in the line memory 121 for every four lines that are the width of the microlens block. The LF data stored in the line memory 121 is divided for each microlens block and output to the dispersion calculation unit 123. The variance calculation unit 123 calculates a variance value of pixel values for each microlens block with respect to the input 16-pixel data of the microlens block. Note that specific processing of the distributed calculation unit 123 will be described later. The CPU 125 determines a selection method (selection type) for selecting a pixel to be processed from the pixel group of the microlens block based on the dispersion value calculated by the dispersion calculation unit 123, and stores information on the selection type as a pixel. The data is output to the selection unit 107. A specific description of the selection type will be described later.

  The pixel selection unit 107 sequentially reads out the LF data stored in the memory 105 and performs a process of selecting a pixel to be processed according to the selection type information specified by the CPU 125. The pixel selection unit 107 thins out pixels by pixel selection processing, and sequentially outputs LF data (compressed LF data and simplified LF data) with a reduced amount of information to the recording unit 119. As will be described in detail later, the compressed LF data output from the pixel selection unit 107 is output as data with a reduced amount of information as shown in FIG. The recording unit 119 records the compressed LF data output from the pixel selection unit 107 in, for example, a nonvolatile memory.

  Further, the pixel selection unit 107 outputs the compressed LF data to the primary developing unit 111 in addition to the output of the compressed LF data to the recording unit 119. For example, the primary development unit 111 performs flaw correction, shading correction, color balance correction, and the like using preset conversion characteristics, and outputs the corrected compressed LF data to the pixel interpolation unit 113.

  The pixel interpolation unit 113 determines a selection type from the read compressed LF data, and performs interpolation of pixels in the microlens block based on the selection type. Then, the pixel interpolation unit 113 outputs the LF data decoded by interpolation to the reconstruction calculation unit 115. In the description of the present invention, performing interpolation processing of each microlens block on compressed LF data to obtain LF data having an information amount before compression is referred to as decoding.

  The reconstruction calculation unit 115 performs a process of reconstructing an image focused on an arbitrary subject distance using the decoded LF data, and outputs the reconstructed image data to the compression unit 117. The compression unit 117 compresses the amount of information using a predetermined compression method such as JPEG for still images and MPEG for moving images and outputs the compressed image data to the recording unit 119. The recording unit 119 records on a memory card or the like. The compressed image data is recorded on the medium.

  Note that the present invention may be applied to a monochrome imaging device or a Bayer array imaging device.

(2 LF data reduction processing)
The outline of the LF data reduction processing according to the present invention will be described in detail with reference to the drawings.

  The light beam that has passed through the pupil of the imaging optical system 129 and formed an image at the position of each microlens of the microlens array 127 is developed and received by the pixel group of the microlens block associated with the microlens, and the LF data Is output as At this time, since the light beam formed at the position of each microlens is a light beam that has passed through the entire area of the pupil, the output from the plurality of pixels that have received the developed light beam is the corresponding pupil area (part) The output corresponds to the light beam that has passed through. In other words, pupil division is realized by the microlens, and the pixels of the LF data indicate the amounts of light beams having different incident directions that have passed through different divided pupils.

  Based on such characteristics of LF data, the dispersion of pixel values in the microlens block corresponding to each microlens changes depending on the relationship between the conditions of the imaging optical system at the time of imaging and the subject. First, of the subjects included in the range of subject distance that can be included in the depth of field (reconstruction effective range) when a reconstructed image is generated, the depth of field is determined under the conditions of the imaging optical system at the time of imaging. The subject included in is described. For a subject included in the depth of field, that is, a subject that is in focus due to the conditions of the imaging optical system at the time of imaging, a light beam that is reflected and incident in various directions from any one point of the subject Converges to one point on the imaging surface, that is, one microlens. Therefore, the light beams developed by the microlenses and separated and received by the corresponding microlens blocks are different light fluxes in the path reflected from one point of the same subject, so even if they are separated, substantially the same pixel value is obtained. Is output. That is, the dispersion of the pixel values of the microlens block where the reflected light beam from the subject that is in focus at the time of imaging converges becomes extremely small.

  Next, among the subjects included in the reconstruction effective range, for a subject that is not included in the depth of field due to imaging conditions, the light beam reflected and incident in various directions from one point of the subject is focused. The image formation position is deviated from the subject in the state. That is, a light beam incident through various paths from the one point is diffused to neighboring microlenses and forms an image on microlenses that are not necessarily the same. In addition, a light beam reflected from one point of another subject can be imaged on the microlens. In other words, in a microlens block corresponding to a microlens on which an image of a subject that is not included in the depth of field is formed, each pixel indicates a pixel value corresponding to a light flux from a different subject, and thus dispersion increases. .

  For a subject outside the reconstruction effective range, the light beam reflected and incident in various directions from one point of the subject is already blurred when it reaches the microlens, and the image formation position is also re-established. Compared to the luminous flux from the subject within the effective range of construction, it is greatly deviated. That is, statistically, since the light beams from various subjects are incident as a mixed light beam, the dispersion becomes smaller.

  Here, when the dispersion of pixel values is small in the microlens block, that is, when the pixel values are similar, even if it is assumed that some pixel groups have the same pixel value, the generated reconstructed image The impact on image quality is considered low. That is, if all pixel groups having similar pixel values are considered to have the same pixel value, even if only one pixel value is recorded for the pixel group, at least the same pixel value is obtained by interpolation after recording. Can be restored. Therefore, in the digital camera according to the present invention, according to the dispersion of each microlens block included in the obtained LF data, recording is performed by representing all pixels with pixel values equal to or less than the number of pixels in the block. By doing so, it is possible to reduce the amount of LF data to be recorded in association with the block. In the present embodiment, a part of the pixels included in the microlens block is selected and recorded as a value representative of the pixel value of the pixel, thereby reducing the amount of LF data. That is, if the pixel values are selectively recorded according to the dispersion of the pixel values in each microlens block, compressed LF data capable of generating LF data equivalent to that at the time of shooting can be generated by interpolation. it can.

  In the present embodiment, as described above, the dispersion value of the pixel value in the microlens block is calculated, and the pixels in the pixel group to be discarded when generating the compressed LF data are determined. More specifically, the smaller the variance value of the pixel values in the microlens block, the smaller the pixel selected (remaining pixel after thinning), and the larger the variance value, the closer to the number of pixels in the block. Select as a pixel. The dispersion value of the pixel value in the microlens block is calculated using the following formula 1, for example. The variance calculation unit 123 reads the pixel value of 4 × 4 pixels from the line memory 121 and calculates the variance value of each microlens block.

here,
Indicates.

If the dispersion value S ² in the pixel group of the micro lens block is small, the pixel value of the pixel group of the microphone lens block may be determined to be high similarity. On the other hand, when the variance value S ² is large, the pixel value of the pixel group of the microphone lens block can be determined that similarity is low.

  FIG. 6 is a diagram illustrating the relationship between the dispersion value of the pixel value in the microlens block (hereinafter simply referred to as the dispersion value of the microlens block) and the subject distance for a subject in a predetermined state. The subject distance indicates the distance between the digital camera and each subject at the time of shooting. The subject distance related to the microlens block is defined as follows. When a reconstructed image that focuses on an arbitrary subject distance is generated from LF data obtained by shooting, a microlens corresponding to the pixel position of the focused subject in the image can be specified. At this time, the subject distance related to the microlens block corresponding to the specified microlens in the LF data obtained by photographing is defined as the subject distance used to generate the reconstructed image. That is, the subject distance associated with one microlens block indicates the subject distance of the subject that is in focus at the pixel position of the reconstructed image corresponding to the block.

The horizontal axis in FIG. 6 shows the object distance, and the vertical axis represents the variance value S ² of the micro lens block. In the example of FIG. 6, at the time of shooting, the subject at a subject distance of 2.5 m is focused under the conditions of the imaging optical system, and the subject at a subject distance around it is included in the depth of field. . As described above, since the subject included in the depth of field at the time of shooting converges on the same microlens on the imaging surface, the dispersion value in the corresponding microlens block is the dispersion of other subject distances. Small compared to the value. On the other hand, the dispersion value increases as the subject distance moves away from the periphery of the center of the depth of field at the time of imaging, and the subject distance peaks at about 1.5 m and 4 m. In addition, even if the subject is closer to the reconstruction effective range or the subject is farther than the reconstruction effective range, the dispersion value is small when the light flux of a similar subject passes through a specific microlens. . Note that the variance values shown in FIG. 6 are merely examples, and deviations and fluctuations may occur in the distribution of variance values depending on the composition and subject.

  Thus, depending on the focus state at the time of shooting and the arrangement of the subject within the imaging range, each microlens block can take various dispersion values. Therefore, in this embodiment, in order to reduce the amount of information while reducing the deterioration in image quality when generating a reconstructed image, the method of selecting the pixels to be recorded for the microlens block is classified into three selection types according to the dispersion value. Choose from. Specifically, the pixel selection unit 107 divides the size of the dispersion value into three value ranges, and determines which of the three selection types should be selected depending on which value range the calculated dispersion value of the microlens block falls into. judge. In this embodiment, the selection type “1 × 1” in which a pixel to be selected is selected in a value range determined to have a small variance value, and all pixels (16 pixels) are selected in a value range to be determined to have a large variance value. The type “4 × 4” is set, and the selection type “2 × 2” for selecting four pixels is set in the middle value range. For example, when the range of the dispersion value is set as shown on the vertical axis in FIG. 6, the dispersion value is small in the microlens block in which the corresponding subject is included in the depth of field, and therefore the 1 × 1 selection type is Selected for the block. This is because the difference in pixel value between one representative pixel (representative pixel) and a non-selected pixel is small, so that pixel values equivalent to all pixel values in the block can be interpolated from one pixel value. by. On the other hand, in the case of a microlens block that falls within a value range in which the corresponding variance value is determined to be large in FIG. 6, it is not possible to select a representative pixel that can interpolate all the pixels. select.

  Next, a pixel selection method corresponding to the selection type will be described. In the case of a microlens block having a small dispersion value, that is, when the selection type is “1 × 1”, the pixel selection unit 107 selects one of the 16 pixels as a representative pixel (for example, a pixel F in FIG. 4A). In FIG. 4A, it is selected as 311). In the case of a microlens block having a medium dispersion value (that is, the selection type is “2 × 2”), the pixel selection unit 107 selects four pixels out of 16 pixels (for example, the pixels F, G, and J in FIG. , K) are selected as representative pixels. Each representative pixel, for example, the pixel F, represents four pixels including the pixel and adjacent pixels A, B, and E. The representative pixel in this case is the representative pixel 312 in FIG. In the case of a microlens block having a large dispersion value (that is, the selection type is “4 × 4”), the pixel selection unit 107 selects all of the 16 pixels as representative pixels. In FIG. 4C, all the pixels A to P are selected as representative pixels. In selecting the representative pixel in FIG. 4, a method of preferentially selecting a pixel corresponding to the vicinity of the center of the microlens is employed. Thus, by selecting the pixel that receives the light beam passing through the center of the pupil, it is possible to reduce the influence of the insufficient light quantity due to the light vignetting on the peripheral pixels of the microlens block. In addition, a pixel having a pixel value close to the average value or the median value in the pixel group obtained when calculating the variance value may be preferentially selected. By selecting pixels close to the average value or the median value, it is possible to select pixels that reflect the distribution tendency of the pixel values in the microlens block.

  The obtained representative pixel is recorded in a new data format as compressed LF data. The compressed LF data is composed of data representing the separation between the representative pixel and each microlens block. When 16 pixels are recorded from the microlens block, an effect of reducing the information amount cannot be obtained. However, when a representative pixel having a number of pixels smaller than 16 is recorded, an effect of reducing the information amount is produced. Note that the data indicating the separation of the microlens block may have data indicating the coordinates of the microlens corresponding to the representative pixel, the coordinates of the representative pixel in the microlens block, and the determined selection type. FIG. 2 schematically shows an example of the data format of the compressed LF data. In FIG. 2, elements indicated by “Q”, for example, 3241, 3242, and 3243 represent delimiters of the microlens block, and other elements represent representative pixels. A pixel F indicated by 3211 and 3212 indicates that one pixel at the position F is selected and recorded from the corresponding microlens block, and the amount of information is compared with the case of recording 16 pixels. Can be seen to be reduced.

(3 LF data reduction processing operation)
An operation corresponding to the above-described LF data reduction process will be described. In the following, in order to simplify the description, it is assumed that processing is performed on data of one of the RGB channels of LF data.

  The dispersion calculation unit 123 sequentially acquires the LF data stored in the line memory 121 in units of microlens blocks, calculates a dispersion value, and outputs the dispersion value to the CPU 125. The CPU 125 performs selection type determination processing based on the acquired variance value. The flowchart of FIG. 7 shows the flow of processing of the dispersion calculation unit 123 and the CPU 125 for each microlens block. The flowchart shown in FIG. 7 is started when the dispersion calculation unit 123 acquires the LF data stored in the line memory 121 for each microlens block. The variance calculation unit 123 and the CPU 125 perform the process shown in FIG. 7 by the number of microlens blocks in the LF data.

  In S <b> 201, the variance calculation unit 123 acquires the pixel values of each of the 16 pixels of the acquired microlens block, calculates the variance value of the pixel values by the calculation shown in Equation 1 above, and outputs it to the CPU 125.

  In S <b> 203, the CPU 125 determines the selection type based on the variance value of the pixel values acquired from the variance calculation unit 123. The CPU 125 refers to a predetermined threshold table for determining each selection type according to the variance value as in the selection type shown in FIG. The CPU 125 compares the acquired variance value with a threshold value A for determining a small variance value and a threshold value B for determining a large variance value. If the acquired dispersion value is equal to or less than the threshold value A, the CPU 125 advances the process to S205. On the other hand, when the obtained variance value is larger than the threshold value A and equal to or smaller than the threshold value B, the CPU 125 advances the process to S207. Further, when the obtained variance value is larger than the threshold value B, the CPU 125 advances the process to S209.

  In S <b> 205, the CPU 125 stores a value indicating “1 × 1” in the identifier for specifying the selection type for the acquired microlens block. When a value is stored in the identifier for specifying the selection type, the process of the flowchart is finished.

  In S207, the CPU 125 stores a value indicating “2 × 2” in the identifier for specifying the selection type for the acquired microlens block, and ends the process of the flowchart. On the other hand, in S209, the CPU 125 stores a value indicating “4 × 4” in the identifier for specifying the selection type, and ends the process of the flowchart.

  Note that the memory capacity required for the calculation in the pixel selection unit 107 is reduced by using a memory for four lines, which is the minimum line width of the microlens block necessary for the distributed calculation. As a result, even when the digital camera has a limited memory capacity, a high-speed operation can be realized, and thus a memory band for generating a reconstructed image with heavy processing can be secured.

  When the selection type for the microlens block is set by the series of processes shown in FIG. 7, the CPU 125 outputs the set selection type to the pixel selection unit 107, and selects one or more representative pixels according to the selection type. Let it be done. The pixel selection unit 107 that has acquired the selection type from the CPU 125 reads out the LF data stored in the memory 105 and performs a process of selecting a representative pixel for each microlens block. The pixel selection unit 107 accesses a pixel in the microlens block determined in advance according to the selection type, and records a predetermined pixel. That is, as described above, when the acquired selection type indicates “1 × 1”, the pixel selection unit 107 accesses a predetermined pixel (for example, pixel F) determined in advance among the 16 pixels, and the pixel value thereof. Is output to the recording unit 119. Similarly, when the selection type indicates “2 × 2”, similarly, the pixel selection unit 107 accesses four predetermined pixels out of 16 pixels and outputs the pixel values. When the selection type indicates “4 × 4”, the pixel selection unit 107 accesses and outputs all 16 pixels.

  In addition, the pixel selection unit 107 inserts a bit indicating the selection type corresponding to the selected pixel, for example, into data indicating the delimiter of the microlens block, forms data indicating the delimiter, and outputs the data together with the pixel value. The pixel selection unit 107 ends the process when outputting the selected pixel value and the data indicating the delimiter of the microlens block to the recording unit 119 for all the microlens blocks.

(4 Interpolation processing and operation for compressed LF data)
Next, processing for decoding LF data used for generating a reconstructed image using compressed LF data whose information amount has been reduced by such reduction processing will be described.

  FIG. 5 is a diagram schematically illustrating pixel interpolation processing according to the present embodiment. In the microlens block having a small dispersion value, one pixel out of 16 pixels is recorded as a representative pixel, and therefore the pixel of the microlens block is interpolated using the representative pixel. For example, in FIG. 5A, when the pixel 311 is selected, the pixel interpolation unit 113 interpolates the other 15 pixels using the pixel value of the representative pixel F. Further, in the case of a microlens block having a medium dispersion value, four pixels out of 16 pixels are selected as representative pixels, and therefore the pixels of the microlens block are interpolated using the selected four pixels. For example, when four pixels shown as the representative pixel 312 in FIG. 5B are selected, the pixel interpolating unit 113 uses the pixel values of the representative pixels F, G, J, and K, and pixels near each pixel. Is interpolated. For the microlens block having a large pixel dispersion value, since all the pixels of the 16 pixels are selected as the representative pixels, the pixel interpolation unit 113 uses all the pixels as they are. In FIG. 5C, all the pixels are used as they are.

  Next, the operation of each functional block will be described with respect to the operation of interpolation processing for the above-described microlens block. The compressed LF data output from the pixel selection unit 107 is subjected to a predetermined correction process by the primary development unit 111 and is output to the pixel interpolation unit 113. The pixel interpolation unit 113 acquires the compressed LF data corrected by the primary developing unit 111, performs an interpolation process on the microlens block for generating a reconstructed image, and obtains LF data. The pixel interpolation unit 113 scans the acquired compressed LF data, specifies data indicating the delimiter of the microlens block and the representative pixel, and acquires the pixel value of the pixel. First, for example, the pixel interpolation unit 113 reads a bit string indicating a selection type from data indicating a delimiter of the specified microlens block, determines the selection type, and specifies the number of pixel values to be read according to the selection type. Thereafter, the pixel interpolation unit 113 reads pixel values by the number of specified pixels, and performs interpolation processing on the microlens block according to the selection type. Note that the pixel interpolation unit 113 may count the number of representative pixels when scanning LF data, and determine the selection type from the number of representative pixels. When the selection type indicates “1 × 1”, the pixel interpolation unit 113 interpolates all the pixels of the microlens block using the pixel value of one representative pixel acquired. Similarly, when the selection type indicates “2 × 2”, for example, as illustrated in FIG. 5B, the pixel interpolation unit 113 interpolates pixels at predetermined positions for each of the four pixels. On the other hand, when the selection type indicates “4 × 4”, the pixel interpolation unit 113 uses the read pixel value as it is as the pixel value of the microlens block. The pixel interpolation unit 113 outputs the pixel values of all the interpolated microlens blocks to the reconstruction calculation unit 115 as LF data.

  The reconstruction calculation unit 115 uses the LF data acquired from the pixel interpolation unit 113 to generate a reconstructed image that focuses on the subject distance determined by the setting of the user or the like.

  As described above, the information amount of the LF data can be reduced by selecting and recording the pixels of the microlens block according to the dispersion value of the pixel values. Further, by using the dispersion value of the pixel value, it is possible to generate compressed LF data in which the degradation of the image quality of the reconstructed image is reduced.

(Embodiment 2)
Next, a second embodiment of the present invention will be described. The digital camera of the present embodiment may have the same configuration as that of the first embodiment, and is common except for operations related to pixel value selection and interpolation processing. For this reason, the overlapping description will be omitted, and differences will be mainly described.

(1 LF data deletion process and operation)
In the first embodiment, the pixels of each microlens block are selected and the pixel value of the pixel is recorded as the pixel value of the representative pixel, whereas in the second embodiment, the sum of the predetermined pixel values in the microlens block is calculated. It records as a pixel value of a representative pixel. In the following description, the pixel value of the representative pixel is referred to as a representative value. An area in the microlens block indicating the pixels selected for obtaining the sum is referred to as a divided block. Then, each pixel in the microlens block is interpolated by a value (average value of summed pixel values) obtained by dividing the sum total as a representative value by the number of pixels used for the sum.

  In the case of a microlens block having a small dispersion value, a pixel value having 16 pixels as one divided block and 16 pixels as a representative value is determined. For example, in FIG. 8A, the total value T1 is a representative value of the corresponding microlens block. The total value is calculated by adding the value of 16 pixels in the following formula 2.

here,
Indicates.

  In the case of a microlens block having a medium dispersion value, 16 pixels are divided into four divided blocks, the sum is calculated for each divided block, and the representative value of each divided block is determined. For example, in FIG. 8B, the total values T2, T3, T4, and T5 are determined as representative values. The sum T2 is the sum of the pixel values of the pixels A, B, E, and F, the sum T3 is the sum of the pixels C, D, G, and H, and the sum T4 is the sum of the pixel values of the pixels I, J, M, and N. The total value T5 is calculated by the total sum of the pixel values of the pixels K, L, O, and P. In the case of a microlens block having a large dispersion value, all 16 pixels are determined as representative values. In FIG. 8C, the pixels A to P are used as they are.

  Next, the operation of the pixel selection unit 107 will be described. When the pixel selection unit 107 reads out the LF data stored in the memory 105, the pixel selection unit 107 acquires the pixel values of 16 pixels in one microlens block according to the selection type output from the CPU 125. The pixel selection unit 107 selects a predetermined pixel according to the selection type, calculates the total value shown in FIG. 8, and outputs it to the recording unit 119 as a representative value. The pixel selection unit 107 ends the process when the sum value as a representative value and the data indicating the delimiter of the microlens block are output to the recording unit 119 for all the microlens blocks.

(2 Interpolation processing of compressed LF data and its operation)
The interpolation processing according to the second embodiment will be described with reference to FIG. In the interpolation processing of the present embodiment, the representative value of the microlens block determined by the pixel selection unit 107, that is, the total value T is substituted into Equation 3 below, and an average value for a predetermined pixel is calculated. Each corresponding pixel is interpolated using the calculated average value.

here,
Indicates.

  For a microlens block having a small variance value, a single sum value is determined from 16 pixels, and an average value R obtained by dividing the sum value by the number of pixels of 16 is interpolated as the pixel value of each pixel. For example, in FIG. 9A, the pixel value in the microlens block is interpolated using the average value R obtained from the sum. In the case of a microlens block having a medium dispersion value, four divided blocks are formed from 16 pixels, and an average value obtained from each of the four representative values is interpolated as a pixel value. In the example of FIG. 9B, the average value S obtained by substituting the total value T2 into Equation 3 interpolates four pixels in the corresponding divided block, and the average value U obtained from the total value T3 corresponds. Interpolate four pixels in a divided block. Similarly, the average value V obtained from the total value T4 and the average value W obtained from the total value T5 interpolate each pixel in the corresponding divided block. In the case of a microlens block having a large dispersion value, since the sum is not calculated, each pixel value of 16 pixels is used as it is as shown in FIG. 9C.

  Next, the operation of the pixel interpolation unit 113 will be described. The pixel interpolation unit 113 scans the acquired compressed LF data, specifies data and pixel values indicating the delimiters of the microlens blocks, and acquires the pixel values. Specifically, for example, the pixel interpolation unit 113 specifies data indicating a delimiter of the specified microlens block, reads a bit string indicating the selection type from the data, and determines the selection type. Then, the pixel interpolation unit 113 specifies the number of pixels to be read according to the selection type, and acquires pixel values for the specified number of pixels. Thereafter, the pixel interpolation unit 113 performs interpolation processing according to the selection type. When the selection type indicates “1 × 1”, the pixel interpolation unit 113 divides the acquired pixel value (total value) by the number of target pixels 16 to calculate an average value, and calculates the pixels in the microlens block. Interpolate. Similarly, when the selection type indicates “2 × 2”, the pixel interpolation unit 113 acquires four pixel values (total values), calculates an average value for each total value, and determines a predetermined division. Interpolate block pixels. On the other hand, when the selection type indicates “4 × 4”, the pixel interpolation unit 113 uses the read pixel value as the pixel value of the microlens block without interpolation. When the scanning of the acquired compressed LF data is completed, the pixel interpolation unit 113 completes the process.

  In this embodiment, since the average value obtained from the total value is used in the interpolation of the LF data, even if there is a pixel value affected by the jumping noise or the like in the pixels used for the total, in the decoded LF data, It is possible to reduce the influence of changes in pixel values due to noise. That is, in the digital camera of the present embodiment, in addition to the effect of reducing the amount of information described in the first embodiment, it is possible to reduce the influence of alteration on specific pixels such as jumping noise originally included in the LF data. .

  In the present embodiment, the representative value determined by the pixel selection unit 107 has been described as an example in which the total value of the predetermined pixel values is the representative value. However, the representative value may be an average value of the predetermined pixel values. The effects described above can be obtained. In addition, when the variance value is small or medium, a peculiar pixel that affects the calculation of the total value, for example, a pixel whose pixel value is larger than a predetermined deviation is excluded from the target of the total, and the total value is calculated. You may make it calculate. As a result, it is possible to calculate a representative value in which the fluctuation of the total value due to a specific pixel value having a specific pixel value is reduced.

  As described above, according to the present invention, reconfigurable ray space information with a reduced amount of information can be generated.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (13)

An acquisition means for acquiring light space information in which each pixel is incident on a region corresponding to the pixel and records a light beam obtained by dividing a light beam from a subject into pupils;
Calculating means for calculating a dispersion of pixel values for a pixel group in which a light beam obtained by pupil-dividing a light beam incident on the same region included in the light space information is recorded;
Generating means for generating ray space information in which the information amount of the ray space information is compressed by representing the pixel group corresponding to the variance with pixels equal to or less than the number of pixels of the pixel group based on the calculated variance. When,
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the generation unit reduces the number of pixels representing the pixel group corresponding to the variance as the calculated variance is smaller.   The generation unit is characterized by representing the pixel group by a predetermined number of pixels with respect to the calculated magnitude of dispersion among the pixels included in the pixel group corresponding to the dispersion. Item 3. The image processing apparatus according to Item 1 or 2.   The said generation means selects the said predetermined number of pixels from the pixels which recorded the light beam which passed through the pupil central part contained in the pixel group corresponding to the said dispersion | distribution. The image processing apparatus described.   The generating means represents the pixel group by a predetermined number of pixels having a value generated from pixel values of the pixels included in the pixel group corresponding to the variance, with respect to the calculated variance size. The image processing apparatus according to claim 1, wherein the image processing apparatus includes: The value generated from the pixel values of the pixels included in the pixel group corresponding to the dispersion is at least the sum of the pixel values of some of the pixel groups included in the pixel group,
The image processing apparatus according to claim 5, wherein the generation unit increases the number of pixels of the partial pixel group for calculating the sum as the calculated variance is smaller. An acquisition means for acquiring light space information in which each pixel is incident on a region corresponding to the pixel and records a light beam obtained by dividing a light beam from a subject into pupils;
Calculating means for calculating a dispersion of pixel values for a pixel group in which a light beam obtained by pupil-dividing a light beam incident on the same region included in the light space information is recorded;
Based on the calculated variance, generation means for thinning the pixel group corresponding to the variance to the number of pixels equal to or less than the number of pixels of the pixel group and generating ray space information in which the amount of information of the ray space information is compressed When,
An image processing apparatus comprising: An acquisition means for acquiring ray space information generated by the image processing apparatus according to claim 1 and having a compressed amount of information;
A specifying unit for specifying a pixel corresponding to a pixel group that records a light beam obtained by dividing a light beam incident on the same region from the light space information obtained by compressing the information amount acquired by the acquisition unit;
A light beam having an information amount before compression by interpolating pixels included in a pixel group in which a light beam obtained by dividing the light beam incident on the same region based on the pixel value of the pixel specified by the specifying unit is recorded. Decoding means for decoding the spatial information;
Reconstructing means for generating an image focused on a predetermined subject distance from the light space information having the information amount before compression decoded by the decoding means;
An image generation apparatus comprising: An acquisition step in which an acquisition unit acquires ray space information in which each pixel is incident on a region corresponding to the pixel and records a light beam obtained by dividing a light beam from a subject into pupils; and
A calculation step of calculating a variance of pixel values for a pixel group in which a calculation unit records a light beam obtained by pupil-dividing a light beam incident on the same region included in the light space information;
Based on the calculated variance, the generation means represents the pixel group corresponding to the variance with pixels equal to or less than the number of pixels of the pixel group, and the ray space information obtained by compressing the information amount of the ray space information is obtained. A generation process to generate;
A control method for an image processing apparatus, comprising: An acquisition step in which an acquisition unit acquires ray space information in which each pixel is incident on a region corresponding to the pixel and records a light beam obtained by dividing a light beam from a subject into pupils; and
A calculation step of calculating a variance of pixel values for a pixel group in which a calculation unit records a light beam obtained by pupil-dividing a light beam incident on the same region included in the light space information;
Based on the calculated variance, the generation means thins out the pixel group corresponding to the variance to the number of pixels equal to or less than the number of pixels of the pixel group, and obtains the ray space information obtained by compressing the information amount of the ray space information. A generation process to generate;
A control method for an image processing apparatus, comprising: A control method for an image generation apparatus that generates an image focused on a predetermined subject distance from light-space information whose amount of information is compressed, which is generated by the image processing apparatus according to claim 1. There,
An acquisition step of acquiring light space information in which the amount of information is compressed;
A specifying step of specifying a pixel corresponding to a pixel group that records a light beam obtained by dividing the light beam incident on the same region from the light space information in which the information amount acquired in the acquisition step is compressed; ,
Information before compression is performed by interpolating pixels included in a pixel group that records a light beam obtained by dividing the light beam incident on the same region based on the pixel value of the pixel specified in the specifying step. A decoding step of decoding ray space information having a quantity;
A reconstruction step for generating an image focused on a predetermined subject distance from the ray space information having the information amount before compression decoded in the decoding step;
A control method for an image generating apparatus, comprising:   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 7. The program for functioning a computer as each means of the image generation apparatus of Claim 8 .