JP2022122350A - Imaging apparatus and control method of imaging apparatus - Google Patents

Abstract

To provide an imaging apparatus for acquiring image data whose resolution is increased by pixel shift and a control method thereof which suppress an information amount of image data subjected to synthesis processing by reducing the number of image data to be acquired by the pixel shift.SOLUTION: An imaging apparatus comprises: an imaging lens; an imaging element in a lamination structure; a pixel shift mechanism; and a pixel shift synthesis processing unit. The pixel shift synthesis processing unit comprises: an image data input part which inputs two pieces of image data acquired by pixel shift; an image data arrangement part which arranges image data in each layer constituting each of the two pieces of image data by shifting the image data; a re-sampling part which re-generates the image data in each layer by re-sampling processing from the image data in each layer arranged by being shifted; a pixel mixing part which generates one piece of image data by pixel mixing processing by using the re-generated image data in each layer; and an image data output part which outputs one piece of image data generated by the pixel mixing processing.SELECTED DRAWING: Figure 1

Description

The present invention relates to an imaging apparatus that acquires image data using an imaging element and a control method thereof. More particularly, the present invention relates to an imaging apparatus that employs an imaging element having a laminated structure in which photoelectric conversion units are stacked and obtains image data with high resolution by pixel shifting, and a control method thereof.

2. Description of the Related Art Conventionally, imaging apparatuses that acquire image data using an imaging element having color filters have been widely used. In each pixel that constitutes an imaging device having a color filter, a red (R), green (G), or blue (B) color filter is arranged in front of a photodiode that is a photoelectric conversion unit. . Thereby, the pixel value of the wavelength component corresponding to the color filter arranged in each pixel can be acquired by the photodiode. Then, an array of pixel values obtained from each pixel is obtained as image data, and image data for recording is obtained by performing demosaic processing on this image data.

Also, there is known an imaging apparatus that obtains image data with high resolution by pixel shifting. In pixel shifting, all or part of an image sensor or a photographing lens is shifted on a plane perpendicular to the optical axis to obtain a plurality of image data. Then, it is possible to combine the obtained image data of a plurality of sheets and finally obtain the image data with the high resolution.

Patent Document 1 discloses an electronic still camera that obtains image data by synthesizing data of four images obtained by pixel shifting. Further, an electronic still camera is disclosed that can acquire image data of higher image quality by synthesizing eight pieces of image data acquired by pixel shifting.

Patent Document 2 discloses an imaging apparatus that obtains image data by synthesizing eight image data obtained by pixel shifting, and performs chromatic aberration correction before synthesizing the image data obtained by pixel shifting. An imaging device is disclosed that can suppress the capacity of the buffer memory by doing so.

Further, there is known an imaging device that is configured by stacking a plurality of photodiodes so that each pixel value corresponding to the wavelength component of subject light that is vertically color-separated can be directly obtained in each pixel. Hereinafter, such an image pickup device may be referred to as an image pickup device having a laminated structure.

Patent Document 3 discloses an imaging device with a stacked structure (vertical color filter sensor group array), and furthermore, in order to eliminate the complexity of the wiring structure of the imaging device with a stacked structure, the top layer is made full resolution, An imaging device is disclosed in which the lower layer has a lower resolution.

JP-A-6-225317 JP 2017-46186 A U.S. Pat. No. 7,339,216

However, the conventional techniques disclosed in the above patent documents have the following problems.

In an imaging device having color filters, each pixel can acquire only one pixel value corresponding to each color filter. For this reason, in order to acquire image data with high resolution by pixel shifting using an image pickup device having color filters that employ a general Bayer array pattern, eight image data are acquired by pixel shifting. Must. As the number of pieces of image data to be acquired in pixel shifting increases, the time required to acquire the image data and the time required to perform correction processing, synthesis processing, and the like on the image data become longer. In addition, the capacity of the buffer memory required for recording the image data of a plurality of sheets becomes large.

When less than 8 pieces of image data are acquired by pixel shifting using an image pickup device having color filters, and the high-resolution image data is synthesized, the arrangement of the color filters of the high-resolution image data is arranged in a normal manner. It cannot be made the same as the image data, and an interpolation process, a replacement process, or the like is required to compensate for this. Furthermore, in this case, it becomes difficult to obtain high-resolution image data that is finally obtained by the combining process.

On the other hand, an imaging device with a laminated structure can directly acquire a pixel value corresponding to a wavelength component of subject light in each pixel. Therefore, when image data with high resolution is acquired by pixel shifting using an imaging device with a laminated structure, the number of image data to be acquired by pixel shifting can be reduced.

However, since each layer of the image data acquired by the image sensor with the laminated structure has image data of the same amount of information for each wavelength component of the subject light, the image data acquired by the image sensor with the laminated structure amount of information becomes very large. If the information amount of the image data acquired by pixel shifting using the image pickup device with the laminated structure increases, the load for transferring the image data and the load for combining the image data will increase.

In order to solve the above problems, an imaging apparatus according to a first aspect of the present invention generates image data having a higher resolution than normal image data by synthesizing a plurality of pieces of image data acquired by pixel shifting. an image pickup device having a laminated structure in which a photographing lens for forming an image of subject light as a subject image on an imaging plane; a photoelectric conversion section for converting the subject image into an image signal; A pixel shift mechanism that shifts the position relative to the layered structure image pickup device on a plane perpendicular to the optical axis, and a pixel shift that generates a single image data by synthesizing multiple image data obtained by pixel shifting. A synthesis processing unit is provided, and the pixel shift synthesis processing unit includes an image data input unit for inputting two pieces of image data acquired by pixel shifting, and an image data input unit for inputting two pieces of image data, and shifting the image data of each layer constituting the two pieces of image data. A resampling unit that regenerates the image data of each layer by resampling processing from the image data of each layer that is staggered and arranged, and a pixel mixing processing using the regenerated image data of each layer. The image processing apparatus is characterized by comprising a pixel mixing section for generating image data for one image, and an image data output section for outputting the image data for one image generated by the pixel mixing process.

Further, in the image pickup apparatus according to the second invention, the pixel shift mechanism moves the image pickup device with the laminated structure in a direction perpendicular to the optical axis, thereby shifting the relative positions of the object image and the image pickup device with the laminated structure. is shifted on a plane perpendicular to the optical axis.

Further, in the image pickup apparatus according to the third invention, the pixel shift mechanism moves all or part of the photographing lens in a direction perpendicular to the optical axis, so that the subject image and the image sensor having the laminated structure are shifted. It is characterized by shifting the relative position on a plane perpendicular to the optical axis.

In the image pickup apparatus according to the fourth aspect, the pixel shift mechanism shifts the relative position of the subject image and the image pickup device with the laminated structure in the XY directions on a plane perpendicular to the optical axis. It is characterized by shifting by 1/2 of the pixel pitch.

Further, in the image pickup apparatus according to the fifth aspect, the pixel shift synthesis processing unit converts the image data of each layer constituting two pieces of image data obtained by pixel shift using the image pickup device having the layered structure into luminance. Divided into layers and color layers, re-sampling process regenerates image data with higher resolution than normal image data in luminance layer, and image data with lower resolution than normal image data in color layer. is regenerated.

Further, in the image pickup apparatus according to the sixth invention, the pixel shift synthesis processing unit includes image data of each layer constituting two pieces of image data obtained by pixel shifting using the image pickup device having the layered structure. The image data of the layer with the highest contrast is used as the luminance layer, and the image data of the other layers are used as the color layer.

Further, in the image pickup apparatus according to the seventh invention, the pixel shift synthesis processing unit includes image data of each layer constituting two pieces of image data acquired by pixel shift using the image pickup device having the layered structure. The image data of the layer with the highest signal value is used as the luminance layer, and the image data of the other layers are used as the color layer.

A control method for an imaging apparatus according to an eighth aspect of the present invention is an imaging apparatus that generates image data having a higher resolution than normal image data by synthesis processing using a plurality of pieces of image data acquired by pixel shifting. The control method of , wherein an image data acquisition step of acquiring two image data by pixel shifting using an image pickup device with a laminated structure, an image data input step of inputting the acquired two image data, and an input an image data arrangement step of shifting and arranging the image data of each layer constituting the two pieces of image data obtained respectively; and regenerating the image data of each layer using the image data of each layer shifted and arranged by resampling processing A resampling processing step, a pixel mixing processing step of generating one image data using the image data of each layer regenerated by the pixel mixing processing, and an image data output step of outputting the generated one image data. characterized by having

According to the present invention, in an image capturing apparatus and a control method for an image capturing apparatus, image data having a higher resolution than normal image data is generated by synthesis processing using a plurality of image data obtained by pixel shifting. By using an imaging device having a laminated structure in which photoelectric conversion units that convert an image into an image signal are laminated, the number of pieces of image data to be acquired by pixel shifting can be reduced.

In addition, the image data of each layer constituting two pieces of image data obtained by pixel shifting using an image pickup device with a laminated structure is divided into a luminance layer and a color layer, and a normal image is obtained in the luminance layer by resampling processing. By regenerating image data with a higher resolution than the data, and regenerating image data with a lower resolution than the normal image data in the color layer, a multi-layered image sensor is used to shift the pixels. Even with an imaging device that acquires image data and a control method for the imaging device, it is possible to suppress the amount of information in the image data to be synthesized, and to suppress an increase in the burden of image data transfer and image processing. .

1 is a system block diagram showing an imaging device of this embodiment; FIG. It is a schematic diagram which shows the structure of the depth direction of the imaging device of a three-layer structure. It is a schematic diagram which shows the arrangement|positioning method of the image data of 2 sheets acquired by pixel shifting. 4 is a schematic diagram showing how image data of each layer is shifted based on a pixel pitch P and arranged. FIG. It is a schematic diagram explaining the method of the re-sampling process of a luminance layer. FIG. 4 is a schematic diagram illustrating a method of resampling processing of color layers; 4 is a flow chart showing each step of the photographing operation of the imaging apparatus of the present embodiment; 4 is a flow chart showing each step of image data acquisition by pixel shifting. 5 is a flowchart showing steps of pixel shift synthesis processing; 4 is a flowchart showing each step of pixel mixing processing;

Embodiments of an image pickup apparatus and an image pickup apparatus control method according to the present invention will be described below in detail with reference to the drawings. In the description of the present embodiment, constituent elements necessary for describing the technical features of the present invention will be described.

(Fig. 1: Components of imaging device)
The constituent elements of the imaging apparatus of this embodiment will be described with reference to FIG. FIG. 1 is a system block diagram showing the imaging apparatus of this embodiment. In FIG. 1, 100 is an imaging device, 110 is an imaging lens, 120 is an imaging device, 130 is a shift mechanism, 140 is a pixel shift synthesis processor, 150 is a bus, 160 is a DRAM, and 170 is an external memory.

(Fig. 1: Components of imaging device)
The photographing lens 110 is an optical system that forms an image of subject light on an imaging plane as a subject image. The imaging device 120 converts the subject image into an image signal (analog signal) by using photodiodes, which are photoelectric conversion units, arranged two-dimensionally on the imaging plane. The image pickup device 100 of this embodiment employs an image pickup device having a laminated structure as the image pickup device 120 . The image sensor having a laminated structure will be described later.

A shift mechanism 130 moves the imaging device 120 in a direction perpendicular to the optical axis. In the imaging apparatus 100 of this embodiment, the shift mechanism 130 functions as a pixel shift mechanism that shifts the relative position between the subject image and the imaging element 120 on a plane perpendicular to the optical axis. A pixel shift mechanism will be described later.

An array of image signals acquired by the imaging device 120 is converted into digital signals by an A/D converter (not shown) or the like, and acquired as image data. The image data is transferred to DRAM 160 via bus 150 and held temporarily.

In the image pickup apparatus 100 of this embodiment, the image pickup device 120 moved in the direction perpendicular to the optical axis by the shift mechanism 130, which is a pixel shift mechanism, acquires image data at each moved position. A plurality of pieces of image data thus acquired are temporarily held in the DRAM 160 .

A plurality of image data temporarily held in the DRAM 160 are transferred to the pixel shift synthesis processing section 140 via the bus 150 . The pixel shift synthesizing processing unit 140 uses a plurality of image data to generate one image data whose resolution is finally increased by synthesizing processing. The pixel shift synthesis processing unit 140 will be described later.

The final high resolution image data is transferred to the external memory 170 via the bus 150 and stored.

(Fig. 1: Image sensor with laminated structure)
Next, an image sensor having a laminated structure will be described. An imaging device having a laminated structure refers to an imaging device configured by laminating photodiodes, which are photoelectric conversion units. In particular, this embodiment employs a three-layer structure imaging device in which photodiodes, which are photoelectric conversion units, are stacked in three layers in the depth direction.

(Figure 2: 3-layer structure imaging device)
FIG. 2 is a schematic diagram showing a cross-sectional structure of an imaging device having a three-layer structure. As shown in FIG. 2A, each pixel that constitutes the three-layered imaging device is configured by stacking blue (B), green (G), and red (R) photodiodes in order in the depth direction. there is Accordingly, pixel values corresponding to each color can be directly obtained for each pixel.

Also, as shown in FIG. 2B, the array of pixel values corresponding to each color obtained in each pixel is obtained as image data of each layer corresponding to each color. In this embodiment, a set of image data of each layer obtained by using an imaging device with a three-layer structure is referred to as one piece of image data.

A semiconductor such as silicon used as a material for a photodiode has a property that the absorption rate depending on the penetration depth differs for each wavelength component of incident light. Due to the properties of semiconductors, of the wavelength components of incident light, light components with shorter wavelengths are absorbed in shallower portions of the semiconductor, and light components with longer wavelengths are absorbed in deeper portions of the semiconductor.

An imaging device with a laminated structure utilizes this property of a semiconductor, and by stacking photodiodes made of silicon or the like, it is possible to vertically color-separate incident light for each wavelength component. As a result, in the image pickup device with the stacked structure, it is possible to directly obtain the pixel value corresponding to each color in each pixel. Therefore, compared to an image sensor with a color filter, an image sensor with a laminated structure does not need to place a color filter for color separation of incident light above the photodiode, and false colors occur when acquiring image data. No demosaicing is required as a factor.

The imaging apparatus 100 of the present embodiment can reduce the number of pieces of image data to be acquired by pixel shifting by using an imaging device with a laminated structure for the imaging device 120 .

A color filter for acquiring a pixel value corresponding to each color is arranged for each pixel in the image sensor with color filters. The color filters arranged for each pixel are regularly arranged two-dimensionally on the plane of the imaging element with color filters, such as a Bayer arrangement.

When acquiring image data using an imaging device with color filters, first, pixel values corresponding to respective colors are acquired for each pixel, and the pixel values of each pixel are acquired as an array. Then, image data is generated by performing demosaic processing on the array of pixel values of each pixel.

Similarly, when acquiring a plurality of image data by pixel shifting using an imaging device with color filters, it is necessary to form an array of pixel values on the premise of demosaic processing. However, a large number of pieces of image data must be acquired in order to configure the array of pixel values on the premise of demosaic processing from a plurality of pieces of image data.

On the other hand, an imaging device with a laminated structure can directly acquire a pixel value corresponding to each color for each pixel. When acquiring multiple image data by pixel shifting using an image sensor with a laminated structure, unlike the case of using an image sensor with color filters, it is necessary to configure an array of pixel values on the premise of demosaic processing. do not have.

Therefore, when acquiring a plurality of pieces of image data by pixel shifting using an imaging device with a laminated structure, the number of pieces of image data to be acquired by pixel shifting can be reduced, and at least two pieces of image data can be acquired. By simply performing a synthesizing process on these, it is possible to generate image data with a resolution higher than that of normal image data.

In particular, the image pickup apparatus 100 of the present embodiment acquires two pieces of image data by pixel shifting using an image pickup device with a three-layer structure, and performs synthesis processing on the two pieces of image data, thereby achieving higher resolution than normal image data. image data.

(Fig. 1: Pixel shift mechanism)
Next, the pixel shift mechanism will be described. In this embodiment, a shift mechanism 130 that shifts the imaging device 120 on a plane perpendicular to the optical axis is employed as a pixel shift mechanism. The shift mechanism 130 can shift the relative position between the subject image and the imaging device 120 on a plane perpendicular to the optical axis. A driving means for shift mechanism 130 is, for example, a voice coil motor.

The shift mechanism 130 shifts the imaging element 120 on a plane perpendicular to the optical axis, and can perform accurate position control to the target position. The target position is determined based on the pixel pitch P of the image pickup device 120 . The imaging element 120 is position-controlled to a plurality of target positions by the shift mechanism 130, and acquires image data at each target position.

Note that the pixel pitch P of the image pickup device 120 is set by setting the intervals in the XY directions of the pixels two-dimensionally arranged at equal intervals in the image pickup device 120 to PX and PY, respectively. In this embodiment, the pixel pitch P in the XY directions is expressed as (PX, PY). Regarding the pixel pitch P in the XY directions, PX and PY may be different from each other.

Further, a shift mechanism that shifts all or part of the photographing lens on a plane perpendicular to the optical axis may be employed as the pixel shift mechanism. By shifting all or part of the photographing lens on a plane perpendicular to the optical axis, the relative position between the subject image and the imaging device 120 can also be shifted on the plane perpendicular to the optical axis.

(Fig. 1: Pixel shift synthesis processing unit)
Next, the pixel shift synthesis processing unit 140 will be described. 1, 141 is an image data input unit, 142 is an image data arrangement unit, 143 is a resampling unit, 144 is a pixel mixing unit, and 145 is an image data output unit.

The image data input unit 141 inputs a plurality of image data acquired by pixel shifting and temporarily held from the DRAM 160 to the pixel shifting synthesis processing unit 140 . In this embodiment, two image data are input from the DRAM 160 to the pixel shift synthesis processing section 140 .

The image data arrangement unit 142 arranges the input image data of a plurality of sheets by shifting them based on the pixel pitch P. FIG. In this embodiment, an image sensor having a three-layer structure is adopted as the image sensor 120, and image data of each layer is acquired for one image data. Therefore, the image data arrangement unit 142 arranges the image data of each layer with respect to the input image data of a plurality of sheets. A method of arranging the input image data by the image data arranging unit 142 will be described later.

The re-sampling unit 143 re-generates the image data of each layer with high resolution or low resolution by re-sampling processing from the image data of each layer that is staggered. The resampling process by the resampling unit 143 will be described later.

The pixel mixing unit 144 generates image data for recording by pixel mixing processing using the image data of each layer regenerated by the resampling processing. Pixel mixing processing by the pixel mixing unit 144 will be described later.

The image data output unit 145 outputs the image data for recording generated by the pixel mixing process. The output image data for recording is recorded in the external memory 170, transferred to an image compression unit (not shown) for image compression, or sent to an image processing unit (not shown) that performs other post-process image processing. transferred.

(Fig. 3: Arrangement method of multiple image data)
Next, a method of arranging the input plural image data by the image data arranging unit 142 will be described.

The imaging apparatus 100 of the present embodiment employs an imaging device with a three-layer structure as the imaging device 120, and acquires two pieces of image data by pixel shifting. The image data arrangement unit 142 arranges the image data of each layer constituting the image data of the two sheets by shifting them based on the pixel pitch P. FIG.

FIG. 3 is a schematic diagram showing a method of arranging two pieces of image data acquired by pixel shifting by the image data arranging unit 142. As shown in FIG. In FIG. 3, image data A and image data B respectively indicate two pieces of image data acquired by pixel shifting.

One piece of image data acquired by the three-layered imaging device is composed of image data of the B layer (blue), the G layer (green), and the R layer (red). That is, in FIG. 3, the image data A is composed of B-layer image data AB, G-layer image data AG, and R-layer image data AR. The image data B is composed of B-layer image data BB, G-layer image data BG, and R-layer image data BR.

As shown in FIG. 3, the image data arrangement unit 142 arranges the image data of each layer constituting the image data A and the image data B in a shifted manner. At this time, the image data arrangement unit 142 arranges the image data of each layer constituting the image data A and the image data B by shifting them based on the pixel pitch P so that the pixel positions of the pixels do not overlap each other.

FIG. 4 is a schematic diagram showing how the image data of each layer constituting the image data A and the image data B are shifted based on the pixel pitch P and arranged. FIG. 4 shows the image data of one layer among the image data of each layer of the image data A and the image data B, and the image data of the other layers are the same as FIG. The schematic diagram of FIG. 4 is schematically represented by the central position of each pixel constituting image data of each layer two-dimensionally arranged on a plane. That is, in FIG. 4, the arrangement of the center positions of the pixels indicates the arrangement of the pixels forming the image data of each layer.

As shown in FIG. 4, the image data of each layer constituting the image data A and the image data B are shifted based on the pixel pitch P so that the center positions of the pixels constituting the image data of each layer do not overlap each other. are placed. In the present embodiment, they are arranged with a shift of 1/2 of the pixel pitch P. FIG. That is, in each layer, they are arranged with a shift of (1/2PX, 1/2PY) which is 1/2 of the pixel pitch P in the XY direction. The amount of shift in each layer is not limited to this embodiment, and by setting it to a non-integer multiple of the pixel pitch P, the pixel centers of the respective pixels do not overlap each other, and the images of each layer constituting the image data A and the image data B are formed. Data can be staggered.

If the image data of each layer constituting the image data A and the image data B are arranged so that the center positions of the respective pixels overlap with each other, the pixel values of the pixels constituting the image data of each layer are simply added. It will be combined. In this case, it becomes impossible to regenerate the image data with high resolution by resampling processing.

(Figure 1: Resampling process)
Next, resampling processing will be described. In this embodiment, the re-sampling process is defined as a new pixel position in the image data of each layer that is shifted, and calculates the pixel value of each layer by interpolation operation at the newly defined pixel position. Regenerate data.

In this embodiment, an image sensor with a three-layer structure is adopted as the image sensor 120, and two pieces of image data are acquired by pixel shifting. Then, the image data of each layer is regenerated by resampling processing from the image data of each layer in which the image data of each layer constituting the two pieces of acquired image data are shifted and arranged. At this time, since the image data of each layer arranged in a staggered manner has twice the information amount of the image data of each layer constituting one piece of image data, the image data of each layer arranged in a theoretically staggered manner is It is possible to regenerate the image data of each layer whose resolution is doubled at maximum from the data.

Here, in the present embodiment, a problem caused by adopting an image sensor with a stacked structure (image sensor with a three-layer structure) as the image sensor 120 and acquiring image data for a plurality of images (two images) by pixel shifting will be described. do.

2. Description of the Related Art In the structure of an imaging device having a laminated structure, photoelectric conversion units that convert a subject image into image signals are arranged in a laminated manner. Therefore, the image data of one sheet acquired by the image pickup device having the laminated structure is composed of the image data of each layer corresponding to the photoelectric conversion units arranged in layers. Further, the resolution of the image data of each layer is the same as the resolution of one image data obtained by the image sensor with color filters. Therefore, the information amount of image data acquired by the image sensor having the laminated structure is much larger than the information amount of the image data acquired by the image sensor with the color filter.

In addition, when an imaging device with a laminated structure is employed and a plurality of pieces of image data are obtained by shifting pixels, the information amount of the image data to be handled is further increased.

As the information amount of the image data to be handled increases, the load for transferring the image data and the load for image processing the image data in the post-process increase. Also, there arises a problem that the capacity of the buffer memory for temporarily holding the image data must be increased.

In this embodiment, an image pickup device with a laminated structure is used, and image data for each layer constituting multiple image data obtained by pixel shifting is divided into a luminance layer and a color layer, and image data with different resolutions are generated for each layer. An increase in the amount of information in the image data is suppressed by regenerating the image data by resampling.

Since human visual characteristics are more sensitive to luminance information than color information, luminance signals among image signals constituting image data have a greater effect on human resolution than color signals. In the present embodiment, in the resampling process, the image data of each layer is divided into a luminance layer and a color layer to increase the amount of information in the image data regenerated in the luminance layer, while increasing the amount of information regenerated in the color layer. The information amount of the image data is reduced, and the information amount of the image data is suppressed while maintaining the resolution of the image data.

Next, a description will be given of a method of dividing the image data of each layer, which are shifted and arranged in the resampling process, into a luminance layer and a color layer. In this embodiment, an image pickup device with a three-layer structure is adopted, and the image data of each layer constituting two pieces of image data acquired by pixel shifting are arranged in a staggered manner. layer, and another layer as a color layer. In particular, in this embodiment, among the image data of each layer, the B layer is the luminance layer, and the G and R layers are the color layers. The method of dividing the image data of each layer into the luminance layer and the color layer is not limited to the method of this embodiment.

Further, instead of dividing the image data of each layer into the luminance layer and the color layer in advance as in the method of the present embodiment, the acquired image data of each layer may be evaluated and divided. For example, among the image data of each layer, one layer from which image data with the highest contrast is obtained may be set as the luminance layer, and the other layers may be set as the color layers. Further, for example, among the image data of each layer, one layer from which the image data with the highest signal value is obtained may be set as the luminance layer, and the other layers may be set as the color layers.

In this embodiment, in order to reduce the amount of information in the image data, in the resampling process of the image data of each layer divided into the luminance layer and the color layer, the luminance layer from which luminance information is obtained is higher than the normal image data of each layer. Image data with reduced resolution is regenerated, and in color layers from which color information is obtained, image data with lower resolution than normal image data of each layer is regenerated.

The resolution of the image data of each layer regenerated by the resampling process in the luminance layer and the color layer is determined by the pixel pitch P' of the image data of each layer regenerated by the resampling process. The pixel pitch P' is determined based on the normal pixel pitch P of the image data of each layer (the pixel pitch P of the image sensor 120).

When the pixel pitch P' of the image data to be regenerated is smaller than the pixel pitch P of the image data of each normal layer, the image data with high resolution is regenerated, and when it is larger, the resolution is reduced. Image data is regenerated.

In this embodiment, the pixel pitch P' of the image data regenerated by the resampling process is 1/√2 times the pixel pitch P in the luminance layer and 2/√2 times the pixel pitch P in the color layer. . As a result, in the luminance layer, image data with twice the resolution of the normal image data in each layer is regenerated, and in the color layer, the resolution is reduced to 1/2 the resolution of the normal image data in each layer. Image data is regenerated.

In this embodiment, for the image data of each layer arranged in a staggered manner, the resolution of the image data regenerated by the resampling process is different between the luminance layer and the color layer, thereby adopting an image pickup device with a laminated structure. It is also possible to suppress the amount of information of the entire image data when acquiring image data with high resolution by pixel shifting.

In particular, in this embodiment, the resolution of the image data regenerated in the luminance layer is increased to increase the amount of information, while the resolution of the image data regenerated in the color layer is decreased to decrease the amount of information. As a result, the information amount of the entire image data including the image data of the luminance layer and the color layer can be suppressed.

Next, in the luminance layer and the color layer, a method of regenerating the image data of each layer by resampling processing will be described.

First, in the image data of each layer that is shifted and arranged, the pixel position of each pixel of the image data regenerated by the resampling process is newly defined. The pixel position of each pixel is newly defined based on the normal pixel pitch P of the image data of each layer.

Next, peripheral pixels to be used for calculating the pixel value at the pixel position of each newly defined pixel are determined. The peripheral pixels are determined so as to include pixels at a predetermined distance from the pixel position of each newly defined pixel in the image data of each layer that is staggered.

Next, at the newly defined pixel position of each pixel, the pixel value is calculated using the determined peripheral pixels. Calculation of the pixel value is performed by an interpolation operation using the pixel values of the determined peripheral pixels. A conventional method such as a bicubic method or a bilinear method may be appropriately adopted as the interpolation calculation method.

Then, the image signal array of the pixel values calculated at all of the newly defined pixel positions of each pixel is regenerated as the image data of each layer.

Next, a method for resampling the luminance layer and the chrominance layer will be described.

(Fig. 5: Re-sampling method of luminance layer)
A method of resampling the luminance layer will be described with reference to FIG. In FIG. 5, an imaging device with a three-layer structure is adopted, and the image data of the luminance layers that constitute two pieces of image data (image data A and image data B) obtained by pixel shifting are shifted and arranged. FIG. 10 is a schematic diagram showing that image data having a higher resolution than normal image data of each layer is regenerated from the image data of the luminance layer arranged in the same manner as the normal image data of each layer by resampling processing.

First, as shown in FIG. 5A, the pixel position of each pixel of the image data regenerated by the resampling process is newly defined in the image data of the luminance layer that is shifted.

The pixel positions newly defined in the luminance layer are determined based on the normal pixel pitch P of the image data of each layer. In order to regenerate image data with higher resolution than normal image data of each layer in the luminance layer, the pixel pitch P′L of the image data regenerated by the resampling process in the luminance layer is defined as P′L<P Set to be In this embodiment, P′L is set to 1/√2P in order to regenerate image data whose resolution is doubled in the luminance layer.

As shown in FIG. 5A, the pixel position of each pixel is newly defined based on the pixel pitch P'L within the range of the image data of the luminance layer that is shifted. The origin (0, 0) is the center position of the upper leftmost pixel of the image data arranged in the luminance layer, and the pixel pitch P′L (1/√ 2PX, 1/√2PY). As a result, the pixel position of each pixel constituting the image data of the luminance layer regenerated by the resampling process is newly defined.

In the luminance layer, the number of pixels in the XY directions of the image data composed of each pixel whose pixel position is newly defined is √2 times that of the normal image data of each layer. Also, the resolution is doubled while keeping the aspect ratio the same as that of the normal image data of each layer. Since the number of pixels in the XY directions that constitute the image data must actually be an integer, the number of pixels in the XY directions must be appropriately replaced with the number of pixels obtained by rounding down the fractions so that the number of pixels in the XY directions becomes an integer.

Next, as shown in FIG. 5B, in the luminance layer, peripheral pixels to be used for calculating the pixel value at the pixel position of each newly defined pixel are determined.

Peripheral pixels are determined so as to include pixels at a predetermined distance from pixels forming image data of the luminance layer that is shifted and arranged for the newly defined pixel position of each pixel.

In this embodiment, the neighboring pixels for the newly defined pixel position of each pixel in the luminance layer are determined by the following equation. The predetermined distance from the newly defined pixel location can be adjusted by defining the range of surrounding pixels in the luminance layer to include one or more surrounding pixels with arbitrary setting of k in the following equation: . Note that the method for determining the peripheral pixels is not limited to the following formula.
|x−x′|+|y−y′|<k (where k is any positive number)
(x, y): Newly defined pixel position coordinates (x', y'): Pixel position coordinates of surrounding pixels

Next, for the pixel position of each newly defined pixel in the luminance layer, the pixel value is calculated using the surrounding pixels.

The pixel value at the pixel position of each newly defined pixel is calculated by interpolation using the pixel values of the surrounding pixels. Conventional interpolation calculation methods such as the bicubic method and the bilinear method may be appropriately adopted as the interpolation calculation method. By calculating pixel values at all pixel positions for each newly defined pixel, it is possible to regenerate image data with high resolution in the luminance layer.

(Fig. 6: How to resampling color layers)
A method of resampling processing for color layers will be described with reference to FIG. In FIG. 6, an imaging device with a three-layer structure is adopted, and image data of color layers constituting two pieces of image data (image data A and image data B) obtained by pixel shifting are arranged in a shifted manner. FIG. 10 is a schematic diagram showing that image data having a lower resolution than normal image data of each layer is regenerated from the image data of the color layers arranged in the same manner by resampling processing.

First, as shown in FIG. 6A, the pixel position of each pixel of the image data regenerated by the resampling process is newly defined in the image data of the color layers that are shifted.

The pixel positions newly defined in the color layer are determined based on the normal pixel pitch P of the image data of each layer. In order to regenerate image data with lower resolution than normal image data of each layer in the color layer, the pixel pitch P'C of the image data regenerated by the resampling process in the color layer is P'C>P. Set to be In this embodiment, P'C is set to 2/√2P in order to regenerate image data whose resolution has been reduced to 1/2 in the color layer.

As shown in FIG. 6A, the pixel position of each pixel is newly defined based on the pixel pitch P'C within the image data of the color layers that are shifted. The origin (0, 0) is the center position of the upper leftmost pixel of the image data arranged in the color layer. Starting from the pixel position (1/2√2PX, 1/2√2PY) shifted by √2, the pixel position is renewed based on the pixel pitch P′C (2/√2PX, 2/√2PY) in the XY direction. defined in As a result, the pixel position of each pixel constituting the image data of the color layer regenerated by the resampling process is newly defined.

Also, as shown in FIG. 6B, one pixel position of each pixel newly defined in the color layer is defined for four pixel positions of each pixel newly defined in the luminance layer. . That is, the resolution of the image data regenerated in the color layer is 1/4 times that of the image data regenerated in the luminance layer.

In the color layer, the number of pixels in the XY directions of the image data composed of each pixel whose pixel position is newly defined is 1/√2 times that of the normal image data of each layer. Also, the resolution is reduced to 1/2 times while keeping the aspect ratio the same as that of the normal image data of each layer. Since the number of pixels in the XY directions that constitute the image data must actually be an integer, the number of pixels in the XY directions must be appropriately replaced with the number of pixels obtained by rounding down the fractions so that the number of pixels in the XY directions becomes an integer.

Next, as shown in FIG. 6C, in the color layer, peripheral pixels to be used for calculating the pixel value at the pixel position of each newly defined pixel are determined.

Peripheral pixels are each determined so as to include a pixel at a predetermined distance from among the pixels forming the image data of the shifted color layers for the newly defined pixel position of each pixel.

In this embodiment, the pixel pitch P'C of the pixel position of each pixel newly defined in the color layer is larger than the pixel pitch P'L of the pixel position of each pixel newly defined in the luminance layer. , the peripheral pixels of the color layer also include pixels at a greater distance from the pixel positions of the newly defined pixels.

In this embodiment, the neighboring pixels for the pixel position of each newly defined pixel in the color layer are determined by the following equation. By defining the range of surrounding pixels in the color layer to include one or more surrounding pixels with arbitrarily setting 2k in the following equation, the predetermined distance from the newly defined pixel location can be adjusted. . Note that the method for determining the peripheral pixels is not limited to the following formula.
|x−x′|+|y−y′|<2k (where k is any positive number)
(x, y): Newly defined pixel position coordinates (x', y'): Pixel position coordinates of surrounding pixels

Next, for the pixel position of each newly defined pixel in the color layer, the pixel value is calculated using the surrounding pixels.

The pixel value at the pixel position of each newly defined pixel is calculated by interpolation using the pixel values of the surrounding pixels. Conventional interpolation calculation methods such as the bicubic method and the bilinear method may be appropriately adopted as the interpolation calculation method. By calculating pixel values at all pixel positions for each newly defined pixel, it is possible to regenerate image data whose resolution has been reduced in the color layer.

(Fig. 1: Pixel mixture processing)
Next, pixel mixing processing by the pixel mixing unit 144 will be described.

In the pixel mixing process, the luminance layer image data and the color layer image data regenerated by the resampling process are combined to finally generate image data with higher resolution than the normal image data of each layer. .

In this embodiment, the pixel mixing process is performed by performing noise reduction processing and color separation processing on the image data of each layer using the image data of each layer having different resolutions regenerated by the resampling processing, and matching the resolutions. It is image signal processing that generates image data with high resolution by synthesizing the images.

A control method of the imaging apparatus 100 of the present embodiment will be described with reference to FIGS. 7 to 10. FIG. The imaging operation in the control method of the imaging apparatus 100 of this embodiment will be described below. Further, among various photographing operations, the explanation will be limited to the photographing operation for generating image data whose resolution is finally increased by pixel shifting.

(Fig. 7: Each step of shooting operation)
Each step of the photographing operation will be described with reference to FIG.

First, in S001, the control unit of the imaging device 100 detects whether or not the release switch has been turned ON. If the release switch has not been turned on, the process returns to S001 to detect whether the release switch has been turned on again. When the release switch is turned on, focusing operation is subsequently performed.

Next, in S002, the control unit of the imaging device 100 performs a focusing operation. The control unit of the imaging device 100 drives the focus lens group included in the photographing lens 110 so as to focus on the subject based on the focus evaluation value detected by the contrast detection method, the phase difference detection method, or the like.

Next, in S003, the control unit of the imaging apparatus 100 acquires two pieces of image data by pixel shifting. The pixel shift is performed by the control unit of the imaging device 100 shifting the imaging element 120 (three-layered imaging element) using the shift mechanism 130 within a plane perpendicular to the optical axis. The imaging device 120 acquires one image data at each of the two positions shifted by the pixel shift, ie, the first position and the second position. In this embodiment, the first position is the origin position (0, 0), and the second position is 1/2 of the pixel pitch P (PX, PY) in the XY direction from the origin position (0, 0). The shifted position is (1/2PX, 1/2PY). Each step of acquiring image data by pixel shifting in S003 will be described later.

Next, in S004, the pixel shift synthesis processing unit 140 performs pixel shift synthesis processing using the two image data acquired at the first position and the second position by pixel shifting. Here, the acquired two image data are acquired by an imaging device with a three-layer structure, and are composed of image data of each layer (B layer, G layer, and R layer). In the pixel shift compositing process, image data with higher or lower resolution than the normal image data in each layer is regenerated from the image data in each layer shifted by the resampling process, and regenerated by the pixel mixing process. This is done by generating image data whose resolution has been increased finally using the image data of each layer. Each step of the pixel shift synthesis processing in S004 will be described later.

Next, in S005, the finally generated high-resolution image data is recorded. The finally generated high-resolution image data is recorded in the external memory 170 for storage, or in a temporary memory inside the image sensor for other image processing.

(FIG. 8: Image data acquisition S003 by pixel shift)
Next, each step of image data acquisition (FIG. 7: S003) by pixel shifting will be described with reference to FIG.

First, in S301, the control unit of the imaging apparatus 100 moves the imaging element 120 to the origin position (0, 0), which is the first position, by pixel shifting. Next, in S302, the imaging device 120 is exposed at the first position to acquire first image data, and in S303, the acquired first image data is temporarily stored in the DRAM 160. FIG.

Subsequently, in S304, the control unit of the imaging device 100 moves the imaging device 120 from the origin position (0, 0), which is the second position, to 1/2 the pixel pitch P (PX, PY) in the XY directions by pixel shifting. 1/2PX, 1/2PY). Next, in S305, the image sensor 120 is exposed at the second position to acquire second image data, and in S306, the acquired second image data is temporarily held in the DRAM 160. FIG. At this time, the DRAM 160 records two pieces of image data acquired by pixel shifting.

In the present embodiment, in acquiring image data by pixel shifting, the image sensor 120 is arranged at each position relatively at a pixel pitch P (PX , PY).

(FIG. 9: Pixel shift synthesis processing S004)
Next, each step of the pixel shift synthesis process (FIG. 7: S004) will be described with reference to FIG.

First, in S<b>401 , the image data input unit 141 inputs two image data recorded in the DRAM 160 to the pixel shift synthesis processing unit 140 .

Next, in S402, the pixel shift synthesis processing unit 140 divides the image data of each layer constituting the image data of the two sheets into a luminance layer and a color layer. In this embodiment, the image data of the B layer is set as the luminance layer in advance, and the image data of the G layer and the R layer are set as the color layers.

Next, in S403, the pixel shift synthesis processing unit 140 causes the image data arrangement unit 142 to shift and arrange the image data of the luminance layer and the color layer, which respectively constitute the image data of the two sheets. At this time, the image data arrangement unit 142 arranges the image data of each layer with a relative shift of 1/2 of the pixel pitch P (PX, PY) in the same manner as the pixel shift.

Next, in S404, the pixel shift synthesis processing unit 140 uses the resampling unit 143 to convert the image data of each layer, which are shifted and arranged in the luminance layer and the color layer, into each layer having a resolution different from that of the normal image data of each layer. regenerate the image data of At this time, in the luminance layer, image data with a higher resolution than the normal image data of each layer is regenerated, and in the color layer, image data with a lower resolution than the normal image data is regenerated.

Next, in S405, the pixel shift synthesis processing unit 140 uses the regenerated image data of the luminance layer or the color layer to generate image data whose resolution is finally increased by the pixel mixing unit 144. FIG. Each step of the pixel mixing process of S405 will be described later.

(FIG. 10: Pixel mixture processing S405)
Next, each step of the pixel mixing process (FIG. 9: S405) will be described with reference to FIG.

In the following description, one piece of image data composed of the image data of the luminance layer and the color layer is expressed using the resolution ratio of the image data of each layer. Also, the ratio of the resolution of the image data of each layer is expressed in the order of the R layer, the G layer, and the B layer.

In this embodiment, the pixel shift synthesis processing unit 140 divides the image data of each layer constituting one image data in advance into a B layer as a luminance layer and G and R layers as a color layer. The resolution ratio of the image data of the luminance layer and the color layer regenerated by the resampling process is 4:1.

Therefore, for example, one sheet of image data composed of luminance layer and color layer image data regenerated by the resampling process can be expressed as 114 image data using the resolution ratio of the image data of each layer. can. Further, for example, one sheet of image data obtained by deleting the image data of the R layer and the G layer, which are color layers, from the 114 image data can be expressed as 004 image data.

First, in S501, the pixel shift synthesis processing unit 140 generates 111 image data and 004 image data from the 114 image data regenerated by the pixel mixing unit 144 by resampling processing.

Here, the 111 image data is image data composed of the image data of the R layer and the G layer of the 114 image data, and the image data of the B layer obtained by summing the pixel values of a set of four adjacent pixels of the 114 image data. is. The 004 image data is image data composed only of the image data of the B layer of the 114 image data by deleting the image data of the R layer and the G layer from the 114 image data.

Next, in S502, 111 image data and 001 image data are further generated from the 111 image data generated in S501.

Here, the 001 image data is image data composed only of the image data of the B layer of the 111 image data by deleting the image data of the R layer and the G layer from the 111 image data.

Next, in S503, the 111 image data generated in S502 are subjected to noise reduction processing and color separation processing. to generate

The image data of each layer forming the 444 image data generated in S504 corresponds to the image data whose resolution is doubled as compared with the normal image data of each layer.

On the other hand, in S505, noise reduction processing is performed on the 004 image data generated in S501. In S506, the 001 image data generated in S502 is subjected to upsampling processing to generate 004 image data.

Then, in S507, difference image data is generated by calculating the difference between the 004 image data processed in S505 and the 004 image data generated in S506.

Finally, in S508, the difference image data generated in S507 is reflected on the 444 image data generated in S504, and finally each layer whose resolution is doubled with respect to the normal image data of each layer is obtained. 444 image data composed of image data of .

In the pixel mixing process of this embodiment, 111 image data and 004 image data are once generated from 114 image data composed of image data of the luminance layer and color layer regenerated by the resampling process. Then, from the 111 image data, 444 image data is generated by performing noise reduction processing and the like and upsampling processing, and from the 004 image data, 004 image data is generated without performing the upsampling processing while performing the noise reduction processing. is generating Furthermore, by reflecting the 004 image data, which is the differential image data, on the 444 image data, the final image data of each layer is composed of the image data of each layer whose resolution is twice as high as that of the normal image data of each layer. 444 image data is generated. As a result, it is possible to prevent the problem of coloring of the image data that occurs when the upsampling process is performed after the noise reduction process or the like is performed.

As described above, according to the present invention, in an imaging device and a control method for an imaging device that acquires image data with high resolution by pixel shifting, by adopting an imaging device with a laminated structure, acquisition is performed by pixel shifting. The number of pieces of image data to be processed can be reduced. In addition, it is possible to reduce the load of data transfer and image processing of multiple image data, and to reduce the capacity of the buffer memory for temporarily holding multiple image data.

In addition, in the resampling process of the image data of each layer that constitutes a plurality of image data acquired by pixel shifting, the image data of each layer is divided into a luminance layer and a color layer, By making the resolution of the reproduced image data different between the luminance layer and the color layer, the information amount of the entire image data can be suppressed.

REFERENCE SIGNS LIST 100 imaging device 110 photographic lens 120 imaging device 130 shift mechanism 140 pixel shift synthesis processing unit 141 image data input unit 142 image data arrangement unit 143 resampling unit 144 pixel mixing unit 145 image data output unit 150 bus 160 DRAM
170 external memory

Claims (8)

An imaging device that generates image data having a higher resolution than normal image data by synthesizing a plurality of pieces of image data acquired by pixel shifting,
A photographing lens that forms an image of subject light on an imaging surface as a subject image; an imaging device with a laminated structure in which a photoelectric conversion unit that converts the subject image into an image signal is laminated; Equipped with a pixel shift mechanism that shifts the relative position on a plane perpendicular to the optical axis, and a pixel shift synthesis processing unit that generates one image data by synthesis processing from multiple image data acquired by pixel shifting,
The pixel shift synthesis processing unit includes an image data input unit for inputting the two image data acquired by pixel shifting, and an image data arrangement unit for shifting and arranging the image data of each layer constituting the two image data respectively. a resampling unit that regenerates the image data of each layer by resampling processing from the image data of each layer that is staggered; 1. An imaging apparatus comprising: a pixel mixing section for generating; and an image data output section for outputting one image data generated by the pixel mixing processing. The pixel shift mechanism shifts the relative position between the subject image and the image pickup device with the laminated structure on a plane perpendicular to the optical axis by moving the image pickup device with the laminated structure in a direction perpendicular to the optical axis. The imaging device according to claim 1, characterized by: The pixel shift mechanism shifts the relative position between the subject image and the image sensor having the laminated structure on a plane perpendicular to the optical axis by moving all or part of the photographing lens in a direction perpendicular to the optical axis. 2. The image pickup apparatus according to claim 1, wherein the image pickup apparatus The pixel shift mechanism is characterized in that the relative position between the subject image and the image sensor with the stacked structure is shifted in the XY directions on a plane perpendicular to the optical axis by 1/2 of the pixel pitch of the image sensor with the stacked structure. 4. The imaging apparatus according to any one of claims 1 to 3. The pixel shift synthesis processing unit divides the image data of each layer constituting the two image data obtained by pixel shifting using the image sensor of the laminated structure into a luminance layer and a color layer,
The resampling process regenerates image data having a higher resolution than normal image data in the luminance layer, and regenerates image data having a lower resolution than normal image data in the color layer. 5. The imaging apparatus according to any one of claims 1 to 4. The pixel shift synthesis processing unit sets the image data of the layer having the highest contrast among the image data of each layer constituting the two pieces of image data obtained by pixel shifting using the image sensor having the laminated structure as the luminance layer. 6. The image pickup apparatus according to claim 5, wherein the image data of layers other than that are used as color layers. The pixel shift synthesis processing unit synthesizes the image data of the layer having the highest signal value among the image data of each layer constituting the two pieces of image data obtained by pixel shifting using the image pickup device having the layered structure. 6. The imaging apparatus according to claim 5, wherein the image data of other layers are used as color layers. A control method for an imaging device for generating image data having a higher resolution than normal image data by synthesizing multiple pieces of image data acquired by pixel shifting, the method comprising:
An image data acquisition step of acquiring data of two images by pixel shifting using an imaging device with a laminated structure, an image data input step of inputting the acquired image data of the two images, and the input image data of the two images. a resampling step of regenerating the image data of each layer using the image data of each layer shifted by resampling; a pixel characterized by having a pixel mixing processing step of generating one image data using the image data of each layer regenerated by the mixing processing, and an image data output step of outputting the generated one image data A control method for an imaging device.

Images