WO2015053080A1

WO2015053080A1 - Image pickup device and image pickup method

Info

Publication number: WO2015053080A1
Application number: PCT/JP2014/075224
Authority: WO
Inventors: 康祐梶村
Original assignee: オリンパス株式会社
Priority date: 2013-10-10
Filing date: 2014-09-24
Publication date: 2015-04-16
Also published as: JP2015076796A

Abstract

The present invention acquires an image with satisfactorily improved resolution by performing synthesis processing corresponding to a difference in a plurality of frames. Provided is an image pickup device provided with: an image pickup element (11) which acquires a plurality of frames of an optical image of a subject on a time-series basis; a movement means (12) which relatively moves the position of the image pickup element with respect to the optical image; a movement control means which controls the relative movement directions and movement amounts of the optical image and the image pickup element by the movement means; a pixel placement means which, in accordance with the relative movement directions and movement amounts of the optical image and the image pickup element, performs associations of respective pixel positions between the plurality of frames; a difference calculation means which calculates a difference between pixels the pixel positions of which are associated with each other in at least two frames among the plurality of frames; and an image synthesis means which, according to the difference, determines the synthesis ratio of corresponding pixels of another frame with respect to a frame as a reference among the plurality of frames, and synthesizes the corresponding pixels of the frame as the reference and the other frame according to the determined synthesis ratio.

Description

Imaging apparatus and imaging method

The present invention relates to an image pickup apparatus and an image pickup method, and more particularly to an image pickup apparatus and an image pickup method for increasing the resolution by synthesizing a plurality of images taken continuously by pixel shifting.

As a technique for generating a composite image from a plurality of images, there is a technique for calculating a correlation amount between a plurality of images and generating a composite image by controlling a composite ratio between images based on the correlation amount. For example, Patent Document 1 discloses that, between a reference image and a plurality of other images, a combination ratio is increased for a portion with a large correlation amount such as a difference value, and a portion with a small correlation amount is used. By reducing the composition ratio, artifacts due to subject movement and displacement are prevented from occurring.

By the way, a technique for improving the resolution by shifting the pixels of the reference image and the pixels of each of a plurality of other images is known. That is, for example, a technique is known in which super-resolution processing is performed by arranging a plurality of low-resolution Bayer array images taken continuously by pixel shifting on a high-resolution image. In such a technique, an effect of improving the resolution can be expected in a region such as a fine repetitive pattern.
However, if a low-resolution Bayer array image is arranged on a high-resolution image and super-resolution processing is performed as it is from the arranged pixel information, artifacts such as multiple images and false colors are generated in the area where the subject has moved.
Therefore, it is conceivable to apply the technique of Patent Document 1 described above. In this case, after performing demosaicing processing or the like on a plurality of Bayer array images, a correlation amount is calculated between a low-resolution G image, a luminance image, or an image obtained by expanding them to the same image size as the above-described high-resolution image. A method of calculating and controlling the composition ratio can be applied.

JP 2011-199786 A

However, in the case of performing the high resolution processing as described above, if the technique of Patent Document 1 is applied as it is, imaging is performed by relatively shifting the position of the optical image and the imaging system for each image by pixel shifting. The correlation amount may be small not only in the moving subject area but also in the still area. In particular, the above-described areas such as the fine repetitive patterns are areas where aliasing (moire) can occur in the low-resolution image after the demosaicing process. Since all the images do not have the same color pixel at the same position and the aliasing (moire) pattern generated in the low-resolution image after demosaicing is different, the amount of difference between images increases and the amount of correlation decreases. . For this reason, if a static region that is originally highly effective in improving the resolution is subjected to synthesis control by applying the technique of Patent Document 1, it is erroneously determined that there is a positional shift, and the effect of improving the resolution cannot be obtained.

The present invention has been made in view of the above-described circumstances, and an object of the present invention is to obtain an image in which resolution is improved satisfactorily by performing a synthesis process according to a difference between a plurality of frames.

In order to achieve the above object, the present invention provides the following means.
One aspect of the present invention is an image sensor that acquires a plurality of frames of an optical image of a subject in time series, a moving unit that relatively moves the position of the image sensor relative to the optical image, and the optical image by the moving unit. And a movement control means for controlling a relative movement direction and movement amount between the image sensor and the relative movement direction and movement amount between the optical image and the image sensor. A pixel arrangement unit that performs association; a difference calculation unit that calculates a difference between pixels corresponding to pixel positions in at least two frames of the plurality of frames; A composite ratio of corresponding pixels of the frame is determined according to the difference, and the corresponding pixels of the reference frame and the other frames are combined according to the determined composite ratio. An image synthesizing means for, to provide an imaging device comprising a.

According to this aspect, a plurality of frames are acquired in time series by relatively moving the position of the image sensor with respect to the optical image, and one image with a high resolution is acquired by combining these plurality of frames. . In this case, the pixel placement unit associates each pixel position between a plurality of frames in accordance with the relative movement direction and movement amount of the optical image and the image sensor by the movement control unit. That is, each pixel position between a plurality of frames is shifted in the movement direction by the movement amount while considering the movement amount between the frames caused by the relative movement. Then, the difference calculation means calculates a difference between the pixels corresponding to the pixel positions in at least two frames among the plurality of frames that are associated with each other, and becomes a reference among the plurality of frames according to the difference. A combination ratio of corresponding pixels of another frame with respect to the frame is determined, and corresponding images of the reference frame and the other frame are combined according to the determined combination ratio to obtain one image.

By doing this, the difference between multiple frames is small for a static region that should be small, and large for a region where there is a moving subject that should be large. can do. Then, the composition ratio between frames is determined in accordance with the correctly calculated difference, and a composite image is generated based on this, so that an image with improved resolution can be obtained with certainty.

In the aspect described above, it is preferable that the movement control unit controls the movement direction and the movement amount of the imaging element so that a pixel for which a difference is calculated by the difference calculation unit is located within a predetermined pixel range.
By doing this, it is possible to eliminate the acquisition of an image that does not contribute to the increase in resolution, and to reliably acquire an image with improved resolution when performing synthesis processing according to the difference between multiple frames. Can do.

In the aspect described above, it is preferable that the difference calculating unit calculates a difference between corresponding pixels between frames acquired at least two frames apart in time.
By doing this, it is possible to eliminate the acquisition of an image that does not contribute to the increase in resolution, and to reliably acquire an image with improved resolution when performing synthesis processing according to the difference between multiple frames. Can do.

In the aspect described above, it is preferable that the difference calculating unit calculates a difference between corresponding pixels between frames acquired at a predetermined time interval.
By doing this, it is possible to eliminate the acquisition of an image that does not contribute to the increase in resolution, and to reliably acquire an image with improved resolution when performing synthesis processing according to the difference between multiple frames. Can do.

In the aspect described above, when the predetermined pixel range is an enlargement magnification M (M ≧ 2) at the time of image composition by the image composition unit, it is preferable that the predetermined pixel range is ½ M pixels or less.
By doing in this way, when acquiring a plurality of frames by so-called pixel shifting, by setting the moving range of the image sensor to an appropriate range, it is possible to eliminate the acquisition of an image that does not contribute to high resolution, It is possible to reliably acquire an image with improved resolution when performing synthesis processing according to the difference between a plurality of frames.

In the above-described aspect, it is preferable that the movement control unit controls the movement amount of the imaging element to a range of less than two pixels in the horizontal direction and the vertical direction.
By doing this, it is possible to eliminate the acquisition of an image that does not contribute to the increase in resolution, and to reliably acquire an image with improved resolution when performing synthesis processing according to the difference between multiple frames. Can do.

In the aspect described above, it is preferable that one of the frames used for the difference calculation by the difference calculation unit is not used for image synthesis.
By doing this, it is possible to eliminate the acquisition of an image that does not contribute to the increase in resolution, and to reliably acquire an image with improved resolution when performing synthesis processing according to the difference between multiple frames. Can do.

According to another aspect of the present invention, there are provided a step of acquiring a plurality of frames of an optical image of a subject in time series using an image sensor, a step of moving the position of the image sensor relative to the optical image, and the optical image. And a step of controlling a relative movement direction and a movement amount between the image sensor and a correspondence between each pixel position between the plurality of frames according to a relative movement direction and a movement amount between the optical image and the image sensor. A step of calculating a difference between pixels corresponding to pixel positions in at least two frames of the plurality of frames, and combining corresponding pixels of other frames with a reference frame of the plurality of frames The ratio is determined according to the difference, and the corresponding pixels of the reference frame and the other frames are combined according to the determined combination ratio A step, to provide an imaging method comprising.

According to the present invention, there is an effect that it is possible to obtain an image with improved resolution reliably by performing a synthesis process according to a difference between a plurality of frames.

1 is a block diagram illustrating a schematic configuration of an imaging apparatus according to a first embodiment of the present invention. FIG. 2 is a reference diagram illustrating a structure of an imaging element in the imaging apparatus according to the first embodiment of the present invention. In the 1st Embodiment of this invention, it is a reference drawing which shows the concept in the case of moving the position of the image pick-up element relatively with respect to an optical image. In the first embodiment of the present invention, it is a reference diagram showing the relationship between the number of frames taken and the shift control according to the time axis. FIG. 6 is an explanatory diagram illustrating an example of a pixel arrangement diagram when a low resolution frame is arranged on a high resolution image in the first embodiment of the present invention. FIG. 5 is a reference diagram illustrating an example of combining a standard high resolution frame and a reference high resolution frame in the first embodiment of the present invention. In the 1st Embodiment of this invention, it is a graph which shows the relationship between the difference value by a difference calculation part, and the synthetic | combination ratio in each pixel. In the 1st Embodiment of this invention, it is explanatory drawing which concerns on the predetermined pixel range when the shift | offset | difference of an image sensor, etc. are considered. In the 2nd Embodiment of this invention, it is a reference figure which shows the relationship between the imaging | photography frame number according to a time-axis, and shift control. In the 2nd Embodiment of this invention, it is explanatory drawing which shows the example of a pixel arrangement | positioning map at the time of arrange | positioning a low resolution frame on a high resolution image. In the 3rd Embodiment of this invention, it is a reference figure which shows the relationship between the imaging | photography frame number according to a time-axis, and shift control. In the 3rd Embodiment of this invention, it is explanatory drawing which shows the example of a pixel arrangement | positioning map at the time of arrange | positioning a low resolution frame on a high resolution image.

(First embodiment)
Hereinafter, an imaging apparatus according to a first embodiment of the present invention will be described with reference to the drawings.
As shown in FIG. 1, the imaging device includes an imaging element 11 that acquires an optical image of a subject, a shift mechanism (moving means) 12 that moves the imaging element 11, and the moving direction and amount of movement of the imaging element 11 by the shift mechanism 12. The shift control unit 13 to be controlled, the pixel placement unit 14 that associates each pixel position between a plurality of frames, the difference calculation unit 15 that calculates the difference between the corresponding pixels between the plurality of frames, and the combination ratio of the corresponding pixels are determined Thus, an image composition unit 16 that generates a composite image by combining the corresponding pixels is provided.

The image sensor 11 collects an optical image of the subject that is condensed by the imaging lens 10 and formed on the image plane of the image sensor 11, thereby obtaining an image related to the optical image of the subject (hereinafter referred to as “frame”). To get. In the imaging apparatus according to the present embodiment, a so-called super-resolution process is performed in which a plurality of relatively low resolution frames are acquired in time series by the imaging element 11 and a high resolution image is acquired by combining these frames. Yes. For this reason, in this embodiment, the image sensor 11 has the Bayer arrangement structure shown in FIG. 2 and acquires a plurality of frames of low resolution images related to the optical image of the subject in time series.

As shown in FIG. 3, the shift mechanism 12 moves the position of the image sensor 11 relative to the optical image, so that the image sensor 11 can be relatively moved in units of subpixels in the horizontal and vertical directions. It has become.
The shift control unit 13 controls the direction and amount of relative movement of the image sensor 11 by the shift mechanism 12 for each frame. That is, the shift control unit 13 outputs a control signal related to the moving direction and the moving amount to the shift mechanism 12 so as to move the image sensor 11 relative to the optical image according to a predetermined condition.

Hereinafter, the control in the shift control unit 13 will be specifically described. Here, a case will be described in which eight frames are acquired by the image sensor 11 and a high-resolution image that is twice as long and horizontally is generated.

The shift control unit 13 sets the first frame to be acquired as a reference frame, and when acquiring the second and subsequent frames with respect to this reference frame, as shown in FIG. The shift mechanism 12 is controlled so that the movement direction and the movement amount of 11 are as follows.
The second image is 0.5 pixels in the X direction, 0.5 pixels in the Y direction, the third image is 1.0 pixels in the X direction, the 0.0 pixels in the Y direction, and the fourth image is 1.5 pixels in the X direction. 0.5 pixels in the Y direction 5th image is 1.0 pixel in the X direction, 1.0 pixel in the Y direction 6th image is 1.5 pixels in the X direction, 1.5 pixels in the Y direction 7th image is in the X direction 0.0 pixel in the direction, 1.0 pixel in the Y direction, the eighth image is 0.5 pixel in the X direction, and 1.5 pixel in the Y direction. While shifting from the first image to the eighth image, 8 images are captured in time series to obtain a plurality of low-resolution frames.
By controlling the moving amount and moving direction of the image sensor 11 as in the above example, the difference calculation target pixels of the image that is four frames away from the reference frame are at the same position.

The pixel arrangement unit 14 is acquired in time series by the image sensor 11, and the relative movement direction of the image sensor 11 by the shift mechanism 12 with respect to a plurality of frames converted into digital signals by the A / D converter 17. And according to the movement amount, each pixel position between the frames is associated. That is, as shown in FIGS. 5A and 5B, the pixel arrangement unit 14 moves the amount of movement between the frames caused by the relative movement when acquiring a high resolution image from a plurality of low resolution frames. In consideration of the above, each pixel position between a plurality of low-resolution frames is arranged on the high-resolution image while performing the association by shifting in the movement direction by the movement amount. In the example of FIG. 5, the corresponding pixels of the frames that are photographed with four frames separated in time are associated with each other and arranged at the same position by the pixel arrangement unit 14.

That is, FIG. 5A shows a state in which the first to eighth Gr pixels (Gr1 to Gr8) are arranged, and FIG. 5B shows the first to eighth Gb pixels. A state in which pixels (Gb1 to Gb8) are arranged is shown.
In this way, Gr1 and Gb5, Gr2 and Gb6, Gr3 and Gb7, Gr4 and Gb8, Gr5 and Gb1, Gr6 and Gb2, Gr7 and Gb3, Gr7 and Gb3 of the image arranged on the high resolution image in the pixel arrangement unit 14 Gr8 and Gb4 are associated with each other and are at the same position. And the difference calculation part 15 mentioned later can calculate G difference value using the pixel value of each same position.

The difference calculation unit 15 calculates the difference between the pixels corresponding to the pixel positions in at least two frames among the plurality of frames that are associated with each other. That is, in the example of FIG. 5, the corresponding pixels of the frames that are photographed with four frames separated in time are associated with each other by the pixel arrangement unit 14 and arranged at the same position. By calculating the component difference (difference between the Gr component and the Gb component), the difference value becomes small for the still region in the frame, and the difference value becomes large for the moving subject region.

The image composition unit 16 determines a composition ratio of corresponding pixels of other frames with respect to a reference frame among the plurality of frames according to the difference calculated by the difference calculation unit 15. Further, the corresponding pixels of the reference frame and other frames are combined according to the determined combination ratio. In the example of FIG. 5, one pixel of the reference high resolution frame in which the acquired 8 frames are arranged and the reference low resolution frame so as to fill the pixels at the same position as this reference high resolution frame is copied to 8 locations. The arranged reference high resolution frames are synthesized in accordance with the synthesis ratio determined according to the difference of the G component to generate a synthesized image (see FIG. 6).

FIG. 7 shows an example of the relationship between the difference value of the G component obtained by the difference calculation unit 15 and the composition ratio of the reference high resolution frame. In the example of FIG. 7, when the G difference value is smaller than the threshold value 1, the composition ratio α is set to 1, and when the G difference value is larger than the threshold value 2, the composition ratio α is linearly changed from the threshold value 1 to the threshold value 2. The synthesis ratio α is set to zero.

The operation of the imaging apparatus configured as described above will be described.
A control signal related to the direction and amount of relative movement of the image sensor 11 with respect to the optical image is output to the shift mechanism 12 by the shift control unit 13, and the shift mechanism 12 moves the image sensor 11 according to the control signal. The image sensor 11 acquires a plurality of frames with low resolution in time series while moving the relative position with respect to the optical image according to the direction and amount of relative movement in the shift control unit 13.

In the image placement unit 14, each pixel position between a plurality of frames is moved in the moving direction by the amount of movement while considering the amount of movement between the low-resolution frames caused by the relative movement of the optical image and the image sensor 11 by the shift control unit 13. The images are arranged on the high-resolution image while being shifted and associated.

Then, the difference calculation unit 15 calculates a difference in G components (difference between the Gr component and the Gb component) between the pixels that are associated with each other and are arranged at the same position. Of these, the combination ratio of the corresponding pixels of the other frame with respect to the reference frame, for example, the first frame is determined, and the corresponding pixels of the reference frame and the other frame are combined according to the determined combination ratio to 1 Get an image.

As described above, according to the present embodiment, the difference between a plurality of frames is small for a static region that should originally be small, and large for a region where a moving subject that should originally be large is present, and can be accurately detected without erroneous determination. The difference can be calculated. Then, the composition ratio between frames is determined in accordance with the correctly calculated difference, and a composite image is generated based on this, so that an image with improved resolution can be obtained with certainty.

In particular, in the present embodiment, since the difference of each pixel is calculated from a frame that is temporally separated by four frames, the difference value can be clearly made different between the stationary region and the moving subject region. Therefore, as shown in FIG. 7, the resolution of the composite image can be improved by increasing the composite ratio α of the reference high-resolution frame in the region where the difference value is small. On the other hand, in an area where the difference value is large, an effect of suppressing artifacts such as multiple images and false colors can be expected by increasing the composition ratio (1-α) of the reference high resolution frame.

In the image composition unit 16, an image obtained by demosaicing one Bayer image is used as a standard high resolution frame, and an image obtained by interposing blank pixels by arranging eight low resolution frames is used as a reference high resolution frame. Synthesis may be performed.
Further, an image obtained by applying a smoothing filter to an image obtained by arranging eight low-resolution frames and interpolating blank pixels in this manner may be used as the reference high-resolution frame.

In any case, it is possible to improve resolution in an area with a small difference value and suppress artifacts such as multiple images and false colors in an area with a large difference value, and an image with improved resolution can be obtained.

Note that when calculating the difference between corresponding pixels, the difference calculation target pixels do not have to be arranged completely at the same position, but may be located within a predetermined pixel range.
FIG. 8 shows a case where the position of relative movement of the image sensor 11 is within a predetermined pixel range. The position 200 is the position of the pixel before the relative movement, and the position 201 represents the position of the difference calculation target pixel.

As illustrated in FIG. 8, when the pixel value is acquired at the position 201 by moving by 0.5 pixel in the X direction and 0.5 pixel in the Y direction from the position 200, the control accuracy of the shift mechanism 12 or accidental The image sensor 11 cannot always be moved to the position 201 due to an error in the amount of movement due to the shake of the image sensor 11, and may move to a position 202 that is deviated from the position 201 indicated by x.

Even in such a case, if the position 202 is within the range of −0.25 to +0.25 in both the X direction and the Y direction with the position 201 of the difference calculation target pixel as the center, the position 202 is acquired at the position 202. The pixel value can be approximated to the pixel value acquired at the position 201, and the difference can be calculated between the pixel value at the position 201 and the pixel value at the position 202.

By doing so, even if an error due to the control accuracy of the shift mechanism 12 or accidental blurring of the image sensor 11 occurs, the composition processing according to the difference is performed between a plurality of frames, and the resolution is improved satisfactorily. The acquired image can be acquired.
When synthesizing a high-resolution image with an enlargement magnification of 2 times in the vertical and horizontal directions, the predetermined pixel range that can tolerate the position error is within ± 0.25 pixels from the ideal shift amount. On the other hand, when the image sensor is relatively moved beyond the allowable range, it is not preferable to apply the frame to difference calculation or image synthesis.

In the present embodiment, an example is shown in which the shift control unit 13 performs control so that the difference calculation target pixels of images separated by four frames are at the same position, but the present invention is not limited to this, and two or more frames are used. The difference target pixels of the images acquired at intervals may be controlled to be at the same position, and the difference calculation target pixels of the images of at least two frames acquired at predetermined intervals are the same position. You may control so that it may become.
In this case, the corresponding pixels of the plurality of frames controlled so that the difference target pixels are located at the same position are associated with each other by the pixel placement unit 14 and disposed at the same position, and the association is performed. The difference between each pixel corresponding to the pixel position is calculated between at least two frames.

(Second Embodiment)
Hereinafter, a second embodiment of the present invention will be described.
The imaging apparatus in the present embodiment has the same configuration as that of the first embodiment, and the control by the shift control unit 13 is different. That is, as shown in FIGS. 9 and 10, in the present embodiment, the shift control unit 13 makes the moving direction of the image sensor 11 different from that in the first embodiment.

The image sensor 11 acquires eight low-resolution frames as in the first embodiment, and the image composition unit 16 generates a high-resolution image that is twice the vertical and horizontal directions.
The shift control unit 13 sets the first frame to be acquired as a reference frame, and when acquiring the second and subsequent frames with respect to this reference frame, as shown in FIG. The shift mechanism 12 is controlled so that the movement direction and the movement amount of 11 are as follows.

The second sheet is 1.0 pixel in the X direction, 0.0 pixel in the Y direction. The third sheet is 1.5 pixels in the X direction, 0.5 pixel in the Y direction. The fourth sheet is 0.5 pixel in the X direction. 0.5 pixels in the Y direction 5th image is 0.0 pixels in the X direction, 1.0 pixel in the Y direction 6th image is 0.5 pixels in the X direction, 1.5 pixels in the Y direction 7th image is in the X direction 1.5 pixels in the direction and 1.5 pixels in the Y direction The eighth sheet is 1.0 pixel in the X direction and 1.0 pixel in the Y direction

By controlling the shift mechanism 12 in this way, the image sensor 11 captures eight images in time series while shifting from the first to the eighth images to obtain a plurality of low-resolution frames.
By controlling the moving amount and moving direction of the image sensor 11 as in the above example, the difference calculation target pixels are located at the same position in frames that are separated by 3 frames or 7 frames in time.
When the low-resolution frame acquired in this way is pixel-arranged on the high-resolution image by the pixel arrangement unit 14, the arrangement relationship is as shown in FIG.

FIG. 10A shows a state in which the first to eighth Gr pixels (Gr1 to Gr8) are arranged, and FIG. 10B shows the first to eighth Gb pixels (Gb1). To Gb8) are shown.
In this way, Gr1 and Gb8, Gr2 and Gb5, Gr3 and Gb6, Gr4 and Gb7, Gr5 and Gb2, Gr6 and Gb3, Gr7 and Gb4, Gr1 and Gb8 of the image arranged on the high resolution image in the pixel arrangement unit 14 Gr8 and Gb1 are associated with each other and are at the same position. Then, the difference calculation unit 15 can calculate the G difference value using the pixel values at the same positions.

Each difference value is calculated from an image of a frame that is three frames or more apart in time, and in particular, a position 300 where the pixel Gr1 and the pixel Gb8 are arranged at the same position, and a position 301 where the pixel Gr8 and the pixel Gb1 are arranged at the same position. Then, the difference is calculated using an image separated by seven frames in terms of time. In this way, in a moving subject area, a certain large difference value is obtained at all positions where pixels are arranged, and even larger difference values can be expected particularly at the pixel position 300 and the pixel position 301.

Since a large difference value can be expected at the position 300 and the position 301 among the positions calculated by the difference between the eight sets of Gr and Gb, the difference calculation unit 15 can, for example, determine the maximum value of the difference value for each arbitrary small divided region. May be set as the difference value of the small area, and the composition ratio may be determined for each small area. Further, an addition average may be calculated for each small divided area, and this addition average may be used as a difference value of the small area. Alternatively, only the difference value between Gr1 and Gb8 and between Gr8 and Gb1 may be calculated using only the first and eighth frames, and may be used as the difference value of the small divided area.
In any case, since a large difference value can be obtained in the moving subject area, the composition ratio can be controlled so that the image composition unit suppresses false colors and artifacts.

(Third embodiment)
A third embodiment of the present invention will be described.
The imaging apparatus according to the present embodiment has the same configuration as that of each of the above-described embodiments, and the control by the shift control unit 13 and the method of generating a composite image by the image composition unit 16 are different.
That is, as shown in FIGS. 9 and 10, in the present embodiment, the shift control unit 13 changes the moving direction of the image sensor 11 from that of each of the above-described embodiments, and the image composition unit 16 accordingly. However, the method for generating a composite image is different from that of the above-described embodiments.

In this embodiment, the number of images N is N = 9, and the image pickup device 11 having the Bayer array structure shown in FIG. 2 is picked up in time series as shown in FIG. 11 to obtain nine low-resolution frames. To do.
In the present embodiment, the shift control unit 13 controls the moving amount and moving direction of the image sensor 11 so that the difference calculation target pixels are located at the same position in frames that are 4 frames or 8 frames away from each other in time. Get low resolution frames. When these low-resolution frames are arranged on the high-resolution image by the pixel arrangement unit 14, the arrangement relationship is as shown in FIG. Since the shift control unit 13 controls the difference between the shift amounts of the first and ninth sheets to be 0, the first and ninth sheets of Gr, as shown in FIG. Gb overlaps at the same position as shown in FIG. Therefore, the difference calculation unit 15 calculates the difference value between Gr1 and Gr9 at the position 400 using only the first and ninth frames, and calculates the difference value between Gb1 and Gb9 at the position 401. For the difference value at other positions, for example, the difference value is referred from the nearest position 400 and position 401.

By doing so, the difference value becomes small in the still area, and a large difference value can be acquired by moving the moving object area by 8 frames in time.
The image composition unit 16 adds and averages the first and ninth pixels arranged at the same position on the high resolution image and rearranges them at the same position on the high resolution image, and then composes the image. Alternatively, the first and ninth pixels may be weighted and averaged based on the difference value and rearranged, and then the images may be combined. In addition, the image may be synthesized using only the pixel value of either the first sheet or the ninth sheet.

In each of the above-described embodiments, the case where the image is enlarged to double the vertical and horizontal resolutions has been described. However, for example, the resolution can be increased to three times the vertical and horizontal. Further, the amount of movement of the image sensor by the shift mechanism 12 is not necessarily limited to subpixels, and if the control accuracy is sufficient, there is no limit to the enlargement magnification N times. Further, the order of movement, direction, and amount of movement by the shift control unit 13 of the image sensor 11 are not limited to the above-described embodiment, and various patterns can be applied.

Furthermore, in each embodiment, although the example which calculates G difference value using G pixel was shown, this invention is applicable not only to G pixel but the pixel which has another color filter.
As described above, according to each of the embodiments described above, in the super-resolution processing by so-called pixel shift imaging, the difference value between a plurality of images in the still region is suppressed to be small, and the difference value in the moving subject region is calculated to be a large value. In addition, it is possible to increase the resolution by performing an appropriate composition process on each of the stationary region and the moving subject region during image composition.

DESCRIPTION OF SYMBOLS 11 Image sensor 12 Shift mechanism 13 Shift control part 14 Pixel arrangement part 15 Difference calculation part 16 Image composition part 17 A / D conversion part

Claims

An image sensor for acquiring a plurality of optical images of a subject in time series, and
Moving means for moving the position of the image sensor relative to the optical image;
Movement control means for controlling the relative movement direction and amount of movement of the optical image and the image sensor by the movement means;
Pixel placement means for associating each pixel position between the plurality of frames according to a relative movement direction and a movement amount of the optical image and the imaging device;
A difference calculating means for calculating a difference between pixels corresponding to pixel positions in at least two frames of the plurality of frames;
A combination ratio of corresponding pixels of another frame with respect to a reference frame among the plurality of frames is determined according to the difference, and corresponding pixels of the reference frame and the other frame are combined according to the determined combination ratio An image synthesizing unit.
The imaging apparatus according to claim 1, wherein the movement control unit controls the movement direction and the movement amount of the imaging element so that a pixel for which a difference is calculated by the difference calculation unit is located within a predetermined pixel range.
The imaging apparatus according to claim 1 or 2, wherein the difference calculating means calculates a difference of corresponding pixels between frames acquired at least two frames apart in time.
3. The imaging apparatus according to claim 1, wherein the difference calculating means calculates a difference between corresponding pixels between frames acquired at predetermined time intervals.
3. The imaging apparatus according to claim 2, wherein the predetermined pixel range is equal to or less than ½ M pixels when the enlargement magnification M (M ≧ 2) at the time of image composition by the image composition means.
6. The imaging apparatus according to claim 5, wherein the movement control means controls the movement amount of the imaging element in a range of less than two pixels in the horizontal direction and the vertical direction.
The imaging apparatus according to any one of claims 1 to 6, wherein the image synthesizing unit does not use one of the frames used for calculation of the difference by the difference calculating unit for image synthesis.
Acquiring a plurality of frames of an optical image of a subject in time series by an image sensor;
Moving the position of the image sensor relative to the optical image;
Controlling the relative moving direction and moving amount of the optical image and the imaging device;
Associating each pixel position between the plurality of frames according to a relative moving direction and moving amount of the optical image and the imaging device;
Calculating a difference between pixels corresponding to pixel positions in at least two frames of the plurality of frames;
A combination ratio of corresponding pixels of another frame with respect to a reference frame among the plurality of frames is determined according to the difference, and corresponding pixels of the reference frame and the other frame are combined according to the determined combination ratio An imaging method comprising the steps of: