JP2019149701A - Imaging apparatus and control method of the same - Google Patents

Abstract

To obtain an optical flow with high accuracy without making a user feel a sense of disparity.SOLUTION: The imaging apparatus includes an imaging device and control means. The control means includes: first and second image signal generation means for generating first and second image signals by performing signal charge transfer n times during first and second imaging periods; first and second averaging means for generating third and fourth image signals in which luminance values of respective pixel units are averaged with respect to the first and second image signals; optical flow candidate calculating means for comparing the first and second image signals to calculate a plurality of optical flow candidates; approximate optical flow calculating means for calculating an approximate optical flow by comparing to the third and fourth image signals; and optical flow estimation means for estimating one optical flow candidate closest to an approximate optical flow among multiple optical flow candidates as a final optical flow.SELECTED DRAWING: Figure 10

Description

  The present invention relates to an imaging apparatus having a function of calculating an optical flow based on a signal from an imaging element, and a control method therefor.

  In recent years, higher functionality of image sensors has been developed, and the number of pixels and the frame rate have been improved. A method for obtaining vector data (optical flow) indicating the movement of an object between a plurality of images using this signal is known. The optical flow serves as an index indicating the amount and direction of camera shake in moving image shooting, for example, and is used for calculating the cutout amount and direction in electronic image stabilization by cutting out an image from a larger image. It is also an index for estimating the moving direction and speed of a moving subject, and is also used for autofocus tracking.

  The acquisition of the optical flow is obtained by calculating the movement amount / direction of the object between the images of the front and back frames in the moving image by comparing the images. At this time, the subject and camera shake are fast, and if the object moves greatly during the exposure time of one frame, the image of the object obtained as a result of exposure blurs in the moving direction. When comparing images of blurred objects, the contours of the images of the objects are not clear, so the position cannot be accurately compared, and the optical flow that is the amount and direction of movement of the objects between the images is precisely There are problems that cannot be sought.

  With respect to the above problem, the imaging apparatus disclosed in Patent Literature 1 sharpens the outline of the target object by increasing the shutter speed per frame and shortening the exposure time according to the speed of the target object. Propose to get a good image. The imaging apparatus of Patent Document 2 transfers charges converted by the photoelectric conversion unit to the storage unit a plurality of times, and accumulates the charges transferred a plurality of times at a time, so that conditions such as exposure time and exposure amount can be increased at high speed. Propose to change freely. In addition, by applying this, it is also possible to obtain a moving image by distributing a short accumulation period evenly in one frame period and accumulating charges transferred multiple times in one frame period.

JP 2010-206522 A JP 2010-157893 A

  The imaging apparatus of Patent Document 1 can obtain a sharp image even on a fast moving object by increasing the shutter speed and shortening the exposure time in one frame during moving image shooting. However, increasing the shutter speed in moving image shooting has the following drawbacks. In general, it is known that when a reproduced moving image has a kind of frame-by-frame feeling, quality is greatly lost. In order to prevent such discomfort, it is necessary to set an accumulation time close to one frame period in a series of shootings. That is, if the frame rate is 30 fps, a relatively long exposure time such as 1/30 second or 1/60 second is appropriate. In other words, the act of obtaining a sharp image by increasing the shutter speed during movie shooting and reducing the exposure time within one frame is contrary to setting a relatively long exposure time close to the one frame period described above. There is a problem that the video becomes easy to appear.

  In addition, the image pickup apparatus disclosed in Patent Document 2 can perform shooting with a relatively long exposure time while suppressing the amount of light by repeatedly performing transmission to the exposure / accumulation unit for a short time a plurality of times in one frame period. . In addition, since the image obtained thereby is an image that is overlapped by the transmission to the exposure / accumulation unit multiple times, compared to the imaging device of Patent Document 1, there is less sense of parallax when viewed as a moving image. Become. However, since the imaging apparatus of Patent Document 2 does not assume that an optical flow is obtained, it is not a multi-time divided accumulation method suitable for obtaining it accurately. Further, when an optical flow is to be acquired by the imaging apparatus disclosed in Patent Document 2, the contours of these images overlap each other when the images are compared between frames. For this reason, it is difficult to accurately select a common contour between frames, and there is a problem that precise detection cannot be performed.

  The present invention provides an imaging apparatus capable of obtaining an optical flow with high accuracy without causing the user to feel a sense of disparity, and a control method therefor.

  According to an example of the present invention, an imaging apparatus includes an imaging device having a plurality of pixel units, each pixel unit including a photoelectric conversion unit and a signal holding unit, and a control unit that controls the imaging device. . The control means converts the signal charge accumulated in the signal holding unit by n signal charge transfers (n is a natural number of 2 or more) from the photoelectric conversion unit to the signal holding unit during the first imaging cycle. Based on the first image signal generation means for generating the first image signal and the signal holding unit by n times of signal charge transfer from the photoelectric conversion unit to the signal holding unit during the second imaging cycle. Second image signal generation means for generating a second image signal based on the accumulated signal charge, and a predetermined number of luminance values of each pixel unit adjacent to the generated first image signal First averaging means for generating a third image signal by taking the average of the luminance values of the other pixel portions, and the luminance value of each pixel portion for the generated second image signal. To average the brightness value of a predetermined number of other pixel parts adjacent to it By comparing the first and second image signals with the second averaging means for generating the fourth image signal, the direction and amount of movement of the subject during the first and second imaging periods The subject is moved between the first and second imaging periods by comparing the third and fourth image signals with an optical flow candidate calculating means for calculating a plurality of optical flow candidates that are vectors indicating And a schematic optical flow calculation means for calculating a schematic optical flow which is a vector indicating a general direction and quantity to be processed, and one optical flow candidate closest to the schematic optical flow among the plurality of optical flow candidates is determined as a final optical flow. And an optical flow estimating means for estimating as follows.

  According to the example of the present invention, it is possible to provide an imaging apparatus capable of obtaining an optical flow with high accuracy and a control method thereof without causing the user to feel a sense of disparity.

It is an external view of a digital still motion camera. It is a block diagram which shows the Example of an imaging device. It is a circuit diagram showing an example of a circuit of an image sensor. 6 is a timing chart illustrating an example of a driving sequence of an image sensor. It is explanatory drawing which shows the example of the optical flow acquisition method. It is a figure which shows the captured image in a 1st comparative example. It is a figure which shows the captured image in the 2nd comparative example. It is a figure which shows the captured image in an Example. It is a figure which shows the image after a low-pass filter application in an Example. It is a flowchart which shows the optical flow estimation flow in an Example. It is a timing chart in case the exposure in an Example is performed at a nonuniform timing. It is a figure which shows the example of an imaging in the case of performing the exposure in an Example at a nonuniform timing.

  Hereinafter, an imaging apparatus in which a photographing optical system for imaging is added to the video processing apparatus will be described as a preferred embodiment of the present invention.

<Imaging device>
1A and 1B are external views of a digital still motion camera as an imaging device according to an embodiment of the present invention. 1A is a front view of the imaging device, and FIG. 1B is a rear view of the imaging device. In these drawings, reference numeral 151 denotes an image pickup apparatus main body in which an image pickup element and a shutter device are housed. Reference numeral 152 denotes a photographing optical system having an aperture inside. Reference numeral 153 denotes a movable display unit for displaying shooting information and video. Reference numeral 154 denotes a switch ST used mainly for photographing a still image. Reference numeral 155 denotes a switch MV which is a button for starting and stopping moving image shooting. The display unit 153 has a display luminance range that can display an image with a wide dynamic range without suppressing the luminance range. Reference numeral 156 denotes a shooting mode selection lever for selecting a shooting mode. Reference numeral 157 denotes a menu button for shifting to a function setting mode for setting a function of the imaging apparatus. Reference numerals 158 and 159 denote up / down switches for changing various setting values. Reference numeral 160 denotes a dial for changing various setting values. Reference numeral 161 denotes a playback button for shifting to a playback mode in which video recorded on a recording medium housed in the imaging apparatus main body is played on the display unit 153.

  FIG. 2 is a block diagram showing a schematic configuration of the imaging apparatus of the present invention. In the figure, reference numeral 184 denotes an image sensor that converts an optical image of a subject formed through the photographing optical system 152 into an electrical video signal. Reference numeral 152 denotes a photographing optical system that forms an optical image of a subject on the image sensor 184. Reference numeral 180 denotes an optical axis of the photographing optical system 152. Reference numeral 181 denotes an aperture for adjusting the amount of light passing through the photographing optical system 152. The diaphragm 181 is controlled by the diaphragm controller 182. Reference numeral 183 denotes an optical filter that limits the wavelength of light incident on the image sensor 184 and the spatial frequency transmitted to the image sensor 184. The image sensor 184 has a sufficient number of pixels, signal readout speed, color gamut, and dynamic range to satisfy the Ultra High Definition Television standard.

  Reference numeral 187 denotes a digital signal processing unit that compresses video data after performing various corrections on the digital video data output from the image sensor 184. A timing generation unit 189 outputs various timing signals to the image sensor 184 and the digital signal processing unit 187. A system control CPU 178 controls various calculations and the entire digital still motion camera. The timing generator 189 and the system control CPU 178 correspond to “control means” in the claims.

  Reference numeral 190 denotes a video memory for temporarily storing video data. Reference numeral 191 denotes a display interface unit for displaying captured images. Reference numeral 153 denotes a display unit such as a liquid crystal display. Reference numeral 193 denotes a removable recording medium such as a semiconductor memory for recording video data, additional data, and the like. Reference numeral 192 denotes a recording interface unit for performing recording or reading on the recording medium 193. Reference numeral 196 denotes an external interface unit for communicating with the external computer 197 and the like. Reference numeral 195 denotes a printer such as a small inkjet printer. Reference numeral 194 denotes a print interface unit for outputting captured images to the printer 195 for printing. Reference numeral 199 denotes a computer network such as the Internet. Reference numeral 198 denotes a wireless interface unit for communicating with the network 199. Reference numeral 179 denotes switch input means including a switch ST154, a switch MV155, and a plurality of switches for switching various modes.

  FIG. 3 is a circuit diagram showing a part of the image sensor 184 of FIG. The figure shows a pixel portion 300 in the first row and first column (1, 1) and an arbitrary m row and first column (m, 1) among a large number of matrix-like pixels included in the image sensor 184 in FIG. The pixel portion 301 is shown. Since the pixel unit 300 and the pixel unit 301 have the same configuration, the components of the pixel units 300 and 301 are numbered with the same number. In addition, since the basic structure of the image sensor 184 having a signal holding unit is disclosed in, for example, Patent Document 2, description thereof is omitted here.

<Description of image sensor configuration and image signal generation process>
In the circuit diagram of FIG. 3, one pixel portion 300 includes a photodiode 500, a first transfer transistor 501A, a signal holding portion 507A, and a second transfer transistor 502A. The photodiode 500 and the signal holding unit 507A correspond to the “photoelectric conversion unit” and the “signal holding unit” in the claims, respectively. Further, one pixel portion 300 includes a third transfer transistor 503, a floating diffusion region 508, a reset transistor 504, an amplification transistor 505, and a selection transistor 506. The image sensor 184 in which a plurality of pixel portions having the above configuration are two-dimensionally arranged corresponds to an “image sensor” in the claims.

  The first transfer transistor 501A is controlled by the transfer pulse φTX1A, and the second transfer transistor 502A is controlled by the transfer pulse φTX2A. The reset transistor 504 is controlled by a reset pulse φRES, and the selection transistor 506 is controlled by a selection pulse φSEL. Further, the third transfer transistor 503 is controlled by the transfer pulse φTX3. Here, each control pulse is transmitted from a vertical scanning circuit (not shown). Further, 520 and 521 are power supply lines, and 523 is a signal output line.

The details of the operation of the image sensor will be described below with reference to FIG.
FIG. 4 is a timing chart showing a driving sequence of the image sensor 184 in FIG. This figure assumes that movie shooting is performed under the condition of 30 fps, and corresponds to the case where an image signal is obtained by adding 4 times the accumulation of 1/480 seconds during 1/30 seconds, which is one shooting period. To do.
Note that the image sensor 184 of this embodiment has a large number of pixel columns in the vertical direction, and FIG. 4 shows the timing of the first row. These controls are scanned in the vertical direction by the horizontal synchronization signal, whereby the accumulation operation of all the pixels of the image sensor 184 is performed.

  In FIG. 4, rising times t1 and t6 of the vertical synchronization signal φV are vertical synchronization times at which the imaging cycle starts, and 1/30 seconds, which is the time from t1 to t6, is “the first imaging cycle”. "Or" second imaging cycle ". In addition, as a shooting condition, a moving image is subjected to exposure four times during a period of 1/30 seconds, that is, accumulation of charge (signal) of 1/480 seconds, and by adding these four charges, It is assumed that the exposure amount is equivalent to one exposure for 1/120 second.

  First, at time t1, the timing generator 189 causes the vertical synchronization signal φV to be at a high level, and at the same time, the horizontal synchronization signal φH is at a high level. In synchronization with the time t1 when the vertical synchronizing signal φV and the horizontal synchronizing signal φH become high level, the reset pulse φRES (1) of the first row becomes low level. Then, the reset transistor 504 in the first row is turned off, and the reset state of the floating diffusion region 508 is released. At the same time, when the selection pulse φSEL (1) of the first row becomes a high level, the selection transistor 506 of the first row is turned on, and the image signal of the first row can be read. Further, an output corresponding to a change in the potential of the floating diffusion region 508 is read out to the signal output line 523 through the amplification transistor 505 and the selection transistor 506. The signal read to the signal output line 523 is supplied to a read circuit (not shown) and output to the outside as an image signal (moving image) on the first row.

  Next, at time t2, when the transfer pulse φTX2 (1) in the first row becomes high level, the second transfer transistor 502A in the first row is turned on. At this time, the reset pulse φRES (1) for all rows is already at the high level and the reset transistor 504 is on, so that the floating diffusion region 508 and the first signal holding unit 507A in the first row are reset. At time t2, the selection pulse φSEL (1) in the first row is at a low level.

  Next, at time t3, the transfer pulse φTX3 (1) in the first row becomes low level. Then, the third transfer transistor 503 is turned off, the reset of the photodiodes 500 in the first row is released, and accumulation of signal charges for moving images in the photodiodes 500 is started. At time t4, the transfer pulse φTX1 (1) of the first row becomes high level. Then, the first transfer transistor 501A is turned on, and the signal charge accumulated in the photodiode 500 is transferred to the signal holding unit 507A that holds the moving image charge in the first row. Further, at time t5, the transfer pulse φTX1 (1) in the first row becomes low level. Then, the third transfer transistor 501A is turned off, and the transfer of the signal charge accumulated in the photodiode 500 to the signal holding unit 507A is completed.

  Here, the period from time t3 to time t5 corresponds to a one-time accumulation time of 1/480 seconds of moving images in the shooting period, and is illustrated as an accumulation time 602-1 in the upward-slashed area. That is, when such an accumulation operation is performed four times in a discrete manner, four accumulation times 602-1, 602-2, 602-3, and 602-4 in the right-upward shaded area are illustrated. Then, by adding the signal charges obtained by these four accumulation times 602-1, 602-2, 602-3, and 602-4, the normal accumulation time (1/480 seconds × 4 times = 1) / Signal amount equivalent to the signal charge obtained in 120 seconds). Note that the control operations in the three accumulation times 602-2, 602-2, and 602-4 following the first accumulation time 602-1 are the same as those in the first accumulation time 602-1, and thus the description thereof is omitted here. To do.

  Next, at time t6, the timing generator 189 causes the vertical synchronization signal φV to be at a high level, and simultaneously the horizontal synchronization signal φH to be at a high level, so that the next imaging cycle is started. The moving image signal charges accumulated and added during the four accumulation times 602-1, 602-2, 602-3, and 602-4 are output to the outside as image signals (moving images) after time t6. Note that the timing chart in the second row is executed in synchronization with the horizontal synchronous vibration φH immediately after time t1. That is, the timing chart for all rows is started between time t1 and time t6. For example, the timing chart started by the horizontal synchronization signal φH at time t0 is the m-th row. In this case, the switch signal can be expressed as φSEL (m), φRES (m), φTX3 (m), φTX1A (m), φTX1B (m), φTX2A (m), and φTX2B (m).

  According to the timing chart as described above, a moving image is exposed equivalent to one exposure of 1/120 seconds by repeating accumulation of signal charges by exposure of 1/480 seconds during a shooting period of 1/30 seconds. Can be obtained as a quantity. The operation of performing the exposure / accumulation a plurality of times during the one photographing cycle to obtain an image signal is “n” from the photoelectric conversion unit to the signal holding unit during the first or second photographing cycle. This corresponds to the operation of generating the first or second image signal by the signal charge transfer of the first time. However, n is a natural number of 2 or more.

  The moving image obtained at this time is configured to obtain one image signal by adding a short accumulation time set at approximately equal intervals during a 1/30 second shooting period. High-quality video without In the above-described embodiment, the number of signal charge accumulation and addition (number of divided exposures) in a normal exposure interval (fps value) is four, but is not limited to this. For example, eight times, sixteen times, It may be 32 times, 64 times, etc.

<Obtain optical flow>
In recent years, higher functionality of image sensors has been developed, and the number of pixels and the frame rate have been improved. A method for obtaining vector data (optical flow) indicating the movement of an object between a plurality of images using this signal is known. In this embodiment, the optical flow is obtained from the through image. The system control CPU 178 in FIG. 2 generates an optical flow based on comparison between a plurality of images obtained from the image sensor 184. The system control CPU 178 refers to “first and second image signal generation means, first and second averaging means, optical flow candidate calculation means, general optical flow calculation means, and optical flow estimation means” in the claims. Is provided.

FIG. 5 is a diagram illustrating an example of how to obtain an optical flow that is a shake detection signal based on a plurality of image comparisons. This corresponds to the specific operation of the system control CPU 178 in FIG. FIG. 5 is based on the so-called block matching method, but other methods may be used. 5 (A) shows the image acquired at time _{t n,} FIG. 5 (B) shows the image acquired at time _{t n + 1} after time _{t n.} Further, FIG. 5 (C) and displayed overlapping the acquired image at time t _n acquired by the image and the time t _{n + 1,} also shows the detected vector schematically.

5, 61 is a acquired image at time _{t n,} 62 are subject, 63 is a region of interest in the image 61. Reference numeral 64 denotes an image acquired at time t _{n + 1} , 65 denotes an area having the same position in the screen as the area of interest 63, and 66 denotes an arrow schematically indicating that searching is performed, 67 Is a region in the image 64 corresponding to the region of interest 63 of the subject 62 in the image 61. Furthermore, 68 is an image obtained by superimposing images acquired at time t _n and time t _{n + 1} , and 69 is a movement vector detected in the region of interest 63 in the image 61.

  First, as shown in FIG. 5, two images 61 and 64 acquired at different times are prepared. Of these, attention is paid to a region of interest 63 in which the subject 62 exists in one image 61. The size of the region of interest 63 can be set arbitrarily, but may be 8 × 8 pixels, for example. Then, a search is performed by comparing the feature points to determine where the attention area 63 has moved in the image 64.

  Specifically, in the image 64, an edge or a corner in the image 64 is gradually shifted within a predetermined range around the region 65 corresponding to the region of interest 63 as indicated by an arrow 66. Extract feature points. Furthermore, matching between the two images 61 and 64 is performed by calculating a feature amount from the surrounding area. Extraction of feature points such as edges and corners is performed by calculating a luminance gradient value in the horizontal direction and the vertical direction with respect to the luminance data, and extracting a portion where the gradient value has a certain value or more in each direction.

  As a result, it can be seen that the region of interest 63 has moved like a vector 69 in the image 68. Then, the above operation is performed for a plurality of regions of interest set in the image 61. In this case, a plurality of movement vectors are detected in the image 68. Thereafter, paying attention to the subject 62, vector selection is performed. For example, by using RANSAC (Random Sample Consensus), an estimated value can be obtained and one evaluation value of the movement vector can be determined. Since RANSAC is a known technique, detailed description thereof is omitted here. At this time, in the conventional imaging method, there is a problem that a precise optical flow is not required when the contour of the image between the frames to be compared becomes unclear due to the movement of the object or camera shake.

<Problems when acquiring optical flow by extracting feature points>
Hereinafter, problems in obtaining an optical flow will be described with reference to schematic diagrams of images obtained when a scene in which a subject passes in the horizontal direction on the screen is photographed with different exposure methods.
6 (A), 6 (B), 6 (C), and 6 (D) are first comparative examples, where the subject S (for example, the Shinkansen) crosses the screen in the horizontal direction. An example in which a scene is captured as a moving image with an exposure time of 30 fps, 1 frame 1/30 second is shown.

  6A is a diagram in which one frame is extracted from the captured moving image, and FIG. 6B is a diagram in which one frame after the one frame in FIG. 6A is extracted. Hereinafter, one frame in FIG. 6A is referred to as a first frame, and one frame in FIG. 6B is referred to as a second frame. At this time, in both figures, the subject S, which is the object to be imaged, moves at high speed, so that the outline of the subject S is blurred by the amount moved during 1/30 seconds that is the exposure time. I understand that.

  Here, the feature amount is calculated from the contours of the images of the subject S in the first frame and the second frame, which are the next frame, shown in FIGS. 6A and 6B, and both are calculated. Consider the case of trying to compare. In general, when comparing feature amounts, first, an outline portion (edge portion) is extracted. As a technique used for extracting the contour (edge), the luminance value I (x, y) of each pixel is spatially differentiated in a specific direction, and the gradient value K (x) of the luminance I (x, y) with respect to that direction. , Y) is generally obtained.

In FIG. 6A and FIG. 6B, in order to simplify the description, consider the case of performing spatial differentiation in the horizontal direction (x direction). Further, it is assumed that the luminance is discussed by paying attention only to the pixels in the row α shown in the figure.
At this time, if the luminance value at the pixel in the X column on the line α is I (X), the gradient value K (x) is expressed by the following equation (1).
K (x) = dI (X) / dx (1)

FIG. 6C is a graph showing the relationship between the position in the x direction (which column is the pixel in the x direction) and the luminance I (x) in the pixel in the row α shown in the figure.
The solid line portion in FIG. 6C indicates the luminance value of the pixel on the line α in the first frame, and the alternate long and short dash line portion indicates the luminance value of the pixel on the line α in the second frame. The luminance value Ia on the vertical axis indicates the luminance value of the vehicle body of the subject S (Shinkansen), and the luminance value Ib on the vertical axis indicates the luminance value of the background. In this example, Ia> Ib.

  Assuming that the subject S is moving at a substantially constant speed, a portion where the contour is blurred because the subject S moves during exposure in FIGS. 6A and 6B is shown in FIG. In the graph of the luminance value I (x), a corresponding portion rises substantially linearly.

  FIG. 6D is a graph showing the relationship between the x-direction position and the luminance gradient value K (x) in the pixels in the line α. As in the case of the luminance graph, the solid line portion indicates the gradient value of the luminance of the pixel on the line α in the first frame, and the alternate long and short dash line portion indicates the luminance of the pixel on the line α in the second frame. The slope value of is shown.

  6A and 6B, the brightness value is a straight line that rises at a substantially constant slope when the brightness value in FIG. 6C is referred to. Therefore, the gradient value of the luminance value in FIG. 6D has a substantially constant value indicated by the value of K1 in the portion where the outline is blurred. Here, normally, a pixel having a gradient value equal to or greater than a predetermined value is determined as a pixel indicating a contour (position). However, when the contour is blurred as in this example, it is apparent from FIG. As described above, a portion where the gradient value (differential value) is equal to or greater than a certain value is spread over a wide range. Therefore, the contour position in the first frame and the contour position in the second frame cannot be determined precisely, and as a result, there arises a problem that the amount of movement of the subject S cannot be determined accurately.

  FIGS. 7A, 7B, 7C, and 7D show a second comparative example, in which the exposure time for one frame is simply shortened with respect to the above-described problem. An example in the case where a measure for obtaining a sharp outline is taken. As an example of the shooting condition, an example in which a scene in which the subject S crosses the screen in the horizontal direction is shot with a moving image at an exposure time of 30 fps, 1 frame 1/30 second is shown.

  FIG. 7A is a diagram in which one frame is extracted from the captured moving image, and FIG. 7B is a diagram in which one frame after the one frame in FIG. 7A is extracted. As can be seen from FIGS. 7A and 7B, exposure and transfer are performed only for a short time (1/480 seconds) immediately after the start of each frame for 1/30 seconds, and signal charges are accumulated. Therefore, the amount of movement of the subject S during the exposure time is small compared to the first comparative example, and an image having a sharp outline is obtained.

  FIG. 7C is a graph showing the relationship between the x-direction position (which column is the pixel in the x direction) and the luminance I (x) in the pixels in the row of the line α ′. In FIG. 7C, the solid line portion indicates the luminance value of the pixel on the line α ′ in the first frame, and the alternate long and short dash line portion indicates the luminance value of the pixel on the line α ′ in the second frame. Show. The luminance value Ia on the vertical axis indicates the luminance value of the vehicle body of the subject S (Shinkansen), and the luminance value Ib on the vertical axis indicates the luminance of the background. In this example, Ia> Ib.

  At this time, the sharp outline portion of the subject S in FIGS. 7A and 7B becomes a graph in which the corresponding portion suddenly rises in the graph of the luminance value I (x) in FIG. 7C. .

  FIG. 7D is a graph showing the relationship between the x-direction position and the luminance gradient value K (x) in the pixels in the row of the line α ′. As in the case of the luminance graph, the solid line portion indicates the gradient value of the luminance of the pixel on the line α ′ in the first frame, and the alternate long and short dash line portion indicates the pixel on the line α ′ in the second frame. The gradient value of the brightness is shown.

  7A and 7B described above have a graph shape that rises with a steep slope in the luminance value of FIG. 7C. That is, it can be seen that the gradient value of the luminance in FIG. 7D rises locally to a large gradient value (denoted as K2) within the x-axis range where the corresponding part is extremely narrow. At this time, since the portion where the luminance gradient value is a certain value or more is a very narrow range, the contour position in the x-axis direction can be obtained accurately. Therefore, the optical flow can be calculated accurately. However, with this method, the optical flow is required precisely, but there is a problem in terms of the quality of the moving image.

  As described above, only the first short time (1/480 seconds) of each frame is exposed, transferred, and stored, so the image is captured for the remaining time of 1/30 seconds after that. It will not be. The next image capture starts at the start of the second frame, and the subject continues to move in the traveling direction during that time. As shown in FIGS. 7A and 7B, the position of the subject S when the second frame starts and the exposure is started is the position of the subject S in the image captured in the first frame. It is in the position that moved greatly from. As a result, as shown in the gradient graph of FIG. 7D, from the raised portion of the graph showing the outline of the subject S of the first frame to the raised portion showing the outline of the subject S of the second frame. The distance will open. When the user views the video, when moving from the first frame to the second frame, the subject S appears as if it has moved this distance momentarily, which causes the above-mentioned feeling of paralysis. To reduce the quality of the video.

  On the other hand, in the above-described example in which the contour of FIG. 6 is blurred, the left edge of the raised portion of the graph showing the blurred contour of the first frame in the gradient value graph of FIG. There is almost no distance between the right ends of the raised portions of the graph showing the blurred outline. At this time, for the user who watches the moving image, when the transition from frame to frame is made, the positions of the outlines of the front and rear frames are almost continuous, so that it appears that the transition is smooth and without a sense of disparity. Therefore, since the contour is blurred, the optical flow cannot be accurately calculated, but the moving image has a high quality without a sense of disparity.

  From the above, as shown in the example of FIG. 6, when the exposure is performed for approximately the same time such as 1/30 seconds with respect to the shooting cycle time of one frame, the contour of the moving object is blurred and the optical flow cannot be accurately calculated. , The video will be high-quality without smooth transition. On the contrary, as shown in the example of FIG. 7, when exposure is performed for a very short time such as 1/480 second with respect to the shooting period time of one frame, a sharp outline is obtained even for a moving object, and the optical The flow can be calculated accurately. However, it can be seen that the video will be low quality with a feeling of flipping.

  As described above, in the first and second comparative examples, precise calculation of the optical flow and high-quality moving image shooting without parallax cannot be achieved at the same time.

<Optical flow calculation using divided exposure>
On the other hand, in the present invention, the above problem is solved by estimating the optical flow using a method of acquiring a moving image by performing exposure, transfer, and accumulation in a plurality of times in 1 frame 1/30 seconds. To solve. The explanation is given below.

  FIG. 8 shows the scene in which the subject S crosses the screen in the horizontal direction according to the configuration of the present embodiment, and the exposure is divided into 4 times of 1/480 seconds in a shooting period of 1 frame / 30 seconds with 30 fps movie shooting. The acquired image is shown.

  FIG. 8A shows one frame extracted from the captured moving image, and FIG. 8B shows the one frame extracted. As shown in FIGS. 8 (A) and 8 (B), the subject S is exposed four times. Therefore, the subject S is overlapped while being shifted by about 1/4 of the moving amount of 1/30 second in the traveling direction. A four-point sharp outline of A1 to A4 in the first frame and B1 to B4 in the second frame is shown.

  FIG. 8C is a graph showing the relationship between the position in the x direction (which column is the pixel in the x direction) and the luminance I (x) in the pixel in the row of the illustrated line α ″. The solid line portion in FIG. 8C indicates the luminance value of the pixel on the line α ″ in the first frame, and the alternate long and short dash line portion indicates the luminance value of the pixel on the line α ″ in the second frame. Show. The luminance Ia shown on the vertical axis indicates the luminance of the subject S (Shinkansen) vehicle body, and Ib indicates the luminance of the background. In this example, Ia> Ib.

  At this time, the sharp outline of the subject S that is quadrupled while being shifted in FIGS. 8A and 8B corresponds to the luminance value I (x) graph of FIG. 8C. A staircase graph where each point rises rapidly.

  FIG. 8D is a graph showing the relationship between the x-direction position and the luminance gradient value K (x) in the pixels in the row α ″ shown in the figure. As in the case of the luminance graph, the solid line portion indicates the gradient value of the luminance of the pixel on the line α ″ in the first frame, and the alternate long and short dash line portion indicates the line α ″ in the second frame. The gradient value of the luminance of the pixel is shown.

  8A and 8B described above, a portion of a sharply contoured portion that overlaps while being shifted rises with a steep slope of the stepped graph in the luminance value of FIG. 8C. It has become. For this reason, the luminance gradient value in FIG. 8D rises locally to a large gradient value (denoted as K3) only in the x-axis range where the corresponding part is very narrow, and this is the first frame. It can be seen that there are four locations for each of the second frames. At this time, since the portion where the brightness gradient value has a certain value or more is a very narrow range, the positions of the four contours in each frame in the x-axis direction can be accurately obtained.

Therefore, as long as the contours in the second frame corresponding to the contours of interest in the first frame can be accurately selected, the positions of the respective contours can be obtained accurately, so the optical flow that is the movement vector is also precisely determined. It will be required. In FIG. _8D , the optical flow is obtained as, for example, the illustrated vector F _A1B1 , which is a movement vector from the leftmost edge A1 in the first frame to the corresponding leftmost edge B1 in the second frame. It is done.

  The operation of obtaining an image in which the plurality of sharp outlines overlap is performed by “n times of signal charge transfer from the photoelectric conversion unit to the signal holding unit during the first or second imaging period” in the claims. This corresponds to “operation for generating first or second image signal”.

  Further, in this example, as described above, the exposure is performed a plurality of times in a short time every 1/30 seconds of the photographing period. For this reason, as shown in the figure, the ends of the ridges indicating the contours A1 to A4 in the graph of the first frame indicated by the solid line and the ridges indicating the contours B1 to B4 in the graph of the second frame indicated by the alternate long and short dash line The distance between the ends is kept small. Therefore, even for a user who views a moving image, the contour of the subject S in the first frame and the contour of the subject S in the second frame can be seen almost continuously, resulting in a high-quality moving image without a sense of disparity.

<Selecting the appropriate contour by stepwise refinement>
However, in the above method, edges of the same shape are equally spaced, such as A1 to A4 indicating the contour of the subject S of the first frame and edges B1 to B4 indicating the contour of the subject S of the second frame. Are lined up. In this case, there is a possibility that the same edge corresponding to the edge of the first frame is erroneously selected from the second frame. That is, in the case of this example, as in the case of Patent Document 2, since the contours of the same shape are arranged at the same interval, there is a possibility that the corresponding part is erroneously recognized in the comparison by the feature point detection with the blocks limited. Because.

  Specifically, in FIG. 8D, when the edge corresponding to the edge A1 is mistakenly recognized as the edge B2 having the same shape instead of B1, the movement from A1 to A1 is actually performed from A1 to B1. It becomes recognition that the movement to B2, and a situation occurs in which the movement amount is reduced from the true value. Therefore, a method for extracting an appropriate edge from a group of edges that overlap while shifting in the moving direction is required.

  As described below, the present invention solves the above problem by performing a two-stage narrowing-down procedure in optical flow estimation.

  That is, after images of the first frame and the next second frame are obtained, as a first procedure, a predetermined number of images in a predetermined direction (horizontal direction, vertical direction, etc.) with respect to the image luminance value The adjacent pixels and luminance values are averaged, and a low-pass filter process for removing high frequency components is performed.

For example, an example of averaging luminance values in the horizontal direction is as follows.
When the luminance value of the pixel in the x-th column of the original data is I ₁ (x), the luminance value I ₂ (x) obtained after performing the low-pass filter process averages a predetermined number of pixels before and after that. Calculated by performing processing. For example, in the case of averaging using the front and rear three pixels, the luminance value I ₂ (x) is the luminance I ₁ (x) in a total of seven pixels: three in front of the target pixel, three in the back, and one target pixel itself. Is expressed as the following formula (2).
I ₂ (x) = {I ₁ (x−3) + I ₁ (x−2) + I ₁ (x−1) + I ₁ (x) + I ₁ (x + 1) + I ₁ (x + 2) + I ₁ (x + 3)} / (3 + 3 + 1) (2)

Although this example has been described with an average of a total of seven pixel signals of three before and after, it is not limited to the number of pixels, and low-pass filter processing is performed using the average of a plurality of adjacent pixel signals. As long as the method is used, the present invention can be applied to any number of pixel signals.
In the above-described operation, in the claims, for the generated first or second image signal, the luminance value of each pixel unit is averaged with the luminance values of a predetermined number of other pixel units adjacent thereto. This corresponds to “the operation of generating the third or fourth image signal”. The means capable of removing high frequency components by the low-pass filter corresponds to “first and second averaging means” in the claims.

The luminance data I ₂ (x) (third) obtained by performing low-pass filter processing on the luminance data I ₁ (x) (first and second image signals) of the moving image obtained by the divided exposure of FIG. 9A and 9B show examples of images of the fourth and fourth image signals. According to these figures, high-frequency components are removed, and luminance data I ₂ (x) in which the contour of the image is blurred is obtained. In this example, simple averaging is used. However, the present invention is not limited to this. Weighted averaging by multiplying the luminance value of each pixel by a weighting coefficient may be performed, and high-frequency component removal using averaging is performed. Any process can be applied to the present invention.

The operation for obtaining the luminance I ₂ (x) is described in the claims as “by averaging the luminance value of each pixel portion with the luminance values of a predetermined number of other pixel portions adjacent thereto. This corresponds to the “operation for generating the third or fourth image signal”.

  FIG. 9C is a graph showing the relationship between the luminance I (x) and the x-direction position (which column of pixels in the x-direction) of the pixels in the row α ′ ″ shown in the figure. is there. The solid line portion in FIG. 8C indicates the luminance value of the pixel on the line α ′ ″ in the first frame, and the alternate long and short dash line portion indicates the luminance of the pixel on the line α ′ ″ in the second frame. Indicates the value. The luminance Ia shown on the vertical axis indicates the luminance of the subject S (Shinkansen) vehicle body, and Ib indicates the luminance of the background. In this example, Ia> Ib.

  9A and 9B, the portion where the outline is blurred by the low-pass filter processing is as follows in the luminance graph of FIG. 9C. That is, as is clear from FIG. 9C, the corresponding contour portion has changed from a staircase shape indicated by a dotted line before low-pass filter processing to a graph that rises substantially linearly indicated by a solid line and a one-dot chain line.

FIG. 9D is a graph showing the relationship between the position in the x direction and the luminance gradient value K ₂ (x) in the pixels in the row α ′ ″ shown in the drawing. As in the case of the luminance graph, the solid line portion indicates the gradient value of the luminance of the pixel on the line α ′ ″ in the first frame, and the alternate long and short dash line portion indicates the line α ′ ″ in the second frame. The gradient value of the luminance of the upper pixel is shown.

At this time, as shown in FIG. 9D, a portion having a differential value greater than a certain value exists in a wide range in the x-axis direction position. Therefore, precise position measurement is not possible. Therefore, a pixel group in which the gradient value in the horizontal direction (x direction) of the luminance I ₂ (x) data is greater than or equal to a predetermined value is extracted, and the center position is obtained. Let G1 be the center position of a pixel group having a gradient value equal to or greater than a certain value in the first frame, and G2 be the center position of a pixel group having a gradient value equal to or greater than a certain value in the second frame.

After _{obtaining the} center position (first center) G1 in the first frame and the center position (second center) G2 in the second frame, the approximate optical flow F _G1G2 is obtained by the block matching method. The schematic optical flow F _G1G2 indicates the moving direction and moving amount of the corresponding center position in the first frame and the second frame. The general optical flow F _G1G2 is an optical flow _obtained with a low-pass filter in the state in which the image contour is unclear, as in the case of FIG. 6 described above, and shows the approximate amount of movement and direction, and the contour position precisely. Has not been estimated.

  However, the gist of the present invention is to perform block matching once again by combining both the approximate movement vector and a plurality of sharp outlines obtained by performing the exposure a plurality of times in the above-described one photographing cycle. It is to obtain a precise contour position.

That is, when the approximate optical flow F _G1G2 is obtained, the data of the luminance I ₁ (x) before applying the low-pass filter in FIG. 8A to FIG. Return. And the operation | work which pinpoints the position of edge A1-A4, B1-B4 of the image which shows a clear outline is performed.

Here, attention is again paid to the gradient value of the luminance shown in FIG. Normally, in block matching in which partial feature point matching is determined, when contours of the same shape are arranged at equal intervals, matching feature points are arranged, for example, an edge corresponding to the edge A1 of the first frame. Cannot determine any of B1 to B4 in the second frame. Therefore, a plurality of optical flows with the same feature point are detected at the same number as the number of edges, and at this time, it cannot be determined which is the correct optical flow. The plurality of detected optical flow groups are hereinafter referred to as optical flow candidates. However, since the optical flow candidate is based on a sharply represented contour, if the optical flow candidate is not selected _erroneously using the approximate optical flow F _G1G2 , a precise optical flow can be obtained as a result.

That is, as described above, in the procedure of the present invention, the approximate optical flow F _G1G2 that is an approximate movement vector is obtained from the data of the luminance I2 (x) to which the low-pass filter has been previously applied. Therefore, it can be estimated that there is an edge corresponding to the edge A1 in the second frame in the vicinity of a point shifted from the edge A1 by the approximate optical flow _FG1G2 . Therefore, block matching is started from a point shifted from the edge A1 by the approximate optical flow F _G1G2 , and it is determined that the contour closest to the starting point is the corresponding edge among the contours having the matching feature points. Thus, the detection accuracy can be increased.

FIG. 10 shows a flowchart of the optical flow estimation procedure.
When the flow starts, first, in step S001, a moving image is acquired by performing exposure, transfer, and accumulation in a plurality of times during one shooting period. That is, the luminance I ₁ (x) that is the first image signal is acquired from the first imaging period (first frame), and the second image signal is acquired from the second imaging period (second frame). The luminance I ₁ (x) is acquired. Next, in step S101, the first and second image signals obtained in the previous step are subjected to low-pass filter processing by averaging with the predetermined number of adjacent pixel signals, respectively. 4 image signals (luminance I ₂ (x) data) are generated.

In step S102, the third and fourth image signals generated in the previous step are compared with each other, and an approximate optical flow F _G1G2 that is an approximate optical flow is calculated. In step S201, the first and second image signals I ₁ (x) obtained in step S001 are compared with each other, and the above-described optical flow candidates are selected. Here, with respect to the first and second image signals, a plurality of the same contours are arranged at the same interval. Therefore, when feature points are compared, a plurality of optical flow candidates with matching features are listed.

Finally, in step S202, the closest optical flow F _G1G2 obtained in step S102 is selected as the final optical flow from the plurality of optical flow candidates selected in the previous step.
Thus, the final optical flow can be accurately estimated, and the flow of the optical flow estimation procedure is completed.

  Note that the above-described optical flow estimation procedure is realized by “optical flow candidate calculation means, general optical flow calculation means, and optical flow estimation means” in the claims.

The operation for selecting the optical flow candidates is “operation for calculating a plurality of optical flow candidates that are vectors indicating the direction and amount of movement of the subject during the first and second imaging periods” in the claims. Correspond. The operation for _obtaining the approximate optical flow F _G1G2 is “the operation for calculating the approximate optical flow that is a vector indicating the approximate direction and amount of movement of the subject during the first and second imaging periods” in the claims. Corresponding to Further, the operation for obtaining the true optical flow corresponds to “operation for estimating one optical flow candidate closest to the approximate optical flow among the plurality of optical flow candidates as the final optical flow” in the claims.

As described above, in the present invention, as a first procedure, a movement vector having a high accuracy is obtained from image data subjected to a low-pass filter. As a second procedure, a precise movement vector is narrowed down from the original image data based on the movement vector. The present invention performs such two-stage narrowing. By such a procedure, the edge corresponding to the transition from the first frame to the second frame can be accurately selected from a plurality of edges appearing in the image by the divided exposure, and even when the divided exposure is used. _Allows precise estimation of the true optical flow F _A1B1 .

<Non-uniform division interval>
The accuracy of selecting the corresponding contours between the preceding and following frames can be improved by using the following method.

  In FIG. 4, the time intervals between the exposure timings 602-1, 602-2, 602-3, and 602-4 are assumed to be uniform when performing exposure four times during one photographing period. As a result, as shown in FIGS. 8A to 8D, an image is obtained in which the outline of the subject S is shifted by a uniform width in the traveling direction, and the contour of the same shape is shifted by a uniform width. Therefore, a plurality of candidates having the same feature point are obtained, and there is a possibility of selecting an incorrect contour.

  Therefore, as shown in FIG. 11, when exposure is performed four times during one photographing period, time intervals T12, T23, T34 between the exposure timings 702-1, 702-2, 702-3, and 702-4 are set to be equal. A configuration may be adopted in which not uniform intervals but non-uniform intervals are employed. In this embodiment, T23 <T12 <T34, but the present invention is not limited to this. These time intervals may be at least two time intervals different from each other.

  Images obtained in this example are shown in FIGS. 12 (A) and 12 (B). When the illustrated subject S moves at a constant speed, the contour of the subject S is an image in which quadruple contours are arranged at non-uniform intervals while shifting in the traveling direction, such as C1 to C4 and D1 to D4.

  FIG. 12C is a graph showing the relationship between the luminance I (x) and the x-direction position (which column is the pixel in the x-direction) in the pixel in the row α ″ ″ shown in the figure. It is. The solid line portion in FIG. 12C indicates the luminance value of the pixel on the line α ″ ″ in the first frame, and the alternate long and short dash line portion indicates the pixel value on the line α ″ ″ in the second frame. Indicates the luminance value.

  At this time, the contour portion of the subject S in FIGS. 12A and 12B becomes a sharp contour portion that is overlapped four times while being displaced at non-uniform intervals. That is, in the graph of the luminance value I (x) in FIG. 12C, a corresponding step becomes a staircase graph in which the portions suddenly rise at non-uniform intervals.

  FIG. 12D is a graph showing the relationship between the x-direction position and the luminance gradient value K (x) in the pixels in the row of the line α ′ ″ ″. As in the case of the luminance graph, the solid line portion indicates the gradient value of the luminance of the pixel on the line α ″ ″ in the first frame, and the alternate long and short dash line portion indicates the line α ″ in the second frame. '' Indicates the gradient value of the luminance of the pixel above.

  12A and 12B, the sharply contoured portions that are quadruply overlapped with each other are unevenly distributed at the four locations in the brightness gradient value in FIG. 12D. Is shifted in the x-axis direction. Further, the location of the contour rises locally to a large gradient value (denoted as K3) only in a very narrow x-axis range. It can also be seen that it exists in each of the first frame and the second frame.

  At this time, since the portion where the luminance gradient value has a certain value or more is a very narrow range, the positions in the x-axis direction of the four contours arranged at non-uniform intervals in each frame can be accurately obtained. . Here, unlike the case of FIG. 8D, the gist of the present invention is that the intervals d12, d23, d34 of the edges C1-C2, C2-C3, C3-C4 are different from each other even if the outline is the same shape. Note that The same applies to the intervals between the edges D1 to D2, D2 to D3, D3 to D4. At this time, since the lines are unevenly arranged, there is a difference in the relative distance between the feature points of the contour, and it is possible to identify which edge the corresponding edge corresponds to. Therefore, by using this configuration, the corresponding edge is not erroneously selected even in block matching. For this reason, even if exposure, transfer, and accumulation are performed in a plurality of times in one shooting cycle, it is possible to accurately estimate the optical flow without erroneous estimation.

  Further, since exposure is performed a plurality of times with non-uniform intervals, in the gradient value graph of FIG. 12D, the left edge of the raised portions C1 to C4 indicating the contour of the subject S of the first frame and the second frame The space | interval opened between the right ends of the protruding parts D1-D4 which show an outline is restrained small. Therefore, even for a user who views a moving image, the outline of the Shinkansen in the first frame and the contour of the subject S in the second frame are almost continuous, and the high-quality moving image has no sense of disparity.

  As described above, by performing a plurality of exposure timings in one shooting cycle at unequal time intervals, it is possible to realize a configuration capable of acquiring a high-quality moving image while calculating a precise optical flow. In addition, performing the plurality of exposure timings at unequal time intervals means that “in each of the first and second imaging cycles, n signal charge transfers start n times in each of the first and second imaging periods”. Are executed such that the time intervals of the transfer timing are different from each other.

<Time interval between signal charge transfer timings>
It is desirable to set the number of exposure / transmission / accumulation times so that the time interval between signal charge transfer timings, that is, the time interval between multiple exposure timings in FIG. This is because it is difficult to read the change in the time when the human reflection speed is generally 1/120 seconds or less, and by setting the time interval below this time, the sense of disparity is not noticeable and a high-quality moving image can be provided. Because.

  Therefore, as an example, if a movie is shot with an exposure time of 1/30 seconds for one frame, the exposure is performed by dividing the frame at least four times within one frame, and the time interval between the four exposures is 1/120 seconds or less. To be. If the exposure time of one frame is 1/60 seconds, the exposure time is divided at least twice so that the time interval between the two exposures is 1/120 seconds or less.

  With the above configuration, it is possible to realize a configuration in which the number of times of multiple exposures is optimized and a high-quality moving image that does not feel a sense of disparity can be acquired.

  The operation of setting the number of times so that the time interval is 1/120 second or less is as follows: “The time interval of the transfer timing of n starting n times of signal charge transfer is 1/120 second. This corresponds to the operation of increasing / decreasing the value of n so as to be as follows.

<Modification>
In the imaging apparatus described above, the system control CPU (control unit) 178 in FIG. 2 may include a moving body determination unit that determines whether or not the subject is a moving body during moving image shooting. In this case, when the moving object determination unit determines that the subject is a moving object, the optical flow estimation procedure shown in FIG. 10 is executed. Thereby, a highly accurate optical flow can be obtained during moving image shooting.

  Further, the system control CPU (control means) 178 in FIG. 2 may include speed determination means for determining the speed of the subject (moving object) or the camera shake speed during moving image shooting. In this case, the optical flow estimation procedure shown in FIG. 10 is executed when the speed determination means determines that the subject speed or the camera shake speed is equal to or greater than a predetermined value. Further, the system control CPU 178 may perform association between the subject speed or camera shake speed detected by the speed determination unit and the number of times signal charges are transferred during one shooting period or the time interval of transfer timing. . This association is stored as a lookup table (LUT). For example, the system control CPU 178 increases the number of signal charge transfers or shortens the time interval of transfer timing as the subject speed or camera shake speed increases. Thereby, a highly accurate optical flow can be obtained during moving image shooting. In the above, the number of signal charge transfers may be the number of exposures, and the transfer timing may be the exposure timing.

  Further, the system control CPU (control means) 178 in FIG. 2 performs the optical flow estimation procedure shown in FIG. 10 based on an operation by the user (photographer), for example, a transition instruction to the high follow-up mode or the image stabilization mode. It may be determined whether or not to execute. That is, the system control CPU 178 includes instruction determination means for determining whether or not a predetermined instruction has been received, and executes the optical flow estimation procedure shown in FIG. 10 when the instruction determination means determines that the predetermined instruction has been received. . Thereby, a highly accurate optical flow can be obtained during moving image shooting.

  Further, the system control CPU (control means) 178 in FIG. 2 executes an exposure / transfer / accumulation sequence a plurality of times during one photographing period. Therefore, the system control CPU 178 determines that the time interval and the transfer timing for starting the signal charge transfer (or the exposure timing for starting the exposure) in one shooting cycle are less than half the one shooting cycle. Controls the number of signal charge transfers. For example, when one shooting period is 1/120 seconds, the system control CPU 178 sets the number of signal charge transfers to 2 and sets the time interval of transfer timing to 1/240 seconds or less. Thereby, a highly accurate optical flow can be obtained during moving image shooting.

  In the imaging apparatus described above, the function executed by the system control CPU (control unit) 178 in FIG. 2 may be realized by hardware, software, or a combination thereof. May be. Whether these functions are realized by hardware, software, or a combination thereof depends on the environment in which the above-described imaging device is used or the above-described imaging device. Depends on design constraints etc. When the function of the system control CPU 178 is executed by software, a program for realizing the function may be provided from the host device to the above-described imaging device via a network. Further, the program may be provided to the above-described imaging apparatus from a storage medium (including a removable memory such as a memory card or a CD) in which the program is stored. In this case, the system control CPU 178 executes a program (software) acquired from a network or a recording medium, and executes processing according to the present invention. The system control CPU 178 can be replaced with a control device having a function equivalent to the system control CPU 178, for example, a microcomputer, a processor, an ASIC (application specific integrated circuit), or the like.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

500 Photoelectric Conversion Unit 507A First Signal Holding Unit 300, 301 Pixel Unit 184 Imaging Element 178, 189 Control Unit 151 Imaging Device

Claims (12)

An imaging device having a plurality of pixel portions and each pixel portion including a photoelectric conversion unit and a signal holding unit;
Control means for controlling the image sensor,
The control means includes
Based on the signal charges accumulated in the signal holding unit by n signal charge transfers (n is a natural number of 2 or more) from the photoelectric conversion unit to the signal holding unit during the first imaging cycle, First image signal generation means for generating an image signal;
A second image signal is generated based on the signal charge accumulated in the signal holding unit by n times of signal charge transfer from the photoelectric conversion unit to the signal holding unit during the second imaging cycle. Image signal generating means;
A first image signal is generated by averaging the luminance value of each pixel unit with the luminance values of a predetermined number of other pixel units adjacent to the generated first image signal. Means of averaging,
A second image signal is generated by averaging the luminance value of each pixel unit with the luminance value of a predetermined number of other pixel units adjacent to the generated second image signal. Means of averaging,
An optical flow candidate calculation that calculates a plurality of optical flow candidates that are vectors indicating the direction and amount of movement of the subject between the first and second imaging cycles by comparing the first and second image signals. Means,
A schematic optical flow that calculates a schematic optical flow that is a vector indicating a general direction and amount of movement of the subject during the first and second imaging periods by comparing the third and fourth image signals. A calculation means;
Optical flow estimation means for estimating one optical flow candidate closest to the approximate optical flow among the plurality of optical flow candidates as a final optical flow;
An imaging apparatus comprising: An imaging device having a plurality of pixel portions and each pixel portion including a photoelectric conversion unit and a signal holding unit;
Control means for controlling the image sensor,
The control means includes
Based on the signal charges accumulated in the signal holding unit by n signal charge transfers (n is a natural number of 2 or more) from the photoelectric conversion unit to the signal holding unit during the first imaging cycle, First image signal generation means for generating an image signal;
A second image signal is generated based on the signal charge accumulated in the signal holding unit by n times of signal charge transfer from the photoelectric conversion unit to the signal holding unit during the second imaging cycle. Image signal generating means;
Optical flow estimation means for estimating a final optical flow that is a vector indicating the direction and amount of movement of the subject between the first and second imaging periods by comparing the first and second image signals. And comprising
In each of the first and second imaging periods, the n signal charge transfers are executed such that the time intervals of the n transfer timings that start them are different from each other.
Imaging device.   The control means sets the value of n so that the time interval of the transfer timing of n starting the n times of signal charge transfer is 1/120 second or less in each of the first and second imaging periods. The imaging device according to claim 1, wherein the imaging device is increased or decreased.   The control means further comprises a moving object determining means for determining whether or not the subject is a moving object, and estimates the final optical flow when the moving object determining means determines that the subject is a moving object. The imaging device according to any one of 1 to 3.   The control means further includes a speed determination means for determining the speed of the subject or the camera shake speed, and when the speed determination means determines that the speed of the subject or the camera shake speed is equal to or greater than a predetermined value, the final control means The imaging device according to claim 1, wherein the optical flow is estimated.   The control means further includes instruction determination means for determining whether or not a predetermined instruction has been received, and estimates the final optical flow when the instruction determination means determines that the predetermined instruction has been received. The imaging device according to any one of claims 1 to 5.   In the first or second imaging cycle, the control means has a time interval of n transfer timings at which the n times of signal charge transfer is started to be at least half of the first or second imaging cycle. The imaging device according to claim 1, wherein the time interval and the value of n are controlled as described above.   The control means further includes a speed determination means for determining the speed of the subject or the camera shake speed, and increases the value of n as the speed of the subject or the camera shake speed detected by the speed determination means increases. The imaging apparatus according to claim 1, wherein a time interval of n transfer timings at which the n times of signal charge transfer is started is shortened. The first image signal is subjected to n first exposures during the first imaging period, and signal charges generated in the photoelectric conversion unit by each first exposure are transferred to the signal holding unit. Generated by
The second image signal is subjected to n second exposures during the second imaging period, and the signal charges generated in the photoelectric conversion unit by each second exposure are transferred to the signal holding unit. Generated by
The imaging device according to any one of claims 1 to 8. In each of the first and second imaging periods, the n signal charge transfers are executed such that the time intervals of the n transfer timings that start them are equal to each other.
The imaging device according to claim 1. An imaging apparatus control method comprising: an imaging device having a plurality of pixel units, and each pixel unit including a photoelectric conversion unit and a signal holding unit; and a control unit that controls the imaging device,
Based on the signal charges accumulated in the signal holding unit by n signal charge transfers (n is a natural number of 2 or more) from the photoelectric conversion unit to the signal holding unit during the first imaging cycle, Generate an image signal,
Based on the signal charges accumulated in the signal holding unit by n times of signal charge transfer from the photoelectric conversion unit to the signal holding unit during the second imaging cycle, a second image signal is generated,
A third image signal is generated by averaging the luminance value of each pixel unit with the luminance value of a predetermined number of other pixel units adjacent to the generated first image signal,
A fourth image signal is generated by averaging the luminance value of each pixel unit with the luminance value of a predetermined number of other pixel units adjacent to the generated second image signal,
By comparing the first and second image signals, a plurality of optical flow candidates that are vectors indicating directions and amounts of movement of the subject between the first and second imaging cycles are calculated,
By comparing the third and fourth image signals, a schematic optical flow that is a vector indicating a general direction and amount of movement of the subject during the first and second imaging cycles is calculated,
One optical flow candidate closest to the approximate optical flow among the plurality of optical flow candidates is estimated as a final optical flow.
Control method. An imaging apparatus control method comprising: an imaging device having a plurality of pixel units, and each pixel unit including a photoelectric conversion unit and a signal holding unit; and a control unit that controls the imaging device,
Based on the signal charges accumulated in the signal holding unit by n signal charge transfers (n is a natural number of 2 or more) from the photoelectric conversion unit to the signal holding unit during the first imaging cycle, Generate an image signal,
Based on the signal charges accumulated in the signal holding unit by n times of signal charge transfer from the photoelectric conversion unit to the signal holding unit during the second imaging cycle, a second image signal is generated,
By comparing the first and second image signals, a final optical flow that is a vector indicating the direction and amount of movement of the subject between the first and second imaging periods is estimated.
However, in each of the first and second imaging periods, the n times of signal charge transfer are executed such that the time intervals of n transfer timings for starting them differ from each other.
Control method.

Images