JP2018148444A - Imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a rolling shutter type imaging apparatus capable of sufficiently resolving distortion of an image of a moving subject while reducing load of arithmetic processing.SOLUTION: An imaging apparatus 11 includes an imaging device for successively transferring electric charge in each pixel 14 synchronously with a clock signal. A control unit 21 selects a first operation mode for transferring electric charge synchronously with a clock signal of a first frequency and completing transfer of electric charge for one frame for first time and a second operation mode for transferring electric charge synchronously with a clock signal of a second frequency higher than the first frequency and completing transfer of electric charge for one frame for second time shorter than the first time. In operation of the second operation mode, time from start of transfer of electric charge for one frame up to start of transfer of next one frame is set as the first time.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a rolling shutter type imaging apparatus.

  A rolling shutter type imaging apparatus sequentially exposes a large number of pixels on an imaging element in time series. A time difference occurs in exposure. As a result, the image of the moving subject imaged is distorted. In Patent Document 1, a continuous two-frame image is generated. A subject moving in the horizontal direction is detected from the two-frame image. The detected moving subject is subjected to correction processing according to the shift in the imaging timing, and distortion is reduced.

JP2015-46812A

  The imaging apparatus described in Patent Document 1 is complicated because it calculates a moving amount of a moving subject, corrects the distortion of the subject according to the moving amount, and synthesizes it with a background image that does not move. An arithmetic circuit is required. Also, distortion other than the subject moving in the horizontal direction cannot be corrected.

  An object of the present invention is to provide a rolling shutter type imaging apparatus that can reduce distortion of an image of a subject moving in an arbitrary direction by a simple method without using a complicated arithmetic circuit.

  According to one embodiment of the present invention, an imaging apparatus includes an imaging device that sequentially transfers charges for each pixel in synchronization with a clock signal, and transfers charges in synchronization with a clock signal having a first frequency. The charge is transferred in synchronization with the first operation mode in which the transfer of the charge for the frame is completed in the first time and the clock signal having the second frequency higher than the first frequency, and the transfer of the charge for the one frame is A controller for selecting a second operation mode to be completed in a second time shorter than the first time, and when operating in the second operation mode, from the start of the transfer of one frame of charge, An imaging apparatus is provided in which the time until the start of transfer is the first time.

  In the first operation mode, the image sensor sequentially transfers charges for each pixel in synchronization with the first frequency clock signal. In the second operation mode, the image sensor charges for each pixel in synchronization with the second frequency clock signal. Transfer sequentially. In the second operation mode, charge transfer is performed based on a second frequency that is higher than the first frequency. Therefore, in which direction (including the horizontal direction and the vertical direction) the image of the subject in the image captured by the image sensor. Even if is moved, the exposure time difference between the pixels is reduced and the distortion of the image of the moving object is reduced as compared with the case of operating only in the first operation mode.

  Moreover, even in the second operation mode, the first time is secured until the start of the transfer of the next one frame for each individual charge (that is, the exposure time for one frame time can be secured). The exposure time can be secured as in the operation mode. A decrease in sensitivity can be avoided as compared with a case where the frame rate is simply set to the second frequency. For example, when the frame rate is simply increased from 60 fps to 120 fps, the difference in exposure time between the pixels is reduced, and the distortion of the image of the moving subject is reduced. However, since the exposure time is reduced by half, the sensitivity is reduced by half.

  The imaging apparatus may further include a movement detection unit that detects movement of a subject image in an image captured by the imaging element. The movement detection unit recognizes the shape of the image of the moving subject, and calculates the speed by detecting the movement amount of the image of the subject having the same shape. At this time, when the movement detection unit does not detect the movement of the subject image, the control unit selects the first operation mode, and when the movement detection unit detects the movement of the subject image, The second operation mode may be selected.

  Unless the movement of the image of the subject is detected in the image, the image sensor transfers the charge for each pixel in synchronization with the clock signal of the first frequency. Although there is a difference in the exposure time for each pixel, the image of the subject is stationary, so image distortion can be avoided. Since the second frequency is set by the image sensor only when movement of the subject image is detected, distortion of the moving subject image is reduced, and is lower than the second frequency when the subject image is stationary. Since the first frequency is set, the power consumption is reduced compared to the case where the second frequency is always set.

  The control unit may change the second frequency according to a moving speed of the image of the subject in the image detected by the movement detection unit. In an image sensor that transfers charges for each pixel in synchronization with a clock signal, the amount of distortion of a moving subject depends on the moving speed of the subject image. Therefore, if the second frequency is set according to the moving speed of the moving subject image, the second frequency is not excessively increased, and an increase in power consumption can be appropriately suppressed.

  The imaging apparatus may further include a height detection unit that detects a height of the image of the subject in an image captured by the imaging element. In an image sensor that transfers charges for each pixel in synchronization with a clock signal, the amount of distortion of a moving subject depends on the height of the subject. Therefore, if the second frequency is set based on the moving speed of the moving subject and the height, the second frequency is not excessively increased, and an increase in power consumption can be appropriately suppressed.

  The control unit may increase the second frequency by a predetermined time constant when the movement of the subject is detected. The exposure time of the pixels on the image sensor is the time from the completion of charge transfer in a certain frame to the start of charge transfer in the next frame. Therefore, when switching from the first frequency to the second frequency that deviates from the first frequency, for example, in one frame, the time from the completion of charge transfer to the start of charge transfer in the next frame is shortened. Time will be shortened. The image of one frame after switching becomes dark. This phenomenon can be reduced by gradually increasing the second frequency. Details will be described later.

  The control unit may set a gain for each individual pixel according to the second frequency instead of the method described above. Even if there is a difference in exposure time for each pixel, the control unit adjusts the gain for each pixel according to the difference in exposure time. Also in this method, the phenomenon that the image becomes dark can be reduced.

  As described above, according to the disclosed imaging apparatus, it is possible to provide a rolling shutter type imaging apparatus that can sufficiently eliminate distortion of an image of a moving subject without using a complicated arithmetic circuit.

1 is a conceptual diagram schematically showing a configuration of an imaging apparatus according to an embodiment of the present invention. It is a conceptual diagram which shows time distribution of the image of a to-be-photographed object within the flame | frame specified by 1st frequency. It is a conceptual diagram which shows time distribution of the image of a to-be-photographed object within a flame | frame when switched to a 2nd frequency. It is a figure which shows notionally the shortening of the exposure time which arises when switching from a 1st frequency to a 2nd frequency.

  Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

  FIG. 1 schematically shows a configuration of an imaging apparatus 11 according to an embodiment of the present invention. The imaging device 11 includes an imaging element 12 that outputs an image signal according to a subject. The image sensor 12 includes pixels 14 arranged in a matrix following the screen of the display device 13. The number of pixels in the horizontal direction (lateral direction) is W pixels, and the number of pixels in the vertical direction (vertical direction) is H pixels. In each pixel 14, charges are accumulated for each color of RGB (red, green, and blue) according to the exposure time. Here, the pixels 14 are arranged in an array of W columns in the horizontal direction and H columns in the vertical direction (W> H horizontally long).

  The image sensor 12 operates by a so-called rolling shutter system. That is, the charges accumulated in the individual pixels 14 are transferred in synchronization with the clock signal. The charge transfer is carried out in order from the pixel “0” in the 0th row and the 0th column in the horizontal direction in the pixel “1”, the pixel “2”,. The pixel “W” from the pixel “W” in the first row and the zeroth column is executed from the pixel “W + 1”... To the pixel “2W−1” and scanned in the horizontal direction while shifting by one row in the vertical direction. W ”... Implemented with pixel“ H × W−1 ”. Thus, an image signal (video signal) for one frame is output. Thereafter, the charge transfer returns to the pixel “0” in the 0th row and the 0th column. Compared with the global shutter system, such a rolling shutter system does not require a storage circuit such as a buffer for holding the charge after exposure, and can be manufactured at a low cost.

  In each pixel 14, the charge accumulated by the reset signal from the control unit 21 becomes zero, and thereafter, charge accumulation (exposure) is performed for a certain period, and the charge is finally transferred. The exposure time is from the input of the reset signal to the pixel in a certain frame until the start of the transfer of the charge of the pixel in the next frame. Charges corresponding to the number of photons entering each pixel from the input of the reset signal are accumulated.

  The imaging device 11 includes a gain amplifier 15 connected to the imaging device 12. The gain amplifier 15 receives a video signal from the image sensor 12. The gain amplifier 15 amplifies the video signal according to the gain parameter supplied from the control unit 21. The image sensor 12 and the gain amplifier 15 may be configured as, for example, a CMOS (complementary metal oxide semiconductor) sensor. In addition, a CCD (charge coupled device) sensor or other image sensor may be used instead of the CMOS image sensor.

  The imaging device 11 includes a clock supply unit 16 connected to the imaging element 12. The clock supply unit 16 supplies a clock signal to the image sensor 12 at a frequency specified by the control signal supplied from the control unit 21. The clock supply unit 16 can be constituted by, for example, a frequency divider circuit or a multiplier circuit combined with a crystal resonator. Based on the clock signal supplied from the clock supply unit 16, the exposure start, exposure end, and reset timing of the pixel 14 on the image sensor 12 are determined. In addition, the clock supply unit 16 may be composed of, for example, a voltage controlled crystal oscillator (VCXO). In this case, a voltage is applied to the clock supply unit 11 as a control signal. The clock supply unit 16 outputs a clock signal having a frequency set according to the magnitude of the voltage.

  The imaging device 11 includes an image signal processor 17 connected to the gain amplifier 15. The image signal processor 17 receives a video signal from the gain amplifier 15. The image signal processor 17 performs gamma correction processing and other image processing on the received video signal and outputs the processed video signal. In the imaging device 11, a display device 13 and a recording device can be connected to the image signal processor 17. For example, a video is displayed on the screen of the display device 13 based on the video signal output from the image signal processor 17.

  The imaging device 11 includes a movement detection unit 19 connected to the image signal processor 17 and a height detection unit 20 connected to the subsequent stage. The height detection unit 20 receives the shape and movement vector of the subject from the moving body detection unit 19 as data, and calculates the height difference between the uppermost position and the lowermost position on the image sensor 12 of the subject. Detection is performed. The movement detector 19 receives a video signal from the image signal processor 17. The movement detection unit 19 detects the movement of the subject image in the image captured by the image sensor 12. In the detection, the movement detection unit 19 compares the images of two frames before and after. The two frames may be frames that are temporally continuous. The frame image may be temporarily stored in a storage device such as a memory. For example, the memory may be built in a microprocessor unit constituting the movement detection unit 19. The movement detector 19 detects a vertical movement speed and a horizontal movement speed based on the movement vector.

  In detecting the movement vector of the movement detection unit 19 and the height of the height detection unit 20 connected to the subsequent stage, for example, a plurality of feature points are detected in the image. In general, a feature point is a point where both a horizontal outline and a vertical outline exist. That is, the corner of the object. The movement detection unit 19 compares the image around the feature point with the image of the previous frame and searches for a similar part. If the similar part is the same in the current frame and the previous frame, the feature point is regarded as not moving. If the similar parts are different, this feature point is considered to have moved from the previous frame location. Thus, the movement vector of the moving image is specified. The length of the movement vector is regarded as the speed E [pixel / 1 frame time]. The height detection unit 20 specifies the height F [pixel] of the moving image based on the uppermost position and the lowermost position of the feature points. Here, when there are a plurality of feature points that move at the same speed in the same direction as this feature point, a rectangle including all of the feature points is regarded as a “moving image”. Further, the movement vector and the height are used when determining the second frequency, and the image itself does not collapse even if the accuracy is not high.

  The imaging device 11 includes a control unit 21 connected to the clock supply unit 16 and the gain amplifier 15. The control unit 21 transmits a control signal designating the frequency of the clock signal to the clock supply unit 16. In the control signal, for example, a first frequency and a second frequency higher than the first frequency are designated. The control unit 21 defines one frame time according to the number of pixels with the number of clocks of the first frequency. The higher the speed E and the higher the height F, the higher the second frequency is specified, thereby reducing image distortion. The control unit 21 sets a non-signal period (blanking period) within one frame time after the transfer of charges for one frame is completed.

  After switching from the first frequency to a predetermined second frequency higher than the first frequency, the control unit 21 may raise the second frequency from the predetermined second frequency with a predetermined time constant. The frequency of the clock signal changes gradually. As a result, the shortening of the exposure time is alleviated, and the phenomenon that the screen becomes dark suddenly is reduced. This principle will be described in detail. FIG. 4 shows the exposure time associated with switching from the first frequency to the second frequency. The maximum exposure time T1 of the first pixel at the upper end of the screen is the time from the transfer of the charge of the pixel to the start of the transfer of the same pixel in the next frame. Similarly, the exposure time T2 of the upper part of the image, the maximum exposure time T3 of the lower part of the image, and the exposure time T4 of the last pixel at the lower end of the screen are defined. The 0th frame operates at the first frequency, and the first frame operates at the second frequency. Since the second frequency is higher than the first frequency, T2, T3, and T4 are shorter than T1 defined above, and T1> T2> T3> T4. That is, the exposure time becomes shorter toward the bottom of the image, and the image becomes darker. This phenomenon becomes more remarkable as the difference between T1 and T2, T3, and T4 increases. Therefore, by gradually increasing the second frequency (with a predetermined time constant), it becomes possible to suppress the difference between T1, T2, T3, and T4, and the phenomenon that the screen becomes dark suddenly is reduced.

  Instead of the above operation, a phenomenon in which an image suddenly becomes dark by amplification of an image signal may be avoided. The control unit 21 transmits a parameter signal that specifies the amplification factor to the gain amplifier 15. The value of the parameter signal that sets the amplification factor may be calculated by the control unit 21 in inverse proportion to the exposure time of each pixel 14 that is transmitted when the control unit 21 controls the image sensor 12.

  FIG. 2 shows a state where the image A exists in the image. This image is picked up by the image pickup device 12 that operates in a rolling shutter system. The frame rate is R [frame / sec], the number of pixels in the vertical direction of the image is H [pixel], and the height of the image A is F [pixel]. The height F [pixel] of the image A is calculated by the height detection unit 20 by the method described above. At this time, the transfer time difference between the uppermost line and the lowermost line is F / (R × H) seconds. Also, assuming that the velocity of the subject image calculated by the above-described method in the movement detector 19 is K [pixel / sec], the shift amount G = time difference (F / (R × H)) between the top and bottom of the image. X Speed K = F x K / (R x H) [pixel] occurs. The shift amount G becomes the distortion of the image A. In suppressing the distortion, the control unit 21 transmits a control signal for setting the second frequency instead of the first frequency to the clock supply unit 16. Thus, the second frequency is changed. As shown in FIG. 3, the clock signal is increased from the first frequency to the second frequency according to the magnification L set by the control unit 21. As a result, the shifting timing is shortened by 1 / L times between the pixels 14 and 14. Thus, the difference in exposure time between the pixels 14 is reduced, so that the distortion of the image of the moving subject is reduced.

  Further, after changing from the first frequency to the second frequency, the value of the second frequency may be changed according to the magnitude of the shift amount G. Specifically, the value is proportional to the deviation amount G. In the change, the control unit 21 may transmit a control signal corresponding to the change to the clock supply unit 16 as described above. Since the shift amount G is proportional to the height F and the speed K of the subject image as described above, the second frequency is changed to be higher as the height F of the subject image increases and the speed K increases. On the contrary, if the height F and the speed K are reduced, the second frequency is changed to be lower. Thereby, distortion can always be reduced for any subject image.

  At this time, the transfer time of all the pixels 14 constituting one frame is shortened to 1 / R × 1 / L seconds. The control unit 21 sets blanking within one frame time after the transfer of charge for one frame is completed. Since the pixel 14 can accumulate charges (perform exposure) also during the blanking period, the exposure time is ensured equivalent to the first frequency. A decrease in sensitivity can be avoided as compared with a case where the frame rate is simply set to the second frequency.

  In the present embodiment, the control unit 21 designates the first frequency with the control signal unless the movement detection unit 19 detects the movement of the subject image. When movement of the subject image is detected by the movement detector 19, the second frequency is designated by the control signal. As long as the movement of the image of the subject is not detected in the image, the image sensor 12 transfers the charge for each pixel in synchronization with the clock signal of the first frequency. Although there is a difference in the exposure time for each pixel, the image of the subject is stationary, so image distortion can be avoided. Since the second frequency is set by the image sensor 12 only when the movement of the subject image is detected in this way, the frequency can be suppressed lower than when the frame rate is simply set to the second frequency, and the power consumption is reduced. It is reduced.

DESCRIPTION OF SYMBOLS 11 ... Imaging device, 12 ... Image sensor, 14 ... Pixel, 16 ... Clock supply part, 19 ... Movement detection part (and height detection part), 21 ... Control part.

Claims (6)

An imaging device including an imaging device that sequentially transfers charges for each pixel in synchronization with a clock signal,
A first operation mode in which charges are transferred in synchronization with a clock signal of the first frequency and transfer of charges for one frame is completed in a first time, and a clock signal of a second frequency higher than the first frequency is synchronized. And a controller that selects a second operation mode for transferring charges and completing transfer of charges for one frame in a second time shorter than the first time,
When operating in the second operation mode, an imaging apparatus characterized in that a time from the start of charge transfer of one frame to the start of transfer of charge of the next one frame is defined as the first time.   The imaging apparatus according to claim 1, further comprising a movement detection unit that detects movement of a subject image in an image captured by the imaging element, wherein the control unit is configured to detect the subject image by the movement detection unit. An imaging apparatus, wherein when the movement is not detected, the first operation mode is selected, and when the movement detection unit detects the movement of the image of the subject, the second operation mode is selected.   The imaging device according to claim 2, wherein the control unit changes the second frequency according to a moving speed of the image of the subject in the image detected by the movement detection unit. apparatus.   The imaging apparatus according to claim 2, further comprising a height detection unit that detects a height of the image of the subject in an image captured by the image sensor, wherein the control unit is configured to detect the image of the moving subject. An imaging apparatus, wherein the second frequency is changed according to height.   5. The imaging apparatus according to claim 3, wherein the control unit increases the second frequency with a predetermined time constant when the movement of the subject is detected.   5. The imaging device according to claim 3, wherein the control unit sets a gain for each individual pixel according to the second frequency. 6.