JP2020031385A - Imaging apparatus - Google Patents

Abstract

To provide an imaging apparatus capable of reliably photographing a subject of interest while suppressing power consumption.SOLUTION: The imaging apparatus includes: a pixel region in which a plurality of pixels is arranged in a matrix; a reading unit for reading a signal from the pixel region; a detection unit for detecting a subject of interest on the basis of a signal read from a first pixel group in the pixel region by the reading unit; a control unit for controlling the reading means to read a signal from a part of a second pixel group different from the first pixel group in the pixel region according to a position of the subject of interest detected by the detection unit.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an imaging device.

  An imaging system including two independent imaging units has been proposed. (See Patent Document 1.) In such an imaging system, a target subject area is determined using an image signal obtained by performing wide-angle imaging using one of the imaging units. Then, based on the determined information, it is possible to record an image having a sufficient resolution at a desired angle of view in an area including the target subject by using the other imaging unit.

JP-A-2006-199949

  However, in an imaging system including a plurality of imaging units as in Patent Literature 1, there is a problem that power consumption is increased as compared with an imaging device including a single imaging unit. Therefore, when photographing a subject whose appearance time is difficult to predict, such as a meteor, it is necessary to continuously operate the imaging device, but there is a concern about the operation time of the imaging device.

  SUMMARY An advantage of some aspects of the invention is to provide an imaging apparatus capable of reliably photographing a target object while suppressing power consumption.

  In order to achieve the above object, an imaging apparatus according to the present invention includes a pixel region in which a plurality of pixels are arranged in a matrix, a reading unit that reads a signal from the pixel region, and a first unit in the pixel region that is read by the reading unit. Detecting means for detecting an object of interest based on a signal read from a pixel group; and different from the first pixel group in the pixel area by the reading means according to the position of the object of interest detected by the detecting means Control means for controlling a signal to be read from a part of the second pixel group.

  According to the present invention, it is possible to provide an imaging apparatus capable of reliably photographing a subject of interest while suppressing power consumption.

FIG. 2 is a diagram illustrating a configuration of an imaging device according to an embodiment. FIG. 2 is a diagram illustrating a configuration of an image sensor according to an embodiment. FIG. 3 is a diagram illustrating an arrangement of pixel groups according to the embodiment. 6 is a timing chart of the embodiment. 6 is a flowchart of the embodiment. It is a figure showing the photography picture of an example. It is a figure showing the photography picture of an example.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The embodiment described below is an example as a means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of an apparatus to which the present invention is applied and various conditions. It is not limited to the embodiment.

(Example)
FIG. 1 is a diagram illustrating the configuration of the imaging apparatus according to the first embodiment. The imaging device includes an imaging device 101, an imaging device driving unit 102, a signal processing unit 103, an image display unit 104, a memory unit 105, an image recording unit 106, a synchronization control unit 107, an operation unit 108, an optical system 109, and an optical driving unit 110. It is composed of

  The image sensor 101 includes a pixel region in which a plurality of pixels are arranged in a matrix in the horizontal and vertical directions, and a circuit that outputs signals read from the pixels in a predetermined order. The image sensor driving unit 102 drives the image sensor 101 to read out pixel signals of the image sensor 101 at a predetermined timing.

  The signal processing unit 103 is controlled by a control signal from the synchronization control unit 107, and converts an output signal from the image sensor 101 into image data. In addition, image data is input to and output from the memory unit 105 and the image recording unit 106, and the image data is output to the image display unit 104.

  The image display unit 104 is controlled by a control signal from the synchronization control unit 107, and displays image data stored in the signal processing unit 103 and the image recording unit 106.

  The memory unit 105 is controlled by a control signal from the synchronization control unit 107, and temporarily stores image data from the signal processing unit 103. The image recording unit 106 is controlled by a control signal from the synchronization control unit 107, and records and reads image data from the signal processing unit 103.

  The synchronization control unit 107 controls the entire imaging device in accordance with an instruction from the operation unit 108. The operation unit 108 includes an input device such as a switch, a push button, and a touch panel mounted on the image display unit 104.

  The optical system 109 includes an imaging lens that forms a subject image on the image sensor 101, a zoom lens that performs optical zoom, a diaphragm that adjusts the brightness of the subject image, a shutter that controls exposure, and the like. The optical drive unit 110 is controlled by a control signal from the synchronization control unit 107 and drives the optical system 109.

  FIG. 2 is a diagram illustrating a configuration of the CMOS image sensor 101. In FIG. 2, for simplification, only a pixel array of m rows to m + 3 columns and n rows to n + 3 rows of 4 rows × 4 columns is shown, but actually more pixels 201 are arranged in a matrix. Is done.

  Each of the plurality of pixels 201 includes a photodiode (PD), a transfer transistor, a floating diffusion (FD), an amplification transistor, a selection transistor, and a reset transistor.

  The vertical scanning circuit 203 controls the reset signal ΦRES supplied via the control signal line RES, the selection signal ΦSEL supplied via the control signal line SEL, and the transfer signal ΦTxn supplied via the control signal line Tx. The pixel 201 in the readout row is controlled by a signal.

  The plurality of pixels 201 output analog signals photoelectrically converted by the PD according to the control signals supplied from the vertical scanning circuit 203. The plurality of pixels 201 are divided into a plurality of pixel groups, and drive control is performed at different timings. Here, a plurality of pixels in n rows are a first pixel group, and a plurality of pixels in n + 1 to n + 3 rows are a second pixel group. Note that the number of pixel groups may be larger.

  The signal output line 202a outputs signals of the plurality of pixels 201 in the first pixel group to the reading circuit 204. Further, the signal output line 202 b outputs signals of the plurality of pixels 201 in the second pixel group to the reading circuit 204. If the number of pixel groups is larger as described above, the number of signal output lines in each column also increases.

  The readout circuit 204 includes an AD conversion circuit that converts analog signals on the signal output lines a and b into digital signals, and a sample and hold circuit. The operation of the reading circuit 204 is controlled by the horizontal scanning circuit 205.

  The signal processing circuit 206 performs predetermined signal processing on the digital signal input from the readout circuit 204 and outputs the digital signal from the signal output ports a and b. Here, the signal output port a outputs a signal of the first pixel group, and the signal output port b outputs a signal of the second pixel group.

  Note that different pixel groups may exist in the same pixel row, but in this case, it is necessary to provide each of the control signal lines RES, SEL, and Tx for each pixel group in each pixel row.

  FIG. 3 is a schematic diagram illustrating an example of the arrangement of each pixel group in the pixel array. Here, only a pixel array of 8 rows × 8 columns is shown, but actually more pixels 201 are arranged.

  FIG. 3A shows an example in which two rows of first pixel groups 301 and six rows of second pixel groups 302 are alternately arranged every eight rows.

  Since the first pixel group is discretely arranged and the number of pixels of the first pixel group in the entire screen is small, a low-resolution signal obtained by thinning out the entire screen is recorded.

  On the other hand, since the second pixel group has a large number of pixels in the entire screen, a high-resolution signal in which a part of the angle of view of the entire screen is read out so as to include the target object is recorded.

  By performing such control, it is possible to simultaneously read out and record the image signal of the entire screen and the image signal including the high-definition subject at an appropriate frame rate.

  Further, each of the pixels 201 in the pixel array is provided with any of R, Gr, Gb, and B color filters so as to form a 2 × 2 Bayer array. In consideration of this arrangement, the arrangement may be such that two rows and two columns of four rows and four columns are the first pixel group 301 and the other twelve pixels are the second pixel group 302 as shown in FIG. 3B. .

  Further, three or more types of pixel groups may be arranged as shown in FIG. That is, two rows and two columns of the four rows and four columns are the first pixel group 301, and two rows and two columns adjacent to the first pixel group on the left and right sides are the second pixel group 302. Two rows and two columns vertically adjacent to the first pixel group are a third pixel group 303, and two rows and two columns obliquely adjacent to the first pixel group are a fourth pixel group 304.

  In the case of FIG. 3C, since the thinning rates of the first to fourth pixel groups are equal, the degree of resolution does not change for each pixel group. , Can be simultaneously recorded at an appropriate frame rate.

  Note that the following description is based on the arrangement of the pixel groups in FIG.

  FIG. 4 is a timing chart for reading an analog signal from the pixel 201 in the n-th row.

  At time t401, the selection signal ΦSELn supplied to the pixel 201 in the n-th row is set to the HIGH level, and the selection transistor is turned on, thereby selecting the pixel 201 in the n-th row.

  At time t402, the reset signal ΦRESn is set to the HIGH level to start resetting the FD of the pixel 201. At time t403, the reset signal ΦRESn is set to the LOW level to finish resetting the FD.

  From time t404 to time t405, the signal value of the pixel 201 in the n-th row at the time when the reset of the FD is completed at time t403 is sampled and held using the sample and hold circuit (not shown) in the readout circuit 204.

  From time t406 to time t407, the signal value at the end of the reset sampled and held by the sample and hold circuit is AD-converted by using an AD conversion circuit (not shown) in the readout circuit 204. Here, the AD-converted value is set as a reference signal value of the pixel 201.

  At time t408, the transfer signal φTxn is set to the HIGH level to turn on the transfer transistor, and transfer of the charge generated in the PD to the FD is started.

  At time t409, the transfer signal ΦTxn is set to the LOW level to turn off the transfer transistor, and the transfer of charges from the PD to the FD is completed.

  From time t410 to time t411, a signal value corresponding to the amount of charge transferred from the PD to the FD is sampled and held using a sample and hold circuit (not shown) in the readout circuit 204.

  From time t412 to time t413, a signal value corresponding to the amount of charge sampled and held by the sample and hold circuit is AD-converted using an AD conversion circuit (not shown) in the readout circuit 204. Here, the value obtained by the AD conversion is set as the optical signal value of the pixel 201.

  Then, a difference between the optical signal value and the reference signal value is output as an imaging signal to the signal output ports 207a and 207b corresponding to the pixel group.

  At time t414, the selection signal ΦSELn supplied to the pixel 201 in the n-th row is set to the LOW level to turn off the selection transistor, the pixel 201 in the n-th row is deselected, and the output of the pixel signal in the n-th row is ended.

  FIG. 5 is a flowchart showing a series of operations in this embodiment.

  In step S501, an image pickup signal from the first pixel group is output. In S502, it is determined whether or not the subject of interest is included in the imaging signal output from the first pixel group. Each time an image pickup signal is output from the first pixel group in S501, the presence or absence of a subject of interest is determined in S502.

  That is, the signal value of the imaging signal output from the G pixel (Gr pixel or Gb pixel) provided with the G color filter is compared with a predetermined threshold value Gth [LSB]. For example, the signal value output from G pixels continuous for 20 pixels or more is equal to or larger than the threshold value Gth [LSB], and the signal value output from the same pixel in the previous frame is smaller than the predetermined threshold value Gth [LSB]. Shall be. In that case, it is determined that the subject of interest exists at the pixel position.

  In the present embodiment, it is assumed that the target subject, such as a meteor, whose appearance location and appearance time are unknown and disappears within a few seconds after appearance, is considered as the target subject. Therefore, a method of determining the presence or absence of an object of interest from the luminance information of the first pixel group will be described as an example. However, if the characteristics of the object can be extracted, the object of interest can be extracted using other information such as color, not limited to the luminance information. It may be determined.

  If it is determined in S502 that the subject of interest does not exist, the process returns to S501, and continuously outputs an imaging signal from the first pixel group at a predetermined frame rate. In step S502, when the subject of interest appears in the image captured using the first pixel group as illustrated in FIG. 6A, it is determined that the subject of interest exists, and the process proceeds to S503.

  In step S <b> 503, while continuing to output the imaging signal from the first pixel group, the imaging signal is also acquired from the second pixel group, and the process proceeds to S <b> 504. In step S504, based on the imaging signal output from the first pixel group in step S502 and the imaging signal output from the second pixel group in step S503, the readout specification of the second pixel group (imaging angle of view, frame rate, Resolution). Note that the shooting angle of view, resolution, and frame rate when the target object is detected may be determined in advance.

  First, as shown in FIG. 6B, the position and the size of the target object are determined based on the image pickup signal output from the first pixel group, and correspond to a shooting angle of view including the target object. An area 701 is set. Further, as shown in FIG. 6C, an area 702 corresponding to the same angle of view as the area 701 is set for the second pixel group, and imaging is performed by the second pixel group. FIG. 6D is a diagram illustrating a captured image obtained by capturing an image of the area 702 using the second pixel group.

  With reference to FIG. 7, a description will be given of a method of obtaining a shooting angle of view when a subject of interest is detected. FIGS. 7A and 7B show the same captured images as FIGS. 6B and 6C, respectively.

  Here, a set of luminance values that are equal to or greater than a predetermined threshold Yth is defined as a subject of interest. Further, the average value of the coordinates of the subject of interest is defined as the center of gravity of the subject of interest, and the center of gravity of the subject of interest is obtained.

For example, the pixel 1, the pixel 2,..., And the pixel n output a signal having a luminance value equal to or larger than the threshold value Yth, and the coordinates of each pixel are set to (x1, y1), (x2, y2),. , The center of gravity (px, py) is as follows.
px = (x1 + x2 + ... + xn) / n
py = (y1 + y2 +... + yn) / n

  If the center of gravity of the subject of interest is defined as the position of the subject of interest in the captured image, the subject position of the subject of interest shown in FIG. 7A is 803, and the subject position of the subject of interest shown in FIG. Note that the barycentric position may be obtained by weighting the luminance value of each pixel of the subject of interest. Further, the position of the subject of interest may be displaced on the motion vector axis according to the magnitude of the motion vector of the subject of interest. Further, the position of the subject of interest may be set using the position of the center of gravity or the vector.

  Next, a circumcircle of the object of interest around the center of gravity is determined, and the size of the circumscribed circle is defined as the size of the object of interest. Here, FIG. 7A is a photographed image at a time when the target object is detected based on the output signal of the first pixel group, whereas FIG. 7B is a second image after the target object is detected. Is a captured image obtained by imaging using the pixel group of FIG.

  Therefore, when the subject of interest is a moving object, the position, size, and the like of the subject of interest in the captured image shown in FIG. 7A and the captured image shown in FIG. For example, the circumscribed circle 802 shown in FIG. 7B is larger than the circumscribed circle 801 of the target object shown in FIG. In this case, it is necessary to determine the shooting angle of view from the circumcircle 802 of the target object in FIG. 7B among the target objects in FIGS. 7A and 7B.

  The shooting angle of view needs to be set such that the diagonal line is larger than the diameter of the circumscribed circle of the object of interest to reduce the frame-out of the object of interest and the aspect ratio conforms to the display standard. For example, the length of the diagonal may be set so as to be 1.5 times or more the diameter of the circumscribed circle of the target subject, and the size may be set to be within 16: 9 of the HD standard.

  Next, how to determine the frame rate will be described. The displacement amount D in the captured image of the subject of interest per frame is determined in advance. The smaller the value of the displacement amount D, the smaller the movement amount of the object of interest between frames, and it is possible to obtain a captured image in which the movement of the object of interest between frames is apparently smooth. Therefore, the displacement of the subject of interest is obtained using the captured image of the area 702 in FIG. 6C obtained after a predetermined time at the same capturing angle of view as the captured image of the area 701 in FIG. 6B.

For example, the displacement amount of the subject of interest in the captured images of FIGS. 6B and 6C is d, and the readout time of the imaging signal in FIG. 6B and the readout time of the imaging signal in FIG. Is t. The frame rate F is obtained from the following predetermined displacement amount D of the subject of interest and the displacement amount d of the subject of interest per time difference t as follows.
F = d / (D × t)

  Then, a frame rate that can be selected at a higher speed than the obtained frame rate F is determined as a frame rate at the time of shooting the subject of interest. Further, the maximum achievable resolution is selected from the set shooting angle of view and frame rate.

  In S505 of FIG. 5, the subject position of the next frame is predicted based on the shooting angle of view, the frame rate, and the displacement amount d of the subject of interest determined in S504, and the second pixel group as shown in FIG. Of the read area 703 is determined.

  FIG. 7C is a diagram illustrating a change in the position of the subject of interest. The coordinates of the initial position 803 where the target object is detected in the region 701 of the captured image shown in FIG. 6B and the position 804 after the target object has moved in the region 702 of the captured image shown in FIG. The displacement amount 805 of the subject of interest can be obtained based on the coordinates. Then, as shown in FIG. 6C, the imaging area 703 of the second pixel group of the next frame is determined.

  In step S506, the image of the area 703 shown in FIG. 6F is photographed using the second pixel group in the entire photographed image of the actual next frame as shown in FIG.

  In step S507, it is determined whether the captured image of the area 703 captured in step S506 includes the subject of interest. If it is determined that the subject of interest is included, the process returns to S505. In FIG. 7C, assuming that the subject of interest moves further from the position 804 with the same vector, it is predicted that the subject of interest moves to the position 807 with the same displacement 806 as the displacement 805.

  That is, from the position 803 of the subject of interest in the region 702 of the captured image of the previous frame shown in FIG. 6C and the position 804 of the subject of interest in the region 703 of the captured image of the current frame shown in FIG. The position 807 of the subject of interest in the frame can be predicted. Then, as shown in FIG. 6E, the read area 704 of the second pixel group in the next frame is determined.

  In step S506, an image of the area 704 shown in FIG. 6H is captured using the second pixel group in the entire captured image of the actual next frame as shown in FIG.

  Again in S507, it is determined whether or not the subject of interest is included in the captured image of the area 703 captured in S506. If it is determined that the subject of interest is not included, the process proceeds to S508, and the imaging by the second pixel group ends.

  As described above, according to the present embodiment, the presence / absence of a subject of interest is determined using a captured image obtained using the first pixel group. Then, the image pickup signal of the second pixel group is read at a higher frame rate than that of the first pixel group, and control is performed so as to predict the movement of the subject of interest and change the area from which the image pickup signal is read from the second pixel group. . By predicting and photographing the position of the subject of interest in this manner, it is possible to shoot at a high frame rate without missing a high-speed subject. In addition, when the target subject is not detected within the shooting angle of view, control is performed so that only the first pixel group performs imaging at a low frame rate, so that power consumption can be reduced.

  If the detected object of interest does not disappear or go out of the frame and stays within the shooting angle of view for a long time, unnecessary data is continuously read at a high frame rate for a long time, and the power consumption increases. there is a possibility. In the meantime, it is also assumed that another subject of interest cannot be photographed. Therefore, control may be performed such that the imaging operation using the second pixel group is forcibly terminated when a predetermined time has elapsed after the start of imaging by the second pixel group.

  That is, in S507 of FIG. 5, not only the presence or absence of the subject of interest, but also the case where the imaging operation of the second pixel group is continuously performed for a predetermined time or more, or the case where the number of shots exceeds a preset number Alternatively, control may be performed so as to end the imaging by the second pixel group. Further, a plurality of conditions may be provided for distinguishing a subject of interest from a subject that is likely to be erroneously detected, such as a change in the color of the subject.

  By adding such processing, for example, when shooting a shooting star, if an erroneous detection of a subject is detected, such as detection of an airplane, the imaging operation by the second pixel group can be immediately terminated, and unnecessary power consumption can be reduced. Can be prevented.

(Other embodiments)
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. This processing can be realized. Further, it can be realized by a circuit (for example, an ASIC) that realizes one or more functions.

201 pixel 202 signal output line 203 vertical scanning circuit 204 readout circuit 205 horizontal scanning circuit

Claims (8)

A pixel region in which a plurality of pixels are arranged in a matrix,
Reading means for reading a signal from the pixel area;
Detecting means for detecting a subject of interest based on a signal read from a first pixel group in the pixel area by the reading means;
Control means for controlling the reading means to read a signal from a part of a second pixel group different from the first pixel group in the pixel area in accordance with the position of the target object detected by the detecting means. When,
An imaging device comprising:   The imaging apparatus according to claim 1, wherein a frame rate for reading signals from the second pixel group is higher than a frame rate for reading signals from the first pixel group.   The imaging device according to claim 1, wherein the first pixel group is discretely arranged in the pixel region.   4. The imaging apparatus according to claim 1, wherein the control unit determines an area from which a signal is read from the second pixel group so as to include the target object. 5.   5. The imaging apparatus according to claim 1, wherein the control unit determines an area from which a signal is read from the second pixel group based on a displacement amount of the target subject. 6.   2. The control unit according to claim 1, wherein when the imaging operation of the second pixel group is continuously performed for a predetermined time or more, the control unit controls to end the imaging by the second pixel group. The imaging device according to any one of claims 1 to 5.   The control unit controls the imaging by the second pixel group to end when the number of images captured using the second pixel group exceeds a preset number. The imaging device according to any one of claims 1 to 5.   The imaging apparatus according to any one of claims 1 to 5, wherein the control unit performs control to end imaging by the second pixel group when the color of the subject changes.

Images